Machine Learning for Online Accessibility: Machine Learning Algorithm to Generate Alternative Text for Digital Social Media Imagery; Comparison of Online Accessibility Policy in the United States vs. the European Union

Despite large-scale societal dependence on the internet, those with visual disabilities face acute difficulties in interacting with many online platforms. The challenges stem from the fact that large portions of information on a web page are generally conveyed through visual elements such as graphs, colors, page layout, and images. The web pages themselves contain technical deficiencies that result in a dramatically different internet experience based on user visual ability. My technical capstone research provides a proposal to address one of these technical shortcomings in the form of a machine learning algorithm that can automatically generate descriptive, alternative text for digital images. My STS research presents a comparison of online accessibility policies in the United States and the European Union to understand the digital experience for visually disabled internet users and the contribution of government involvement through policy.
My capstone technical research is a proposal of a machine learning algorithm to create descriptive text for images. Currently, visually disabled internet users rely on screen readers and oral translations of hard-coded alternative text to access digital imagery, but face accessibility challenges when this crucial text is often missing from the HTML of the page. To address this recurring issue, I propose a machine learning-based browser add-on that generates accurate and descriptive text content for photos. I propose the use of TensorFlow, Google’s open-source machine learning software library, and the Python programming language to write the code for this algorithm. To train the model I propose the use of a large dataset containing images representative of those typically posted on social media sites to specialize the model in generating text for social media imagery. The anticipated outcome of this algorithm is a browser extension that automatically generates descriptive text for social media images that screen readers or text-to-speech translations can communicate to the user. In future development, model accuracy can be improved with more training data, in addition to utilizing user feedback to fine-tune the nature of the text generated.
My STS research explores how in the digital age, accessibility concerns and considerations previously relegated to the physical world expand into online platforms. To ensure adequately accessible products, certain software development practices are enforced through governmental regulation. The content and enforcement of these regulations directly define the online experience of disabled internet users. To understand how effective these policies are in ensuring disabled persons have equal opportunity online, there is valuable insight to be gained in analysis of my research premise: comparing online accessibility policies in the United States (US) and the European Union (EU). My STS research uses the theory of technological politics to contextualize how government regulation shapes the socio-political implications of website technology. Technological politics provides the foundation in understanding how the technology of websites has inherent societal impact. I use accessibility policy source content, as well as case studies demonstrating their application, to conduct comparative analysis on accessibility regulations and their impacts in the US and the EU. I outline criteria by which a policy may be deemed “successful”, and I aim to make determinations about each governmental entities major regulations and whether they are effective. Through comparison of the US and the EU, I expect to reach conclusions regarding which approach methods show more promise in ensuring long-lasting digital inclusivity. As society becomes more technologically dependent, online accessibility will become increasingly critical, so my research analyzing, critiquing, and comparing US and EU accessibility policy through the STS lens of technological politics proves significant in providing background for future policy changes.
My technical and STS research projects both provide a different approach to the issue of online accessibility. Having the opportunity to conduct both research initiatives concurrently allows me to gain a deeper understanding of each one individually. While conducting my STS research, it was extremely valuable to understand the technical components of the website technology I was researching. I could understand in depth where the technical deficiencies lie in web applications and why precisely the government must impose regulation on software developers. Conversely, while I was conducting my technical research, it was paramount to fully understand the social motivations behind the development of my proposed algorithm. Conducting the research simultaneously gave me a more well-rounded and robust understanding of online accessibility in a broader context, which strengthened my specialized research efforts in a way that I would not have accomplished if they had been done separately.

School of Engineering and Applied Science

Bachelor of Science in Computer Science

Technical Advisor: Rosanne Vrugtman

STS Advisor: Bryn Seabrook

Technical Team Members: Kelly Bodner