Abstract
The question that lies at the center of both my STS research and technical capstone projects concerns a critical engineering challenge: how do we design AI systems responsibly in order to avoid algorithmic harm?
My STS research factors in the lack of accountability chains present in AI-driven security systems, especially in how fragmented responsibility perpetuates systemic damages, as no individual actor faces repercussions. Two case studies were analyzed: the Dutch Childcare Benefits Scandal and Chicago’s ShotSpotter deployment. As a result, a pattern emerged where, when millions of people rely on algorithmic systems without any single actor being held liable for consequences despite documented harm, these flawed systems persist. In leveraging the frameworks of Actor-Network Theory and the NIST Cybersecurity Framework, I evaluated five intervention points for which accountability can be integrated into the design process. These five points include pre-deployment validation, transparency/explainability, human oversight and appeals, community consultation, and explicit organizational governance. Moreover, the core argument criticizes the current approach to algorithmic implementation and places responsibility as a design obligation, instead of an afterthought.
My technical project is Cavalier Calendar, an AI-powered course scheduler that puts my STS research to the test. Regarding its use case, the project tackles the current lapses in scheduling design for UVA’s Student Information System, which creates barriers for first-year, transfer, international, and first-generation students. Results from our requirements surveys reveal that 72.7% of students encounter frequent disruptive session timeouts, 45.5% struggle with department codes, and 63.4% have experienced errors in scheduling derived from confusion. Features of the project include natural language processing and a Django backend. This allows students to interface through human-readable text, such as “Find me CS classes related to mobile development that don’t conflict with my schedule.” Effectively, this eliminates department code requirements while validating course prerequisites and resolving conflicts. The system aims to delineate course registration from scheduling, allowing students to have a responsive scheduler without the hassle of interacting with SIS, while combining the functions of extraneous resources such as Hoo’s List and Course Forum all into one app.
Building this system allowed me to reconcile the conditions affecting the algorithms mentioned in my STS research. Mirroring the capacity for harm, private student data was consolidated as the app needed context in order to assist in scheduling. We were developing a machine aimed towards vulnerable student populations with the aim of significantly swaying their enrollment decisions. If Cavalier Calendar were built neglecting the proper guardrails, the project would replicate every single systemic failure perpetuated by systems found in the Dutch case and ShotSpotter.
In realizing this, my intervention points governed design. Firstly, for pre-deployment validation, we built bias auditing into our sprint plan to combat overrepresentations of certain courses, especially non-required electives. Secondly, for transparency, we’re designing the system to articulate its course recommendations to combat epistemic dependence. Thirdly, human oversight is prioritized in the emphasis on the app as an assistive tool rather than a final authority on schedules. Fourthly, usability testing prioritizes the feedback of the targeted student populations, shapes preliminary system design, and enables community input. Lastly, our team has explicitly outlined which team member audits training data, validates prerequisites, and reviews error cases, ensuring that accountability is explicitly assigned rather than fragmented.
The combination of these projects demonstrates a successful approach to an issue core to algorithmic engineering, where ethical responsibility demands infrastructural design implementation rather than acting as a post-deployment afterthought. My STS examines what happens when deviating from this approach, and the sheer scale of harm when whole demographics are affected. Additionally, my technical project gave real-life insights into implementing these design requirements. Ultimately, the two projects investigated the hard question that is core to algorithmic engineering: what happens when accountability is fractured, and how do we fix it?
