Optimizing APIs: Using Precomputation to Reduce Complexity and Latency; Comparison of Automation and Manual Labor Jobs

Automation is changing the way workers accomplish tasks by increasing efficiency and productivity. During my capstone project, I interned at a company providing a cloud-based file storage service, where I extensively used automation tools in my workflow to test, build, and deploy code changes. Automation was an integral part of the company, spanning company-wide tools and infrastructure to team-specific applications. However, as the capabilities of automation technology advance, there is growing concern about job security in the workforce, especially in jobs characterized by repetitive tasks, like manufacturing. I was able to experience the benefits of automation in the field of software engineering, yet, I recognize the risk of automation in other industries. My experience with my capstone project inspired me to focus my STS research paper on how automation technology augments software engineering while displacing workers in manufacturing jobs.
My project was focused on optimizing an API that returns a given user’s file usage that faced latency issues. The issues stemmed from the current implementation of the API essentially “counting” each individual file per customer. To remedy the issue, I worked with a team of software engineers to optimize the API by precomputing these values into a database, essentially reducing the latency to a database read operation. This process involved identifying and modifying key components of the Java Spring backend of the product and interacting with existing and new AWS infrastructure. The P99 latency of the API was reduced from 2 seconds to 40 milliseconds and removed an estimated 800,000 daily file usage aggregation operations from a stressed datastore. Implementing the project in production will involve adding existing user usage into the database and testing for edge cases resulting from data inconsistency and scale.
The focus of the STS research paper is to discuss how software engineering is a highly specialized job that benefits from automation technology by augmenting workers to increase efficiency and reduce cognitive load, while automation in jobs like manufacturing works to replace workers, and face displacement with little opportunity to upskill. It attempts to analyze the forces that drive automation in the context of a business, and how the introduction of automation works to augment or displace workers depending on the characteristics of the job. It also discusses the role objects produced by generative artificial intelligence, such as music, code, and art, could have on society. The paper draws from relevant literature that attempts to quantify the risk of automation on the job security of workers, identifies qualities and characteristics of jobs that are automation-resistant, and analyzes existing legislation or legal texts that attempt to analyze or regulate the effects of automation.
Working on these projects in tandem allowed me to gain a deeper understanding of the benefits and risks of automation in the workplace. By experiencing the benefits of automation in my role as a software engineer intern, I was able to experience firsthand how automation works to augment workers and increase productivity in a highly specialized field. It allowed me to apply my learnings from my capstone project to my research paper by providing a practical context to the theoretical concepts I was exploring. This formed the foundation for my STS research paper where I wanted to focus on how automation is not equally beneficial and can unequally place risk on the job security of other industries. As engineers that directly benefit from or could develop automation technologies, it is important to consider the implication of this work on other professions. This is especially relevant as artificial intelligence continues to evolve and becomes more widespread.