Exploring the Efficacy of a Two-Stage UI Grounding Mechanism; The Shifting Tides of AI-Art : An Actor-Network Perspective 5 views

The sociotechnical research paper explores the shifting interactions between AI technologies and their role in the artistic landscape. There often seems to be a general discourse surrounding the use of AI in artistic media. Many cite AI as innately “anti-art”, soulless, ugly, and unfair. The potentially unfair use of social media data to train AI systems adds fuel to artists' criticisms of AI. However, many of these concerns, in one way or another, have recurred throughout history as responses to changing AI technology landscapes. The advent of photography, for example, was initially thought to completely obviate traditional painting as a medium. This paper explores these struggles and attempts to use historical and technological analyses to reach an understanding of how AI should be viewed from an artistic standpoint. The technical report project explores the use of Reinforcement learning and Supervised fine-tuning to build a custom graphical user interface grounding system. Computer-use agents, agents that can navigate computers, are becoming increasingly popular for automating workflows and tasks. In many ways, this technology is still lacking, and more research is being poured into exactly how these systems should be trained to optimize performance. In this paper, we attempt to do just that. And to add to it, we use mechanistic interpretability to see whether certain inference-time interventions can manually improve grounding abilities.

BS (Bachelor of Science)

UI Grounding; Activation Steering; Artificial Intelligence and Art

School of Engineering and Applied Science Bachelor of Science in Computer Science Technical Advisor: Yen-Ling Kuo STS Advisor: Travis Eliott Technical Team Members:

English

All rights reserved by the author (no additional license for public reuse)

2026-05-09