Abstract
Blind and visually impaired (BVI) individuals use assistive technologies to independently access visual information and interact with their surroundings. These technologies substitute vision with audio or haptic channels, and this substitution introduces a substantial asymmetry in bandwidth. Vision delivers a scene at once and in rich detail, letting sighted individuals perceive and act almost automatically. Audio and haptic channels are sequential and far more limited in what they can present at a time, requiring the user to perceive each piece of feedback, interpret it, and act on it before the next arrives. Reaching a goal can thus take many passes through this loop, making even routine tasks slow, deliberate, and cognitively demanding. In this dissertation, I take this perceive-interpret-act loop, the action cycle underlying any goal-directed task, as a lens for understanding where assistive interaction becomes costly and for designing assistance that acts at those points. Through it, I argue that assistance is most effective when it intervenes in the cycle to advance the user's specific goal, rather than only relaying the scene and leaving the user to reach the goal themselves.
I present three distinct paradigms for intervening in this cycle and instantiate each with a system in a different domain of nonvisual interaction. Each paradigm acts on a different part of the cycle, reshaping its components, taking over the loop itself, or rebuilding the environment it runs in. First, CustomSight optimizes the components of the cycle for visual information access. Where LLM-powered visual assistants still require a well-framed image that is impractical to capture without sight, it generates a tailored computer-vision pipeline that guides the user with contextual audio feedback and captures the relevant image on its own. Next, FingerRover bypasses the cycle for spatial interaction. Rather than have the user interpret audio cues and correct their reach step by step, this small finger-worn robot moves their hand directly to the target. Finally, Navigal restructures the cycle for screen-reader navigation. Rather than have the user listen through an interface in sequence to find what matters, it rebuilds the interface around the elements relevant to their goal. Beyond these systems, I organize the three paradigms into a design space, situating existing assistive technologies within it and examining the tradeoffs that inform which paradigm fits a given task. Together, these contributions describe a way of designing assistance around what a BVI user is trying to accomplish, organizing the interaction towards the goal rather than only conveying the scene.
