Enhancing Safety and Trustworthiness for Autonomous Agents Under Uncertain Effects

Autonomous agents, such as automated vehicles and delivery robots, are increasingly prevalent in human environments. However, autonomous agents’ ability to perceive the environment is often flawed due to sensor noise. These perception uncertainties can lead to safety hazards and can negatively impact human trust in these agents. We aim to design effective policies to guide autonomous agents’ actions for safe operation around humans. We employ partially observable Markov Decision Processes (POMDPs) as the framework for planning under uncertainty. We study policy design in three distinct approaches. Firstly, we focus on safety requirements in static environments, represented as almost-sure reach-avoid specifications. We develop factored shields to restrict the unsafe actions of agents and integrate them into a safe online planning method. Secondly, we extend our analysis for safe planning among other dynamic agents. We use conformal prediction to obtain prediction regions for other agents’ future position and develop an online method to avoid these regions with a probabilistic safety guarantee. We further design a cost-constrained planner for personalized automated vehicle cruise control. Lastly, we explore multi-objective route planning while accounting for human trust dynamics in automated vehicles. By incorporating data-driven models of trust dynamics and takeover decisions into the POMDP framework, we analyze trade-offs between planning goals via multi-objective optimization. All these approaches have been implemented as prototype tools. Experimental results in a range of benchmark studies demonstrate their effectiveness. The proposed approaches can assist in developing safe and trustworthy autonomous agents.