A Debate about Designing for Predictability: Mental Models of AI Error Boundaries

This research explores how human-AI collaboration relies heavily on the ability of people to build accurate mental models of artificial intelligence error boundaries rather than solely depending on algorithmic accuracy. When organizations deploy AI in critical fields like healthcare and criminal justice, team performance often falters because users struggle to recognize when a system will succeed or fail. To fix this, the literature argues that models should be designed with parsimonious and non-stochastic error patterns that make failure modes easy to understand and anticipate. Furthermore, organizations must implement structured training, clear uncertainty communication, and thoughtful software updates to prevent misplaced trust or unwarranted distrust in automated recommendations. Ultimately, successful deployment requires shifting the focus of development toward team performance and human-centered governance instead of prioritizing raw predictive metrics alone.