Building an Enterprise AI Agent for Healthcare

Every capability in an agent needs its own evidence and release bar. A model-provider slip, an incorrect tool call, and a wrong fertility-benefits answer should not be held to the same pass rate.

William Horton, Staff AI Engineer at Maven Clinic, joined us the day after Maven Assistant reached its first external users. The agent helps members inside Maven Clinic’s women’s and family healthcare platform find providers, manage appointments, navigate Maven, and get basic health information. William had spent much of launch day reading chat traces and turning the surprises into product decisions and tests.

William shows how a production failure moves through Maven’s system: the trace becomes a regression case, code handles deterministic checks, and LLM judges cover behavior that cannot be reduced to exact outputs. Human labels calibrate those judges, while the consequence of a wrong answer determines whether the capability ships. You can apply the same release workflow to the agent you are building now.

“For a lot of our tool-call evaluation, I’ll accept that it runs ten times and passes nine times. Going for that ten out of ten is just not worth the effort.”

— William Horton, Staff AI Engineer, Maven Clinic

You can also find the full episode on Spotify, Apple Podcasts, and YouTube.

👉 Want to build agents from the ground up? Registration is open for Build AI Agents from First Principles, a live workshop on the loops, tools, context, harnesses, and engineering decisions behind useful AI agents. You'll learn how to design agent systems from first principles, with enough structure to decide which harness patterns your product actually needs. Sign up today with vg-code for 10% off 👈

In This Episode

* The architecture behind Maven Assistant. A stronger lead agent routes requests to four narrower specialists for appointments, provider search, health questions, and Maven support. Hard guardrails run before dynamic routing.

* Why an enterprise healthcare assistant only needs 15 to 20 tools. Maven divides a manageable toolset across its specialists instead of exposing one model to hundreds of choices. Existing APIs become safer agent tools, with user identity and other application state injected by code.

* Turn failures into the cheapest reliable eval. A response claiming the agent was “made by Google” became a string check, tool calls are verified deterministically, and LLM judges handle clinical accuracy and other qualitative behavior.

* Set release thresholds from the consequences. Nine passes in ten can be acceptable for a cheap failure. Maven withheld benefits answers that could influence tens of thousands of dollars and routes self-harm language directly to human support.