S 933: The AI Medical Diagnosis Problem: Why Accuracy Falls Apart in Real Conversations

<p>AI can score highly on medical exams, identify conditions in written cases, and still perform poorly when real people use it for their health. In this episode, I break down new research that tested what happens when members of the public use AI tools to interpret medical scenarios and decide where to seek care. The results point to a problem that has less to do with the model's knowledge and more to do with the human interaction around it. </p><p><br /></p><p>Topics discussed: </p><p>- The Hemorrhage Case Study</p><p>- The Nature Medicine Study Design</p><p>- 95% to 35% Accuracy Drop</p><p>- The Control Group Outperformed AI</p><p>- Where the AI-Human Handoff Fails</p><p>- Reading Confidence as Competence</p><p>- The 1 in 6 Adults Using Health Chatbots </p><p><br /></p><p>---------- </p><p><a href="http://www.metabolismschool.com/" target="_blank" rel="ugc noopener noreferrer">⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠My Live Program for Coaches: The Functional Nutrition and Metabolism Specialization ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠</a></p><p><a href="http://www.metabolismschool.com/" target="_blank" rel="ugc noopener noreferrer">⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠www.metabolismschool.com⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠</a></p><p>---------- <br />[Free] Metabolism School 101: The Video Series<br /><a href="http://www.metabolismschool.com/metabolism-101" target="_blank" rel="ugc noopener noreferrer">⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠http://www.metabolismschool.com/metabolism-101⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠</a><br />----------<br />Subscribe to My Youtube Channel: <br /><a href="https://youtube.com/@sammillerscience?si=s1jcR6Im4GDHbw_1" target="_blank" rel="ugc noopener noreferrer">⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://youtube.com/@sammillerscience?si=s1jcR6Im4GDHbw_1⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠</a><br />----------<br /><a href="https://metabolismmadesimple.com/" target="_blank" rel="ugc noopener noreferrer">⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Grab a Copy of My New Book - Metabolism Made Simple⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠</a><br />----------<br /> Stay Connected: <br /><a href="https://www.instagram.com/sammillerscience/" target="_blank" rel="ugc noopener noreferrer">⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Instagram: @sammillerscience⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠</a><br /><a href="https://www.youtube.com/channel/UCKi5AgCKH37irVYvlIYC-vA" target="_blank" rel="ugc noopener noreferrer">⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Youtube: SamMillerScience⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠</a> <br /><a href="https://www.facebook.com/groups/nutritioncoachingcollaborative" target="_blank" rel="ugc noopener noreferrer">⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Facebook: The Nutrition Coaching Collaborative Community⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠</a><br /><a href="https://vm.tiktok.com/ZTdPVQtMH/" target="_blank" rel="ugc noopener noreferrer">⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠TikTok: @sammillerscience⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠</a><br />----------<br /><em>“This Podcast is for general informational purposes only and does not constitute the practice of medicine, nursing or other professional health care services, including the giving of medical advice, and no doctor/patient relationship is formed. The use of information on this podcast and the show notes or the reliance on the information provided is to be done at the user's own risk. The content of this podcast is not intended to be a substitute for professional medical advice, diagnosis, or treatment and is for educational purposes only. Always consult your physician before beginning any exercise program and users should not disregard, or delay in obtaining, medical advice for any medical condition they may have and should seek the assistance of their health care professionals for any such conditions. By accessing this Podcast, the listener acknowledges that the entire contents and design of this Podcast, are the property of Oracle Athletic Science LLC, or used by Oracle Athletic Science LLC with permission, and are protected under U.S. and international copyright and trademark laws. Except as other