Evaluating AI World Models w/ Keyon Vafa

<p>Keyon Vafa (postdoc at Harvard) studies the implicit world models that generative models learn (or at least are constrained into representing). How do we make sense of these models and how do they make sense of us? Experimentally, answering this question often looks like giving the model anything and everything: cookie ingredients, language, shrimp taxonomies, protein structures, a sample of ABBA’s discography—and models will try to make sense of it. Under a methodologically unified view of an incredibly variegated world with diverse objects, Keyon has tried to figure out if models can learn the map of Manhattan, the laws of physics, the political ideology of Senators, and more. This is partially an interdisciplinary agenda and partially an anti-disciplinary agenda. </p><p>In this episode, we cover Keyon&#39;s research on world models, human and AI interaction, machine learning methods to study the gender wage gap, the implications of thinking with and through predictability vs. causality, and documentary film making. It&#39;s a blast, as always, to be thinking methodologically and free ourselves from the bounds of disciplinary objects! </p><p><a href="https://keyonvafa.com/" target="_blank" rel="ugc noopener noreferrer">⁠Keyon Vafa&#39;s Website⁠</a></p><p><a href="https://www.youtube.com/watch?v=XG8AGxIVa-k&pp=ygUYa2V5b24gdmFmYSBzZW5hdGUgc3BlZWNo0gcJCU4LAYcqIYzv" target="_blank" rel="ugc noopener noreferrer">⁠Senate Speech presentation⁠</a></p><p><a href="https://arxiv.org/abs/2406.03689" target="_blank" rel="ugc noopener noreferrer">⁠Taxi rides⁠</a></p><p><a href="https://arxiv.org/abs/2202.08370" target="_blank" rel="ugc noopener noreferrer">⁠CAREER paper⁠</a></p><p><a href="https://read.dukeupress.edu/american-speech/article-abstract/85/1/91/5865/INDEXING-POLITICAL-Persuasion-VARIATION-IN-THE?redirectedFrom=fulltext" target="_blank" rel="noopener noreferer">Indexing Political Persuasion: Variation in IRAQ vowels </a></p><p><a href="https://deepmind.google/science/alphafold/" target="_blank" rel="ugc noopener noreferrer">AlphaFold</a></p><p><a href="https://arxiv.org/abs/2310.00865" target="_blank" rel="ugc noopener noreferrer">David Donoho on frictionless reproducibility </a></p><p><a href="https://en.wikipedia.org/wiki/Frederick_Wiseman" target="_blank" rel="ugc noopener noreferrer">Frederick Wiseman</a><br><a href="https://www.pnas.org/doi/10.1073/pnas.1915006117 " target="_blank" rel="ugc noopener noreferrer">Measuring the predictability of life outcomes with a scientific mass collaboration</a><br></p>