188: World Models

Intro topic: Running

News/Links:

Flow matching versus diffusion modelshttps://youtu.be/firXjwZ_6KI?is=QMq8DcCsXTTktOuE OpenCV 5https://opencv.org/opencv-5/Claude fable beats pokemon with no harnesshttps://youtu.be/Ty_50J84fMY?si=EJ1KjZCZegipfCsV Can the stockmarket swallow Anthropic, SpaceX and OpenAI? https://archive.ph/nKEVw

Book of the ShowPatrickStrength of the Few - James Islingtonhttps://amzn.to/4pmr10TJasonDescender - Jeff Lemirehttps://amzn.to/3QKhp3l

Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h

Tool of the ShowPatrickNo Man’s SkyJasonPaperlib https://paperlib.app/en/

Topic: World Models

Making decisions with AIAction-Value (called a Q model): What is the long-term value of making a decision at a positionPolicy: What action should I take (must be a distribution)Value (called a V model): What is the value of a position (depends on policy)Advantage/Disadvantage: difference in value given two policiesWhen advantage is +, do that more.Model-FreeLook at the current situation and suggest an actionRun that action in the real world and measure the effectUse that measurement to suggest better actions next timeModel-BasedObserve rollouts (sequences of situations) and learn the dynamicsChoose an action, use your dynamics model to measure the consequencePotentially do MPC (try many actions and choose the best)World ModelsObserve many many rollouts and learn a full forward model (how to create the input in the future)Train a policy & value inside the world modelDeploy the policy and fine-tune based on the real world