Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 4 - Reinforcement Learning with LLMs

Item: Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 4 - Reinforcement Learning with LLMs
Rating: 5
Author: PostSingularity Institute

by Stanford Online (Afshine Amidi & Shervine Amidi) · View the channel on PSI

Formal & Physical Sciences AI — Models & Research UYQArtificial intelligence (AI)UYQMMachine learning

🎙 Stanford Online (Afshine Amidi & Shervine Amidi) 👥 1.3M 📅 October 19, 2026 ⏱ 103 min 👁 12K 📄 tutorial 🧭 2026-10-20

Available in: English (current) Français

Keywords

policy gradientPPODPOreward modelRLHF

Summary

This Stanford lecture on reinforcement learning (RL) for large language models systematically explains how RL-based methods are used for post-training, focusing on reward design, policy gradient algorithms, and recent optimization variants. The instructors derive the policy gradient from first principles, introducing importance sampling, and discuss challenges like high-variance gradients, credit assignment, and reward hacking. They then present Proximal Policy Optimization (PPO) with clipping and KL penalties, connect it to Direct Preference Optimization (DPO), and compare it with Group Relative Policy Optimization (GRPO). The lecture covers limitations of GRPO and introduces advanced methods such as DAPO, Dr. GRPO, and On-Policy Distillation (OPD, OPSD). Throughout, the emphasis is on intuitive explanations followed by rigorous math, with practical considerations like computational efficiency. The session concludes with a summary of the key methods and their trade-offs, providing a comprehensive overview of the state of the art in RL-based LLM alignment.

146 words

Critical Evaluation

Value of the Information & Strength of the Argument

The lecture excels in presenting complex RL concepts with clarity and depth. It starts with a solid intuition, then derives the policy gradient formula step-by-step, making the mathematics accessible. The instructors address practical challenges (variance, credit assignment, reward hacking) with concrete examples, and justify each solution (e.g., baseline, clipping) with reasoning. The argumentation is logical and progressive, building from basic RL to cutting-edge methods like GRPO and DAPO. They also connect ideas (e.g., PPO to DPO), showing relationships between algorithms. The value lies in its pedagogical approach that bridges theory and practice, preparing viewers to understand research papers and implement these methods.

Scientific Rigor, Source Quality, Title Accuracy

The scientific rigor is high: the mathematical derivations are correct and clearly explained, with careful notation to avoid confusion. The content aligns with established RL literature and recent advances (PPO, DPO, GRPO are well-known). The use of Stanford’s official course materials and the syllabus link as primary sources lends credibility. The title precisely matches the lecture’s scope, which focuses on RL with LLMs. No discrepancies were found between title and content.

188 words

Title / Content Match

The title accurately reflects the content: a lecture on reinforcement learning applied to LLMs, including theoretical foundations and recent variants.

Quality & Reliability

9/10

Lecture from Stanford University with rigorous mathematical derivations, covering established RL methods (PPO, DPO, GRPO) and practical considerations; high credibility and pedagogical quality.

Chapters

Cited Sources

CME295 Syllabus — Official course syllabus provides structure and references for the lecture series.
Stanford Online Graduate Education — Overview of Stanford’s graduate programs, including CME295.
Playlist of CME295 lectures — Access to all lectures in the series.

Concurring Sources

Syllabus of CME295 — Confirms the lecture's place in the course and lists related topics.

Contribution & Novelties

The lecture provides a structured, mathematically rigorous yet intuitive introduction to RL-based post-training for LLMs, uniquely combining derivations of policy gradient with practical discussions of PPO, DPO, and GRPO. It bridges the gap between standard RL textbooks and recent research, offering insights into why certain methods work. The explicit connections between algorithms (e.g., PPO to DPO) and the critical analysis of limitations (especially for GRPO) add original value, making it an excellent resource for both newcomers and practitioners.

Pour aller plus loin :

Proximal Policy Optimization — The core algorithm detailed in the lecture; this article explains the clipping objective and trust region approach.
Reinforcement Learning from Human Feedback — Provides background on reward modeling and the RLHF pipeline central to the lecture’s context.
Importance Sampling — Key statistical technique used to reweight samples from old policies, discussed in the lecture.

141 words

Radar Profile

The radar profile shows high scores across all dimensions, reflecting the lecture's comprehensive coverage, strong mathematical grounding, and up-to-date methodology. The only potential weakness is the lack of hands-on practice, but as a lecture, it sufficiently covers both theory and implementation considerations.

Reliability 9/10