diff --git a/assets/video/courses/frontiercs2-poster.jpg b/assets/video/courses/frontiercs2-poster.jpg new file mode 100644 index 0000000000000..042ac5f45fed1 Binary files /dev/null and b/assets/video/courses/frontiercs2-poster.jpg differ diff --git a/assets/video/courses/frontiercs2.mp4 b/assets/video/courses/frontiercs2.mp4 new file mode 100644 index 0000000000000..a5a4356ffccdf Binary files /dev/null and b/assets/video/courses/frontiercs2.mp4 differ diff --git a/src/content/writing/what-frontiercs-2-teaches-about-agent-results.md b/src/content/writing/what-frontiercs-2-teaches-about-agent-results.md new file mode 100644 index 0000000000000..c6b9136b4e862 --- /dev/null +++ b/src/content/writing/what-frontiercs-2-teaches-about-agent-results.md @@ -0,0 +1,43 @@ +--- +title: 'What FrontierCS 2 teaches about reading agent results' +description: 'I turned the FrontierCS 2 benchmark into an open course. Three details from its pass rule and leaderboard change how I read any agent score.' +publishedAt: 2026-10-28 +language: en +topics: [AI evaluation, Coding agents, Teaching] +audience: developers, students, and research readers +draft: false +--- + +[FrontierCS 2](https://frontier-cs.org/) asks a question that most coding benchmarks avoid: can an AI agent do computer-science research? Each of its 48 tasks starts from a published paper and the code its authors released. The builders delete the paper's key contribution, such as the queue logic in BBQ or the scheduling decision in Shockwave, and the agent has to write it back. I am one of the authors of the [paper](/publications/#mang2026frontiercs2), and I have now built an open course on it: [FrontierCS 2, one lecture per field](https://boxi-cs.github.io/frontiercs2/), fourteen lectures in all, from an introduction through MLSys, databases, cryptography, and robotics. + +Writing the lectures changed how I read agent results. Three details from the first lecture do most of the work. + +## A pass means beating the authors, on every check + +A run passes only when its final submission beats the human reference, which is the authors' own code, on every metric the task checks and on every hidden workload. Some checks ask for a margin. On SVG-EAR, the latency has to be at most 0.96 of the reference's, and image quality has to stay within small margins of it. + +So a candidate that is 3% faster fails, and so does one that is 10% faster but loses too much quality. The course turns this into a ten-line function you can change and rerun. The point is that "better than the reference" is many conditions at once, and a headline number hides which one failed. + +The same idea explains a case in the notes: on KV Compaction, Claude Opus 5.5 has the top score and sits above the human reference, yet its answers fall short of the reference's accuracy on 2 of the 24 articles, so the run fails. A high average can hide a loss in one place that was bought with a gain in another. + +## The agent has to choose its own final version + +During a run the agent sees a score on development workloads for every patch it submits. The hidden workloads stay with the judge, and only the version the agent names as its final submission is scored on them. + +That makes choosing part of the work. The first lecture uses a toy run with five submissions to show the failure: the version with the best development score was fitted to the development workloads and does worse on the hidden ones, while the last version beats the reference. The numbers in that toy are illustrations, not benchmark data. What the benchmark reports is real: it scores what the agent chose, even when an earlier version would have passed. + +When you read any agent evaluation, ask what the agent could see while it worked, and who decided which answer counted. + +## A leaderboard row is a model and its harness + +Each model runs in its own agent harness: Claude Opus 5.5 in Claude Code, the GPT-6 models in Codex CLI, Gemini 4 Argon in Antigravity and in Pi. The harness matters. Under Pi, a provider's content filter refused a request and ended the run in 10 of Gemini 4 Argon's 48 runs, while Antigravity retries refused requests. As of October 26, 2026, the same model shows a 54% pass rate under one harness and 40% under the other. + +So "model X scores Y" is shorthand. The tools, the retry policy, and the way a run ends are part of the measurement. I wrote the lecture so that learners predict this gap before the page explains it. + +## How the course is built + +Each later lecture takes one field and, for every task in it, covers four things: the idea from the paper on a small example you can run, what the agent writes, how the task is judged, and what the agents did. The Python models are ours and run offline, every program has been run, and every output shown is the real output. We use only public material: the task statements the agent reads, the papers, and the pages on frontier-cs.org. Nothing comes from a task's judge, so the course does not spoil the tasks for anyone who wants to run them. + +Results change, so each lecture states the date of its numbers. If you want one place to start, Lecture 4 (Network) shows three different ways a run can pass or fail. + +The course sits with the rest of my [interactive course notes](/courses/). diff --git a/src/data/courses.ts b/src/data/courses.ts index 68195a94fe678..a9dc6af881ac6 100644 --- a/src/data/courses.ts +++ b/src/data/courses.ts @@ -3,7 +3,7 @@ * Courses page, ordered from AI literacy for everyone, through a first * program, discrete mathematics, data structures, linear algebra and machine learning, to coding * agents. Boxi Yu - * designed AI for All (AI 通识课) and LA101, and teaches Data Structures with + * designed AI for All (AI 通识课), LA101 and the FrontierCS 2 course, and teaches Data Structures with * Yintong Huo, and DM101 with Mingtao Xia; AIE1001, CSC3100, DDA3020 and CSC4801 are * CUHK–Shenzhen courses taught by their own instructors, with notes and * videos co-designed by Boxi Yu. Lecture titles follow the published sites @@ -303,5 +303,39 @@ export const featuredCourses: Course[] = [ ], tone: 'strawberry', }, + { + id: 'frontiercs2', + clip: true, + badge: 'FrontierCS 2', + name: 'Can AI Agents Do CS Research?', + level: 'Advanced · Research benchmark', + context: 'Open course in English', + role: 'Designer', + people: [{ name: 'Boxi Yu', href: '/about/' }], + links: [{ label: 'The benchmark', href: 'https://frontier-cs.org/' }], + summary: + 'A tour of FrontierCS 2, a benchmark that removes the key contribution from a research paper and asks an agent to write it back. Each lecture takes one field, from systems and databases to cryptography and robotics: the paper’s idea, how a run is judged, and what the agents wrote. Every idea comes with a small model you can run.', + detail: 'Notes, three or four short videos per lecture, results as of October 26, 2026', + href: 'https://boxi-cs.github.io/frontiercs2/', + unit: 'lectures', + firstLecture: 0, + lectures: [ + 'Introduction', + 'MLSys', + 'HPC', + 'Architecture', + 'Network', + 'Theory', + 'Bioinformatics', + 'Machine Learning', + 'Database', + 'Programming Languages', + 'Software Engineering', + 'Cryptography', + 'Security', + 'Robotics', + ], + tone: 'teal', + }, ]; diff --git a/src/pages/courses.astro b/src/pages/courses.astro index 13f2aac1ca2b2..367e30cd1adbe 100644 --- a/src/pages/courses.astro +++ b/src/pages/courses.astro @@ -5,21 +5,21 @@ import CourseShowcase from '../components/CourseShowcase.astro';

Courses · interactive lecture notes and videos

Interactive notes for every lecture.

- Companion sites for eight courses, from AI literacy for everyone to building software with + Companion sites for nine courses, from AI literacy for everyone to building software with coding agents. Each lecture walks through its ideas one step at a time, asks you to predict before it shows you, and ends with a self-check.

Agentic SE open course ↓
-
+
diff --git a/src/pages/index.astro b/src/pages/index.astro index 1f090faa30486..0f8190b35364d 100644 --- a/src/pages/index.astro +++ b/src/pages/index.astro @@ -103,14 +103,15 @@ const latest = [

Courses · interactive lecture notes

-

Eight courses with interactive lecture notes

+

Nine courses with interactive lecture notes

All courses

- For eight courses, I build or co-build one interactive page per lecture: from AI literacy for + For nine courses, I build or co-build one interactive page per lecture: from AI literacy for everyone, through a first Python program, discrete mathematics, data structures, linear - algebra, and machine learning, to building software with coding agents. Every program and + algebra, and machine learning, to building software with coding agents and testing them on research + tasks. Every program and interactive check has been run, and every output shown is genuine.

diff --git a/src/pages/teaching.astro b/src/pages/teaching.astro index 62b3c21c129e6..c018a4bce4079 100644 --- a/src/pages/teaching.astro +++ b/src/pages/teaching.astro @@ -54,7 +54,7 @@ const mentees = [ concept, build a small artifact, and test what it can actually do.

- Browse the eight courses + Browse the nine courses See supervision See mentoring Try a test-quality exercise