Start course

Week 1 · Day 5 of 10

Evals that tell the truth

Students can build a test set, score their app against it on every push, and prove a change made it better.

Robert introduces today: what you'll learn, what you'll build, and why it matters. About two minutes.
In pink: what day 5 adds to the app.
Day 5 adds Evals to the student's app, which already has Your user, FastAPI app, Model, Typed JSON, Tools, Your documents.
1LearnTest sets and evals in CI

Building a test set from real questions, exact checks versus model-graded checks, running the evals with pytest on every push in GitHub Actions, and reading the failing cases instead of trusting the score. From today, every change to the app is measured.

How it worksStep 1 of 5

How evals prove your app works

  1. Test cases
Each case is a question and what a good answer looks like.
Check your understanding3 questions
  1. When should you use a model-graded check instead of an exact check?

  2. Why run evals in CI on every push?

  3. After a prompt change, your eval pass rate drops from 92% to 85%. What do you do?

2Build20+ eval cases, run in CI

Turn yesterday's 15 questions into 20+ eval cases, run them in CI, and fix the worst failure.

Done when: 20 or more tests score your app on every push, with a pass rate.

3CapstoneA pass rate, and the week 1 demo

A pass rate you can show. Week 1 check-in: a two-minute demo from every student.

4HomeworkProve a prompt change helped

Change one prompt and prove with the evals that it got better, not worse.

For teachers: the day 5 teaching kit.