Week 1 · Day 5 of 10
Evals that tell the truth
Students can build a test set, score their app against it on every push, and prove a change made it better.
1LearnTest sets and evals in CI
Building a test set from real questions, exact checks versus model-graded checks, running the evals with pytest on every push in GitHub Actions, and reading the failing cases instead of trusting the score. From today, every change to the app is measured.
How evals prove your app works
- Test cases
Check your understanding
When should you use a model-graded check instead of an exact check?
Why run evals in CI on every push?
After a prompt change, your eval pass rate drops from 92% to 85%. What do you do?
2Build20+ eval cases, run in CI
Turn yesterday's 15 questions into 20+ eval cases, run them in CI, and fix the worst failure.
Done when: 20 or more tests score your app on every push, with a pass rate.
3CapstoneA pass rate, and the week 1 demo
A pass rate you can show. Week 1 check-in: a two-minute demo from every student.
4HomeworkProve a prompt change helped
Change one prompt and prove with the evals that it got better, not worse.
For teachers: the day 5 teaching kit.