Module 28
Measuring an AI application
Advanced module (after graduation): evaluation set, scorers, model as judge, regressions, cost
When you finish this module you can say:
I can build an evaluation set for an AI application and measure it with scorers, catch examples that break when a prompt changes with tests, and report cost, time and failures next to accuracy.
By the end of this module you will be able to compare two versions of an AI application on an evaluation set and report accuracy, cost and regressions.
StartLessons
6 lessons · ≈ 62 min
- Eyeballing is not enough10 min
- Scorers10 min
- Model as judge10 min
- Regression tests10 min
- Cost, latency, errors10 min
- Project: measure the RAG assistant12 min · project
Practice exercises
Short writing exercises, independent of the lessons: Go to the exercises