Module 20

Inside a language model

Advanced module (after graduating): tokens, bigram, temperature, attention

When you finish this module you can say:

I can show in code how a language model produces text token by token (counting, softmax, temperature, attention).

By the end of this module you'll be able to train a small language model from scratch, have it generate new names with temperature, and explain in code what GPT's attention does.

Lessons

6 lessons · ≈ 62 min

1. Tokens: turning text into numbers8 min
2. Bigram: predicting by counting10 min
3. Softmax and temperature9 min
4. Predicting by learning: a one-layer network11 min
5. Attention: a single head, in numpy12 min
6. Project: a mini language model12 min · project