Module 27

Finish the Transformer

Advanced module (after graduating): multi-head, position, LayerNorm, residual connection, block

When you finish this module you can say:

I can build a Transformer block piece by piece with numpy and check its shapes; I can show what pre-training and fine-tuning do in a small model.

By the end of this module you'll be able to show every piece inside a Transformer block in code and follow the shapes through a GPT's forward pass.

Lessons

6 lessons · ≈ 65 min

1. Multi-head attention11 min
2. Order information: position embeddings10 min
3. LayerNorm and the residual connection10 min
4. A block11 min
5. Pre-training and fine-tuning11 min
6. Project: a mini GPT and the Colab bridge12 min · project

Practice exercises

Short writing exercises, independent of the lessons: Go to the exercises

1. Whom does the last token look at?
2. The norm at the end of the stack
3. A small correction with LoRA