Unit 9: How language models might store facts
About this unit
A transformer's MLP block asks each token vector questions with dot products, gates the answers with ReLU and writes directions back, and nearly perpendicular directions let it hold more features than dimensions.
By the end, you can: Trace a token vector through a transformer's MLP block and explain how superposition lets features outnumber dimensions.
Modules
5 modulesLearn
- Ask, gate, writeArticle · The MLP block's up-projection, bias, nonlinearity, down-projection and residual add, with a toy fact.Not started
- Try the gateExperiment · Compare ReLU and GELU on positive and negative inputs, and why the gate makes the block conditional.Not started
- More features than dimensionsArticle · Superposition: nearly perpendicular directions, crosstalk and sparse features.Not started
- The MLP block in briefReview · The block's steps, how it stores an association, and superposition.Not started