Unit 7: Transformers and word embeddings
About this unit
A transformer turns tokens into learned vectors, refines them through alternating attention and MLP blocks, and turns the last vector into a probability for every possible next token.
By the end, you can: Trace a transformer from tokens through embeddings, attention and MLP blocks to a softmax over the next token.
Modules
5 modulesLearn
- From text to vectorsArticle · Tokens, the embedding matrix and its size, directions as meaning, and the dot product.Not started
- Arithmetic on meaningsExperiment · Add and subtract word vectors in a small 3D embedding and see where analogies land.Not started
- Through the blocks to a predictionArticle · Attention and MLP blocks, the unembedding matrix, logits, softmax with temperature, and context size.Not started
- The whole pipelineReview · Every stage from tokens to probabilities in one place.Not started