Unit 8: Attention, step by step
About this unit
Attention lets each token score every earlier token by query–key dot products, softmax the scores into weights and add the weighted values to itself.
By the end, you can: Compute one attention head: query–key scores, scaling, masking, softmax weights and the weighted sum of values.
Modules
5 modulesLearn
- Words that need contextArticle · Why a token's vector must absorb its neighbours, and queries, keys and values in words.Not started
- One head, in six movesArticle · Project, score, scale by √d, mask, softmax and blend; heads in parallel.Not started
- Watch a pronoun chooseExperiment · Change one word and the temperature, and watch the attention weights shift.Not started
- Attention in briefReview · The whole mechanism and its cost in one place.Not started