Attention Is All You Need
این دوره بازخوانی دقیق، خط به خط و وفادارانه مقاله تاریخی «Attention Is All You Need» است؛ مقالهای که در سال ۲۰۱۷ پایه تمام مدلهای زبانی بزرگ مدرن را بنا نهاد. در این مسیر آموزشی، استدلالهای مقاله را به همان ترتیب نویسندگان مرور میکنید: از چرایی محدودیت مدلهای بازگشتی (RNN) و لزوم جایگزینی آنها با مکانیزم توجه، تا جزئیات ریاضی محاسبات ضرب داخلی مقیاسپذیر، توجه چندسر، کدگذاری موقعیت و بلوکهای انکودر-دیکودر.
آموزش در قالب گامهای کوتاه و پیوسته طراحی شده است که در هر گام ابتدا با یک پرسش یا پیشبینی فکری روبهرو میشوید، سپس روابط ریاضی و تانسورها را گامبهگام با هدایت تیوتور هوش مصنوعی محاسبه و تحلیل میکنید و با حل آزمکهای مفهومی تسلط خود را میسنجید.
با اتمام این دوره، نهتنها چرایی تمام تصمیمات طراحی در معماری ترنسفورمر و مصالحههای محاسباتی آن مانند هزینه مرتبه دو در طول دنباله را درک خواهید کرد، بلکه قادر خواهید بود ابعاد تانسورها را در طول لایهها دنبال کرده و یک نمونه توجه کوچک را روی کاغذ محاسبه کنید.
زمان تقریبی یادگیری: 11 ساعت
آنچه یاد میگیرید
- محاسبه فرمول Scaled Dot-Product Attention و وزندهی به ماتریسهای پرسش، کلید و مقدار بهصورت دستی
- ردگیری ابعاد تانسورها و ماتریسهای وزن در تمام زیرلایههای انکودر و دیکودر
- تحلیل ریاضی ضریب مقیاسبندی و چرایی اشباع تابع Softmax در ابعاد بالا
- تبیین ضرورت Positional Encoding و تزریق اطلاعات ترتیب به ماتریسهای ورودی
- مقایسه پیچیدگی محاسباتی و طول مسیر وابستگی در لایههای خودتوجهی در مقایسه با شبکههای بازگشتی
سرفصلهای دوره
- مقدمه دوره: مقاله Attention Is All You Need
- Opening question: the language models you use every day all trace back to one 2017 paper. What problem were its authors trying to solve?
- What this course is: a slow, faithful walk through a single research paper, in the paper's own order, using its own figures and tables
- The big story in plain words: older translation models read a sentence one word at a time, which made them slow to train; this paper replaced that step-by-step reading with a way for every word to look at every other word at once
- How the pieces connect: a way of letting words look at each other, a way of telling the model where each word sits in the sentence, and a stack of repeated building blocks that turns one language into another
- Roadmap: Module 1 asks why the old approach was a bottleneck; Module 2 builds the core mechanism and computes it by hand; Module 3 assembles the full model and traces the data through it; Module 4 weighs the costs and benefits of the new design; Module 5 covers how it was trained, what results it achieved, and which parts mattered most
- What you will be able to do at the end: work a small example by hand, follow the sizes of the data through one block of the model, and explain the trade-offs the authors made
- گلوگاه ترتیبی مدلهای بازگشتی (RNN)
- Prediction prompt: if computing state h_t requires h_{t-1}, how many of the n steps for one sentence can run at the same time on a GPU?
- The sequence transduction setting from Section 3: an encoder maps input symbols (x_1, ..., x_n) to continuous representations z = (z_1, ..., z_n); a decoder generates (y_1, ..., y_m) one symbol at a time
- Auto-regressive generation: each output step consumes the previously generated symbols as additional input
- How recurrent models factor computation along positions: h_t is a function of h_{t-1} and the input at position t (Section 1)
- Why this precludes parallelization within a training example, and why it hurts more at longer sequence lengths when memory limits batching across examples
- Factorization tricks and conditional computation improved efficiency, but the paper states the fundamental constraint of sequential computation remains
- Attention was already used to model dependencies regardless of distance, but almost always together with a recurrent network
- از RNN همراه با Attention تا Self-Attention
- Question first: what if every position could look directly at every other position in a single step, instead of passing information hop by hop?
- Convolutional alternatives from Section 2 (Extended Neural GPU, ByteNet, ConvS2S) compute positions in parallel, but the operations needed to relate two distant positions grow linearly (ConvS2S) or logarithmically (ByteNet) with distance
- The Transformer reduces this to a constant number of operations, at the cost of reduced effective resolution from averaging attention-weighted positions, which multi-head attention counteracts
- Definition of self-attention (intra-attention): relating different positions of a single sequence to compute a representation of that sequence
- The general attention abstraction from Section 3.2: map a query and a set of key-value pairs to an output, computed as a weighted sum of the values with weights from a compatibility function of query and key
- Intuition with a software analogy: a soft, differentiable lookup where the query is compared against every key and the result blends all values
- The paper's claim: the Transformer is the first transduction model relying entirely on self-attention, without sequence-aligned RNNs or convolution
- Scaled Dot-Product Attention: کوئری، کلید و مقدار
- Prediction prompt: given one query vector and three key vectors, which key will get the largest weight, and what happens to the output if two keys are equally similar?
- Inputs: queries and keys of dimension d_k, values of dimension d_v (Section 3.2.1, Figure 2 left)
- Step by step: dot products of the query with all keys, divide each by sqrt(d_k), apply softmax to get weights, take the weighted sum of values
- Matrix form, Equation (1): Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V
- Shapes: Q is (n_q x d_k), K is (n_k x d_k), V is (n_k x d_v), Q K^T is (n_q x n_k), output is (n_q x d_v)
- Softmax is applied row-wise: each query's weights over all keys sum to 1
- Worked hand example with 2 to 3 tokens and d_k = 4, computing every intermediate number
- Comparison with additive attention: similar theoretical complexity, but dot-product attention is much faster and more space-efficient because it uses optimized matrix multiplication
- چرا مقیاسدهی با 1/sqrt(d_k)؟
- Question first: if you double the length of two random vectors, what happens to the spread of their dot product, and what does that do to softmax?
- The empirical observation cited in Section 3.2.1: for small d_k additive and dot-product attention perform similarly, but additive outperforms unscaled dot-product attention for larger d_k
- The paper's hypothesis: large dot products push softmax into regions with extremely small gradients
- Footnote 4 argument: if components of q and k are independent with mean 0 and variance 1, then q · k has mean 0 and variance d_k
- Dividing by sqrt(d_k) brings the variance back to 1
- Numerical demonstration: softmax of [1, 2, 3] versus softmax of [8, 16, 24] and how the second becomes nearly one-hot
- Connection to the scaling factor being the only difference between the paper's attention and standard dot-product attention
- Multi-Head Attention
- Prediction prompt: a single attention distribution averages over positions; what information might get lost if one word needs to attend to its subject and its object at the same time?
- Section 3.2.2 and Figure 2 right: project Q, K, V h times with different learned linear projections to d_k, d_k, d_v dimensions, run attention in parallel, concatenate, project again
- Formulas: MultiHead(Q, K, V) = Concat(head_1, ..., head_h) W^O, where head_i = Attention(Q W_i^Q, K W_i^K, V W_i^V)
- Parameter shapes: W_i^Q and W_i^K in R^(d_model x d_k), W_i^V in R^(d_model x d_v), W^O in R^(h·d_v x d_model)
- Paper settings: h = 8 heads, d_k = d_v = d_model / h = 64
- Shape trace for n tokens: input (n x 512) -> each head's Q, K, V (n x 64) -> scores (n x n) -> head output (n x 64) -> concat (n x 512) -> output (n x 512)
- Why total cost is similar to single-head attention with full dimensionality: each head works in a reduced dimension
- The paper's rationale: jointly attending to information from different representation subspaces at different positions, which a single head inhibits by averaging
- سه کاربرد Attention و Masking در دیکودر
- Question first: during training the decoder sees the entire target sentence at once; what stops position 3 from peeking at the answer in position 4?
- Section 3.2.3: the Transformer uses multi-head attention in three ways
- Encoder self-attention: queries, keys, and values all come from the previous encoder layer; every position attends to all positions
- Decoder self-attention: each position attends to positions up to and including itself
- Encoder-decoder attention: queries from the previous decoder layer, keys and values (the memory) from the encoder output, so every decoder position attends over the whole input
- Masking: set softmax inputs for illegal connections to negative infinity so their weights become exactly 0 (the optional Mask block in Figure 2 left)
- Hand example: apply a lower-triangular mask to a 3x3 score matrix and recompute softmax
- Why the mask plus offsetting output embeddings by one position preserves the auto-regressive property
- اتصال Residual، Layer Normalization و شبکه Feed-Forward
- Prediction prompt: attention mixes information across positions; what component do you think processes each position on its own?
- Sub-layer wrapper from Section 3.1: output = LayerNorm(x + Sublayer(x))
- Why every sub-layer and embedding outputs dimension d_model = 512: so residual additions are shape-compatible
- Position-wise feed-forward network, Equation (2): FFN(x) = max(0, x W_1 + b_1) W_2 + b_2
- Dimensions: input and output 512, inner layer d_ff = 2048; weight shapes W_1 (512 x 2048) and W_2 (2048 x 512)
- Same parameters across positions within a layer, different parameters from layer to layer; equivalent to two convolutions with kernel size 1
- Division of labor: attention moves information between positions, the FFN transforms each position independently
- Shape trace: (n x 512) -> (n x 2048) -> (n x 512)
- Positional Encoding
- Question first: if you shuffle the words of a sentence, does the output of plain self-attention for each word change? What does that imply?
- Why order must be injected: no recurrence and no convolution means attention alone has no notion of position (Section 3.5)
- Positional encodings have dimension d_model so they can be added to input embeddings at the bottom of encoder and decoder stacks
- Formulas: PE(pos, 2i) = sin(pos / 10000^(2i/d_model)), PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
- Each dimension is a sinusoid; wavelengths form a geometric progression from 2π to 10000 · 2π
- Hand computation: PE values for pos = 0, 1, 2 at i = 0 and a higher i, with d_model = 512
- The relative-position hypothesis: for any fixed offset k, PE(pos+k) is a linear function of PE(pos)
- Learned positional embeddings gave nearly identical results (Table 3 row E); sinusoids chosen because they may extrapolate to longer sequences than seen in training
- ساختار کامل انکودر و دیکودر
- Prediction prompt: looking at Figure 1, how many attention sub-layers does one decoder layer have, and why one more than the encoder?
- Encoder: N = 6 identical layers, each with multi-head self-attention then the position-wise FFN, each wrapped with residual and LayerNorm
- Decoder: N = 6 identical layers with masked self-attention, encoder-decoder attention over the encoder output, and the FFN
- Embeddings (Section 3.4): learned embeddings to d_model; the same weight matrix shared between the two embedding layers and the pre-softmax linear transformation; embedding weights multiplied by sqrt(d_model)
- Output: linear transformation and softmax produce next-token probabilities
- Output embeddings are offset by one position so prediction i depends only on outputs before i
- Complete shape trace through one encoder layer and one decoder layer for a source of n tokens and a target of m tokens
- Walking Figure 1 bottom to top: inputs, positional encoding, N encoder layers, decoder layers, linear, softmax
- چرا Self-Attention: پیچیدگی، موازیسازی و طول مسیر
- Question first: for a sentence of n = 50 tokens and d = 512, which is cheaper per layer, self-attention or a recurrent layer? What about n = 10,000?
- The three desiderata of Section 4: total computational complexity per layer, minimum number of sequential operations, maximum path length between long-range dependencies
- Table 1: Self-Attention O(n^2 · d), O(1), O(1); Recurrent O(n · d^2), O(n), O(n); Convolutional O(k · n · d^2), O(1), O(log_k(n)); Self-Attention (restricted) O(r · n · d), O(1), O(n/r)
- Where n^2 · d comes from: the n x n score matrix, each entry a d-dimensional dot product
- Self-attention is faster than recurrence when n < d, which the paper notes is most often the case for word-piece and byte-pair sentence representations
- The quadratic trade-off: cost grows with the square of sequence length; the paper's proposed remedy is restricted attention over a neighborhood of size r, raising maximum path length to O(n/r)
- Convolutions: a single layer with k < n does not connect all positions; separable convolutions cost O(k · n · d + n · d^2), equal to self-attention plus FFN even when k = n
- Side benefit: interpretability; the appendix figures show different heads learning different tasks, such as resolving which noun a pronoun refers to (Figure 4) and sentence-structure patterns (Figure 5)
- داده آموزشی، Batching، سختافزار و زمانبندی نرخ یادگیری
- Prediction prompt: why might a learning rate that starts at zero, climbs, and then decays help a freshly initialized deep attention model?
- Data (Section 5.1): WMT 2014 English-German, about 4.5 million sentence pairs, byte-pair encoding with a shared source-target vocabulary of about 37000 tokens; WMT 2014 English-French, 36M sentences, 32000 word-piece vocabulary
- Batching by approximate sequence length; each batch has approximately 25000 source tokens and 25000 target tokens
- Hardware and schedule (Section 5.2): one machine with 8 NVIDIA P100 GPUs; base model about 0.4 seconds per step, 100,000 steps or 12 hours; big model 1.0 seconds per step, 300,000 steps (3.5 days)
- Optimizer (Section 5.3): Adam with beta_1 = 0.9, beta_2 = 0.98, epsilon = 10^-9
- Equation (3): lrate = d_model^-0.5 · min(step_num^-0.5, step_num · warmup_steps^-1.5), with warmup_steps = 4000
- Reading the formula: linear increase for the first 4000 steps, then decay proportional to the inverse square root of the step number
- Hand computation of lrate at steps 1000, 4000, and 16000 for d_model = 512
- Regularization: Dropout و Label Smoothing
- Question first: why would deliberately making a model less confident in the correct answer improve its translations?
- Residual dropout (Section 5.4): applied to the output of each sub-layer before it is added to the sub-layer input and normalized
- Dropout also applied to the sums of embeddings and positional encodings in both stacks
- Base model rate P_drop = 0.1
- Label smoothing with epsilon_ls = 0.1: the target distribution puts slightly less than full probability on the correct token
- The paper's stated trade-off: label smoothing hurts perplexity because the model learns to be more unsure, but improves accuracy and BLEU
- Locating exactly where dropout sits in the LayerNorm(x + Dropout(Sublayer(x))) path
- نتایج ترجمه ماشینی
- Prediction prompt: before looking at Table 2, would you expect the smaller base model to beat earlier ensembles? Commit to a guess
- Inference setup (Section 6.1): base models average the last 5 checkpoints written at 10-minute intervals; big models average the last 20; beam search with beam size 4 and length penalty alpha = 0.6; maximum output length input length + 50 with early termination
- Table 2 English-to-German: Transformer (big) 28.4 BLEU, more than 2.0 BLEU above the best previously reported models including ensembles; Transformer (base) 27.3 surpasses all previously published models and ensembles
- Table 2 English-to-French: Transformer (base) 38.1; Transformer (big) 41.8 as reported in the abstract and Table 2 (the Section 6.1 text reads 41.0; read numbers from the table), a new single-model state of the art at less than 1/4 the training cost of the previous state of the art
- The big English-to-French model used P_drop = 0.1 instead of 0.3
- How training cost was estimated: training time x number of GPUs x estimated sustained single-precision capacity per GPU (9.5 TFLOPS for P100 per footnote 5)
- Reading Table 2 critically: quality versus training cost across model families
- آزمایشهای Ablation: تغییرات مدل در جدول ۳
- Question first: if you keep total computation fixed, is one wide head better or worse than many narrow heads? Predict, then check rows (A)
- Setup (Section 6.2): vary the base model, measure English-to-German on newstest2013 development set, beam search but no checkpoint averaging; perplexities are per-wordpiece
- Base reference row: N = 6, d_model = 512, d_ff = 2048, h = 8, d_k = d_v = 64, P_drop = 0.1, epsilon_ls = 0.1, 100K steps, PPL 4.92, BLEU 25.8, 65 x 10^6 params
- Rows (A): varying heads at constant computation; single-head attention is 0.9 BLEU worse than the best setting, and quality also drops with too many heads
- Rows (B): reducing d_k hurts quality, suggesting compatibility is not easy to determine and a more sophisticated function than dot product may help
- Rows (C) and (D): bigger models are better; dropout is very helpful against over-fitting
- Row (E): learned positional embeddings instead of sinusoids give nearly identical results (25.7 BLEU)
- Big row: N = 6, d_model = 1024, d_ff = 4096, h = 16, P_drop = 0.3, 300K steps, PPL 4.33, BLEU 26.4, 213 x 10^6 params
- تعمیم به Constituency Parsing و جمعبندی مقاله
- Prediction prompt: parsing outputs are long and tightly structured; would a translation architecture with almost no task-specific tuning do well?
- Setup (Section 6.3): a 4-layer Transformer with d_model = 1024 on the WSJ portion of the Penn Treebank, about 40K training sentences; semi-supervised setting with approximately 17M sentences; vocabularies of 16K and 32K tokens
- Inference changes: maximum output length input length + 300, beam size 21, alpha = 0.3
- Table 4: 91.3 F1 trained on WSJ only and 92.7 F1 semi-supervised, better than all previously reported models except the Recurrent Neural Network Grammar
- Unlike RNN sequence-to-sequence models, the Transformer outperforms the BerkeleyParser even with only the 40K-sentence WSJ training set
- Conclusion (Section 7): first transduction model based entirely on attention; trains significantly faster; new state of the art on both WMT 2014 tasks
- Future directions stated by the authors: other modalities such as images, audio and video, local restricted attention for large inputs, and less sequential generation
- One pointer after the paper: the encoder and decoder stacks you traced here were later used on their own to build the language models you use daily
شروع این دوره در تیچمی