From Attention-Augmented RNNs to Self-Attention
Attention Is All You Need
- Question first: what if every position could look directly at every other position in a single step, instead of passing information hop by hop?
- Convolutional alternatives from Section 2 (Extended Neural GPU, ByteNet, ConvS2S) compute positions in parallel, but the operations needed to relate two distant positions grow linearly (ConvS2S) or logarithmically (ByteNet) with distance
- The Transformer reduces this to a constant number of operations, at the cost of reduced effective resolution from averaging attention-weighted positions, which multi-head attention counteracts
- Definition of self-attention (intra-attention): relating different positions of a single sequence to compute a representation of that sequence
- The general attention abstraction from Section 3.2: map a query and a set of key-value pairs to an output, computed as a weighted sum of the values with weights from a compatibility function of query and key
- Intuition with a software analogy: a soft, differentiable lookup where the query is compared against every key and the result blends all values
- The paper's claim: the Transformer is the first transduction model relying entirely on self-attention, without sequence-aligned RNNs or convolution
مسیر ارتباط میان موقعیتها
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="tm-arrow" d="M 0 0 L 10 5 L 0 10 z"/></marker></defs><text class="tm-text" x="400" y="48" font-size="23" text-anchor="middle" direction="rtl">دو token با فاصلهٔ 10 موقعیت</text><circle class="tm-accent" cx="150" cy="120" r="18"/><circle class="tm-accent" cx="650" cy="120" r="18"/><text class="tm-text2" x="400" y="126" font-size="21" text-anchor="middle" direction="rtl">فاصلهٔ 10</text><text class="tm-text" x="100" y="210" font-size="20" text-anchor="middle">ConvS2S</text><path class="tm-line" d="M 210 204 L 290 204 L 290 184 L 370 184 L 370 204 L 450 204 L 450 184 L 530 184 L 530 204 L 610 204" marker-end="url(#arrow)"/><text class="tm-text-accent" x="685" y="210" font-size="21" text-anchor="middle">∝ 10</text><text class="tm-text" x="100" y="285" font-size="20" text-anchor="middle">ByteNet</text><path class="tm-line" d="M 210 279 L 330 279 L 330 259 L 490 259 L 490 279 L 610 279" marker-end="url(#arrow)"/><text class="tm-text-accent" x="685" y="285" font-size="21" text-anchor="middle">∝ log 10</text><text class="tm-muted" x="400" y="335" font-size="17" text-anchor="middle" direction="rtl">مقیاس رشد، نه شمارش دقیق عملیات</text><text class="tm-text" x="100" y="385" font-size="20" text-anchor="middle">Transformer</text><path class="tm-line-accent tm-flow" d="M 210 379 L 610 379" marker-end="url(#arrow)"/><text class="tm-text-accent" x="685" y="385" font-size="20" text-anchor="middle" direction="rtl">ثابت</text><text class="tm-muted" x="400" y="427" font-size="16" text-anchor="middle" direction="rtl">مسیر ثابت ≠ هزینهٔ کل ثابت</text></svg>
```
بهای میانگینگیری در attention
دو جفت مقدار با وزنهای برابر \(\frac{1}{2}\)
**جفت اول**
\[ (0,10),\quad (\tfrac12,\tfrac12) \]
**سهمها و حاصل**
\[ \tfrac12\cdot 0+\tfrac12\cdot 10=0+5=5 \]
**جفت دوم**
\[ (4,6),\quad (\tfrac12,\tfrac12) \]
**سهمها و حاصل**
\[ \tfrac12\cdot 4+\tfrac12\cdot 6=2+3=5 \]
**حاصل یکسان، ورودی متفاوت**
\[ (0,10)\neq(4,6),\qquad 5=5 \]
multi-head attention: مسیرهای توجه موازی
ارتباط موقعیتها در self-attention
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="tm-arrow" d="M 0 0 L 10 5 L 0 10 z"/></marker></defs><text class="tm-text" x="400" y="65" font-size="24" text-anchor="middle" direction="rtl">دنباله</text><rect class="tm-surface" x="105" y="100" width="160" height="65" rx="12"/><text class="tm-text" x="185" y="142" font-size="23" text-anchor="middle" direction="rtl">گربه</text><rect class="tm-surface" x="320" y="100" width="160" height="65" rx="12"/><text class="tm-text" x="400" y="142" font-size="23" text-anchor="middle" direction="rtl">کنار</text><rect class="tm-surface" x="535" y="100" width="160" height="65" rx="12"/><text class="tm-text" x="615" y="142" font-size="23" text-anchor="middle" direction="rtl">پنجره</text><path class="tm-line-accent" d="M 185 165 L 270 282" marker-end="url(#arrow)"/><path class="tm-line-accent" d="M 400 165 L 330 282" marker-end="url(#arrow)"/><path class="tm-line-accent" d="M 615 165 L 390 282" marker-end="url(#arrow)"/><rect class="tm-accent" x="180" y="285" width="270" height="65" rx="12"/><text class="tm-text" x="315" y="326" font-size="21" text-anchor="middle" direction="rtl">نمایش موقعیت گربه</text><text class="tm-text2" x="315" y="395" font-size="19" text-anchor="middle" direction="rtl">ساخت نمایش، نه جایگزینی کلمه</text><rect class="tm-soft tm-enter" x="490" y="260" width="270" height="115" rx="12"/><text class="tm-text" x="625" y="290" font-size="19" text-anchor="middle">Transformer</text><text class="tm-text2" x="625" y="320" font-size="17" text-anchor="middle" direction="rtl">نمایش ورودی و خروجی:</text><text class="tm-text2" x="625" y="349" font-size="17" text-anchor="middle" direction="rtl">self-attention، بدون RNN و convolution</text></svg>
```
lookup نرم با attention
query: گربه
keyها: گربه، پنجره | valueهای متناظر: \(10, 2\)
**وزنهای متناظر**
\[ (w_1,w_2)=(0.8,0.2) \]
**سهم value اول**
\[ 0.8\times 10=8 \]
**سهم value دوم**
\[ 0.2\times 2=0.4 \]
**خروجی ترکیبی**
\[ 8+0.4=8.4 \]
جمع وزندار هر دو value، نه بازیابی دقیق یک مقدار
Back to course