Three Uses of Attention and Decoder Masking
Attention Is All You Need
- Question first: during training the decoder sees the entire target sentence at once; what stops position 3 from peeking at the answer in position 4?
- Section 3.2.3: the Transformer uses multi-head attention in three ways
- Encoder self-attention: queries, keys, and values all come from the previous encoder layer; every position attends to all positions
- Decoder self-attention: each position attends to positions up to and including itself
- Encoder-decoder attention: queries from the previous decoder layer, keys and values (the memory) from the encoder output, so every decoder position attends over the whole input
- Masking: set softmax inputs for illegal connections to negative infinity so their weights become exactly 0 (the optional Mask block in Figure 2 left)
- Hand example: apply a lower-triangular mask to a 3x3 score matrix and recompute softmax
- Why the mask plus offsetting output embeddings by one position preserves the auto-regressive property
جابهجایی ورودی و causal mask
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><text class="tm-text" x="400" y="45" font-size="23" text-anchor="middle" direction="rtl">پیشبینی هدف</text><rect class="tm-surface" x="155" y="85" width="130" height="64" rx="10"/><rect class="tm-surface" x="335" y="85" width="130" height="64" rx="10"/><rect class="tm-surface" x="515" y="85" width="130" height="64" rx="10"/><text class="tm-text" x="220" y="125" font-size="23" text-anchor="middle" direction="rtl">الف</text><text class="tm-text" x="400" y="125" font-size="23" text-anchor="middle" direction="rtl">ب</text><text class="tm-text" x="580" y="125" font-size="23" text-anchor="middle" direction="rtl">پ</text><text class="tm-text" x="400" y="215" font-size="23" text-anchor="middle" direction="rtl">ورودی decoder پس از جابهجایی</text><rect class="tm-soft" x="155" y="245" width="130" height="64" rx="10"/><rect class="tm-soft" x="335" y="245" width="130" height="64" rx="10"/><rect class="tm-soft" x="515" y="245" width="130" height="64" rx="10"/><text class="tm-text" x="220" y="285" font-size="21" text-anchor="middle" direction="rtl">نشانه شروع</text><text class="tm-text" x="400" y="285" font-size="23" text-anchor="middle" direction="rtl">الف</text><text class="tm-text" x="580" y="285" font-size="23" text-anchor="middle" direction="rtl">ب</text><path class="tm-line-accent" d="M220 325 V345 H580 V325" fill="none"/><text class="tm-text-accent" x="400" y="380" font-size="20" text-anchor="middle" direction="rtl">دسترسی جایگاه سوم: گذشته و خودش</text><path class="tm-line-accent" d="M680 230 V330" fill="none"/><text class="tm-text-accent" x="710" y="355" font-size="18" text-anchor="middle">mask</text><text class="tm-text2" x="710" y="390" font-size="18" text-anchor="middle" direction="rtl">آینده ممنوع</text></svg>
```
دسترسی کامل در encoder self-attention
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path class="tm-arrow" d="M 0 0 L 10 5 L 0 10 z"/></marker></defs><rect class="tm-surface" x="265" y="20" width="270" height="50" rx="12"/><text class="tm-text" x="400" y="51" font-size="19" text-anchor="middle" direction="rtl">خروجی لایه قبلی encoder</text><path class="tm-line" d="M 400 70 L 400 90 L 180 90 L 180 105" fill="none" marker-end="url(#arrow)"/><path class="tm-line" d="M 400 70 L 400 105" fill="none" marker-end="url(#arrow)"/><path class="tm-line" d="M 400 70 L 400 90 L 620 90 L 620 105" fill="none" marker-end="url(#arrow)"/><rect class="tm-soft" x="110" y="110" width="140" height="42" rx="10"/><rect class="tm-soft" x="330" y="110" width="140" height="42" rx="10"/><rect class="tm-soft" x="550" y="110" width="140" height="42" rx="10"/><text class="tm-text" x="180" y="137" font-size="20" text-anchor="middle">query</text><text class="tm-text" x="400" y="137" font-size="20" text-anchor="middle">key</text><text class="tm-text" x="620" y="137" font-size="20" text-anchor="middle">value</text><text class="tm-text" x="400" y="185" font-size="19" text-anchor="middle" direction="rtl">ماتریس امتیاز: \(6\times 6\)</text><text class="tm-muted" x="400" y="213" font-size="16" text-anchor="middle" direction="rtl">ستونهای key</text><text class="tm-muted" x="220" y="315" font-size="16" text-anchor="middle" direction="rtl">سطرهای query</text><rect class="tm-soft" x="310" y="225" width="180" height="180" rx="4"/><rect class="tm-accent" x="310" y="285" width="180" height="30"/><path class="tm-line" d="M 340 225 V 405 M 370 225 V 405 M 400 225 V 405 M 430 225 V 405 M 460 225 V 405 M 310 255 H 490 M 310 285 H 490 M 310 315 H 490 M 310 345 H 490 M 310 375 H 490" fill="none"/><text class="tm-text" x="325" y="305" font-size="16" text-anchor="middle">\(1\)</text><text class="tm-text" x="385" y="305" font-size="16" text-anchor="middle">\(3\)</text><text class="tm-text" x="475" y="305" font-size="16" text-anchor="middle">\(6\)</text><text class="tm-text-accent" x="535" y="305" font-size="17">\(3\)</text><text class="tm-text" x="610" y="350" font-size="18" text-anchor="middle" direction="rtl">بدون causal mask</text></svg>
```
دسترسی در decoder self-attention
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><rect class="tm-surface" x="55" y="75" width="230" height="85" rx="12"/><text class="tm-text" x="170" y="112" text-anchor="middle" font-size="20" direction="rtl">ورودی لایه decoder</text><text class="tm-text2" x="170" y="142" text-anchor="middle" font-size="19" direction="rtl">Q، K، V</text><text class="tm-text" x="530" y="92" text-anchor="middle" font-size="22" direction="rtl">ماتریس امتیاز</text><text class="tm-text2" x="530" y="128" text-anchor="middle" font-size="19">K →</text><text class="tm-text2" x="346" y="250" text-anchor="middle" font-size="19">Q ↓</text><text class="tm-muted" x="410" y="154" text-anchor="middle" font-size="16">1</text><text class="tm-muted" x="470" y="154" text-anchor="middle" font-size="16">2</text><text class="tm-muted" x="530" y="154" text-anchor="middle" font-size="16">3</text><text class="tm-muted" x="590" y="154" text-anchor="middle" font-size="16">4</text><text class="tm-muted" x="370" y="195" text-anchor="middle" font-size="16">1</text><text class="tm-muted" x="370" y="255" text-anchor="middle" font-size="16">2</text><text class="tm-muted" x="370" y="315" text-anchor="middle" font-size="16">3</text><text class="tm-muted" x="370" y="375" text-anchor="middle" font-size="16">4</text><rect class="tm-soft" x="380" y="165" width="60" height="60"/><rect class="tm-soft" x="440" y="165" width="60" height="60"/><rect class="tm-soft" x="500" y="165" width="60" height="60"/><rect class="tm-soft" x="560" y="165" width="60" height="60"/><rect class="tm-soft" x="380" y="225" width="60" height="60"/><rect class="tm-soft" x="440" y="225" width="60" height="60"/><rect class="tm-soft" x="500" y="225" width="60" height="60"/><rect class="tm-soft" x="560" y="225" width="60" height="60"/><rect class="tm-soft" x="380" y="285" width="60" height="60"/><rect class="tm-soft" x="440" y="285" width="60" height="60"/><rect class="tm-soft" x="500" y="285" width="60" height="60"/><rect class="tm-soft" x="560" y="285" width="60" height="60"/><rect class="tm-soft" x="380" y="345" width="60" height="60"/><rect class="tm-soft" x="440" y="345" width="60" height="60"/><rect class="tm-soft" x="500" y="345" width="60" height="60"/><rect class="tm-soft" x="560" y="345" width="60" height="60"/><text class="tm-text" x="170" y="218" text-anchor="middle" font-size="20">4 query × 4 key</text><text class="tm-text-accent" x="170" y="254" text-anchor="middle" font-size="24">4 × 4</text><rect class="tm-green" x="383" y="168" width="54" height="54"/><rect class="tm-green" x="383" y="228" width="54" height="54"/><rect class="tm-green" x="443" y="228" width="54" height="54"/><rect class="tm-green" x="383" y="288" width="54" height="54"/><rect class="tm-green" x="443" y="288" width="54" height="54"/><rect class="tm-green" x="503" y="288" width="54" height="54"/><rect class="tm-green" x="383" y="348" width="54" height="54"/><rect class="tm-green" x="443" y="348" width="54" height="54"/><rect class="tm-green" x="503" y="348" width="54" height="54"/><rect class="tm-green" x="563" y="348" width="54" height="54"/><text class="tm-text2" x="470" y="202" text-anchor="middle" font-size="24">×</text><text class="tm-text2" x="530" y="202" text-anchor="middle" font-size="24">×</text><text class="tm-text2" x="590" y="202" text-anchor="middle" font-size="24">×</text><text class="tm-text2" x="530" y="262" text-anchor="middle" font-size="24">×</text><text class="tm-text2" x="590" y="262" text-anchor="middle" font-size="24">×</text><text class="tm-text2" x="590" y="322" text-anchor="middle" font-size="24">×</text><text class="tm-text-green" x="170" y="311" text-anchor="middle" font-size="18" direction="rtl">مجاز: خود و گذشته</text><text class="tm-text2" x="170" y="343" text-anchor="middle" font-size="18" direction="rtl">× غیرمجاز: آینده</text><text class="tm-text-accent" x="650" y="320" font-size="18" direction="rtl">← query سوم</text></svg>
```
جهت اطلاعات در encoder-decoder attention
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="8" markerHeight="8" orient="auto"><path class="tm-arrow" d="M 0 0 L 10 5 L 0 10 z"/></marker></defs><rect class="tm-soft" x="35" y="45" width="215" height="100" rx="14"/><text class="tm-text" x="142" y="83" font-size="21" text-anchor="middle">decoder</text><text class="tm-text2" x="142" y="115" font-size="18" text-anchor="middle" direction="rtl">لایه قبلی</text><rect class="tm-surface" x="305" y="60" width="190" height="70" rx="14"/><text class="tm-text" x="400" y="101" font-size="20" text-anchor="middle">attention</text><line class="tm-line-accent" x1="250" y1="95" x2="305" y2="95" marker-end="url(#arrow)"/><text class="tm-text-accent" x="277" y="80" font-size="19" text-anchor="middle">Q</text><rect class="tm-soft" x="550" y="45" width="215" height="100" rx="14"/><text class="tm-text" x="657" y="83" font-size="21" text-anchor="middle">encoder</text><text class="tm-text2" x="657" y="115" font-size="18" text-anchor="middle" direction="rtl">خروجی · memory</text><line class="tm-line-accent" x1="550" y1="95" x2="495" y2="95" marker-end="url(#arrow)"/><text class="tm-text-accent" x="522" y="80" font-size="19" text-anchor="middle">K, V</text><text class="tm-text" x="400" y="197" font-size="20" text-anchor="middle" direction="rtl">ماتریس امتیاز: \(4 \times 6\)</text><text class="tm-text2" x="400" y="225" font-size="17" text-anchor="middle" direction="rtl">ستونها: جایگاههای encoder</text><text class="tm-text2" x="170" y="315" font-size="17" text-anchor="middle" direction="rtl">سطرها: decoder</text><rect class="tm-soft" x="265" y="240" width="270" height="160" rx="8"/><rect class="tm-accent" x="265" y="320" width="270" height="40"/><path class="tm-line" d="M310 240 V400 M355 240 V400 M400 240 V400 M445 240 V400 M490 240 V400 M265 280 H535 M265 320 H535 M265 360 H535"/><text class="tm-text-accent" x="610" y="345" font-size="17" text-anchor="middle" direction="rtl">query جایگاه سوم</text><text class="tm-text2" x="400" y="430" font-size="17" text-anchor="middle" direction="rtl">هر شش جایگاه ورودی، بدون causal mask</text></svg>
```
اثر causal mask بر softmax
ماتریس امتیاز فرضی: \(S=\begin{pmatrix}0&0&0\\0&0&0\\0&0&0\end{pmatrix}\)
**پیش از softmax**
\[ S_{\mathrm{masked}}=\begin{pmatrix}0&-\infty&-\infty\\0&0&-\infty\\0&0&0\end{pmatrix} \]
**نمایی امتیازها**
\[ e^0=1,\qquad e^{-\infty}=0 \]
**سطر اول**
\[ \frac{(1,0,0)}{1}=(1,0,0) \]
**سطر دوم**
\[ \frac{(1,1,0)}{2}=\left(\frac12,\frac12,0\right) \]
**سطر سوم**
\[ \frac{(1,1,1)}{3}=\left(\frac13,\frac13,\frac13\right) \]
**صفر کردن پس از softmax**
\[ \left(\frac13,\frac13,\frac13\right)\longrightarrow\left(\frac13,0,0\right),\qquad \sum_j w_j=\frac13\ne 1 \]
مجموع وزنها دیگر یک نیست.
Back to course