Regularization: Dropout و Label Smoothing
Attention Is All You Need
- Question first: why would deliberately making a model less confident in the correct answer improve its translations?
- Residual dropout (Section 5.4): applied to the output of each sub-layer before it is added to the sub-layer input and normalized
- Dropout also applied to the sums of embeddings and positional encodings in both stacks
- Base model rate P_drop = 0.1
- Label smoothing with epsilon_ls = 0.1: the target distribution puts slightly less than full probability on the correct token
- The paper's stated trade-off: label smoothing hurts perplexity because the model learns to be more unsure, but improves accuracy and BLEU
- Locating exactly where dropout sits in the LayerNorm(x + Dropout(Sublayer(x))) path
از اطمینان تا ترجمه بهتر
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arrow" markerWidth="10" markerHeight="10" refX="8" refY="5" orient="auto"><path d="M0,0 L10,5 L0,10 z" class="tm-arrow"/></marker></defs><rect x="70" y="100" width="270" height="105" rx="18" class="tm-soft"/><text x="205" y="145" class="tm-text" font-size="22" text-anchor="middle" direction="rtl">اطمینان کمتر</text><text x="205" y="178" class="tm-muted" font-size="17" text-anchor="middle" direction="rtl">در آموزش</text><path d="M350 152 H450" class="tm-line-accent" marker-end="url(#arrow)"/><rect x="465" y="100" width="270" height="105" rx="18" class="tm-green"/><text x="600" y="145" class="tm-text" font-size="22" text-anchor="middle" direction="rtl">ترجمه بهتر</text><text x="600" y="178" class="tm-text-green" font-size="17" text-anchor="middle" direction="rtl">ممکن است</text><text x="400" y="285" class="tm-muted" font-size="20" text-anchor="middle">regularization</text></svg>
```
جایگاه dropout در مسیر residual
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arrow" markerWidth="10" markerHeight="10" refX="8" refY="5" orient="auto"><path d="M0,0 L10,5 L0,10 z" class="tm-arrow"/></marker></defs><rect x="35" y="190" width="90" height="54" rx="10" class="tm-surface"/><text x="80" y="224" text-anchor="middle" class="tm-text" font-size="20">x</text><path d="M125 217 H220" class="tm-line" marker-end="url(#arrow)"/><rect x="220" y="190" width="145" height="54" rx="10" class="tm-surface"/><text x="292" y="224" text-anchor="middle" class="tm-text" font-size="18">Sublayer</text><path d="M365 217 H450" class="tm-line" marker-end="url(#arrow)"/><rect x="450" y="190" width="120" height="54" rx="10" class="tm-surface"/><text x="510" y="224" text-anchor="middle" class="tm-text" font-size="18">Dropout</text><path d="M125 217 V345 H625 V272" class="tm-line" marker-end="url(#arrow)"/><path d="M570 217 H625 V272" class="tm-line" marker-end="url(#arrow)"/><rect x="585" y="272" width="80" height="54" rx="10" class="tm-surface"/><text x="625" y="306" text-anchor="middle" class="tm-text" font-size="20">+</text><path d="M665 299 H710" class="tm-line" marker-end="url(#arrow)"/><rect x="710" y="272" width="80" height="54" rx="10" class="tm-accent tm-pulse"/><text x="750" y="306" text-anchor="middle" class="tm-text" font-size="14">LayerNorm</text><text x="625" y="380" text-anchor="middle" class="tm-text2" font-size="17" direction="rtl">۲ + ۰ = ۲</text><text x="400" y="420" text-anchor="middle" class="tm-muted" font-size="15" direction="rtl">جمع کل بردار → LayerNorm</text></svg>
```
محل دوم dropout در دو stack
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arrow" markerWidth="10" markerHeight="10" refX="8" refY="5" orient="auto"><path d="M0,0 L10,5 L0,10 z" class="tm-arrow"/></marker></defs><text x="400" y="40" class="tm-text" font-size="22" text-anchor="middle" direction="rtl">نرخ dropout</text><text x="400" y="85" class="tm-text2" font-size="17" text-anchor="middle">base: 0.1</text><text x="400" y="245" class="tm-text2" font-size="17" text-anchor="middle" direction="rtl">big: 0.3؛ تنظیم مشخص ترجمه: 0.1</text><rect x="120" y="125" width="560" height="52" rx="10" class="tm-accent"/><text x="400" y="158" class="tm-text" font-size="18" text-anchor="middle" direction="rtl">احتمال حذف هر مؤلفه در آموزش: 1/10</text></svg>
```
هدف نرم، اطمینان کمتر
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><text x="400" y="52" class="tm-text" font-size="22" text-anchor="middle" direction="rtl">هدف سخت</text><text x="235" y="112" class="tm-text2" font-size="18" text-anchor="middle" direction="rtl">token درست</text><text x="565" y="112" class="tm-text2" font-size="18" text-anchor="middle" direction="rtl">گزینهٔ دیگر</text><rect x="145" y="145" width="180" height="112" rx="14" class="tm-accent"/><text x="235" y="212" class="tm-text" font-size="30" text-anchor="middle">\(1\)</text><rect x="475" y="145" width="180" height="112" rx="14" class="tm-soft"/><text x="565" y="212" class="tm-text" font-size="30" text-anchor="middle">\(0\)</text><text x="400" y="315" class="tm-muted" font-size="18" text-anchor="middle" direction="rtl">فقط گزینهٔ درست</text><text x="400" y="365" class="tm-text-accent" font-size="20" text-anchor="middle">label smoothing: \(\epsilon_{ls}=0.1\)</text><rect x="145" y="145" width="180" height="112" rx="14" class="tm-green"/><text x="235" y="212" class="tm-text" font-size="30" text-anchor="middle">\(0.9\)</text><rect x="475" y="145" width="180" height="112" rx="14" class="tm-gold"/><text x="565" y="212" class="tm-text" font-size="30" text-anchor="middle">\(0.1\)</text><text x="400" y="415" class="tm-text-green" font-size="18" text-anchor="middle" direction="rtl">مجموع: \(1\) • مثال فرضی با دو token</text></svg>
```
اطمینان، احتمال و کیفیت ترجمه
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><rect class="tm-soft" x="45" y="45" width="330" height="155" rx="16"/><text class="tm-text" x="210" y="78" font-size="20" text-anchor="middle" direction="rtl">یک token درست</text><text class="tm-muted" x="210" y="112" font-size="16" text-anchor="middle" direction="rtl">مدل اول</text><rect class="tm-accent" x="95" y="130" width="180" height="42" rx="10"/><text class="tm-text-accent" x="185" y="158" font-size="20" text-anchor="middle">\(0.9\)</text><rect class="tm-soft" x="425" y="45" width="330" height="155" rx="16"/><text class="tm-text" x="590" y="78" font-size="20" text-anchor="middle" direction="rtl">مدل دوم</text><rect class="tm-accent" x="475" y="130" width="180" height="42" rx="10"/><text class="tm-text-accent" x="565" y="158" font-size="20" text-anchor="middle">\(0.8\)</text><text class="tm-muted" x="565" y="190" font-size="15" text-anchor="middle" direction="rtl">احتمال پاسخ درست کمتر</text><rect class="tm-soft" x="45" y="235" width="330" height="150" rx="16"/><text class="tm-text" x="210" y="273" font-size="19" text-anchor="middle">perplexity</text><text class="tm-muted" x="210" y="310" font-size="17" text-anchor="middle" direction="rtl">کمتر بهتر</text><text class="tm-muted" x="210" y="348" font-size="15" text-anchor="middle" direction="rtl">احتمال token درست</text><rect class="tm-soft" x="425" y="235" width="330" height="150" rx="16"/><text class="tm-text" x="590" y="273" font-size="19" text-anchor="middle" direction="rtl">accuracy و BLEU</text><text class="tm-muted" x="590" y="310" font-size="17" text-anchor="middle" direction="rtl">بیشتر بهتر</text><text class="tm-muted" x="590" y="348" font-size="15" text-anchor="middle" direction="rtl">گزارش مقاله: بهتر</text><rect class="tm-gold" x="260" y="400" width="280" height="38" rx="10"/><text class="tm-text-gold" x="400" y="426" font-size="16" text-anchor="middle" direction="rtl">Sublayer → Dropout → جمع با ورودی → LayerNorm</text></svg>
```
بازگشت به دوره