The Sequential Bottleneck of Recurrent Models
Attention Is All You Need
- Prediction prompt: if computing state h_t requires h_{t-1}, how many of the n steps for one sentence can run at the same time on a GPU?
- The sequence transduction setting from Section 3: an encoder maps input symbols (x_1, ..., x_n) to continuous representations z = (z_1, ..., z_n); a decoder generates (y_1, ..., y_m) one symbol at a time
- Auto-regressive generation: each output step consumes the previously generated symbols as additional input
- How recurrent models factor computation along positions: h_t is a function of h_{t-1} and the input at position t (Section 1)
- Why this precludes parallelization within a training example, and why it hurts more at longer sequence lengths when memory limits batching across examples
- Factorization tricks and conditional computation improved efficiency, but the paper states the fundamental constraint of sequential computation remains
- Attention was already used to model dependencies regardless of distance, but almost always together with a recurrent network
زنجیرهٔ وابستگی در لایه recurrent
```svg
<svg viewBox="0 0 800 300" xmlns="http://www.w3.org/2000/svg"><defs><marker id="a" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path class="tm-arrow" d="M 0 0 L 10 5 L 0 10 z"/></marker></defs><rect class="tm-soft" x="25" y="125" width="90" height="55" rx="12"/><text class="tm-text" x="70" y="159" font-size="23" text-anchor="middle">h<tspan baseline-shift="sub" font-size="15">0</tspan></text><text class="tm-muted" x="70" y="210" font-size="16" text-anchor="middle" direction="rtl">آماده</text><line class="tm-line" x1="115" y1="152" x2="150" y2="152" marker-end="url(#a)"/><rect class="tm-accent" x="155" y="125" width="90" height="55" rx="12"/><text class="tm-text" x="200" y="159" font-size="23" text-anchor="middle">h<tspan baseline-shift="sub" font-size="15">1</tspan></text><text class="tm-text2" x="200" y="60" font-size="17" text-anchor="middle">token 1</text><line class="tm-line" x1="200" y1="72" x2="200" y2="120" marker-end="url(#a)"/><line class="tm-line" x1="245" y1="152" x2="280" y2="152" marker-end="url(#a)"/><rect class="tm-accent" x="285" y="125" width="90" height="55" rx="12"/><text class="tm-text" x="330" y="159" font-size="23" text-anchor="middle">h<tspan baseline-shift="sub" font-size="15">2</tspan></text><text class="tm-text2" x="330" y="60" font-size="17" text-anchor="middle">token 2</text><line class="tm-line" x1="330" y1="72" x2="330" y2="120" marker-end="url(#a)"/><line class="tm-line" x1="375" y1="152" x2="410" y2="152" marker-end="url(#a)"/><rect class="tm-accent" x="415" y="125" width="90" height="55" rx="12"/><text class="tm-text" x="460" y="159" font-size="23" text-anchor="middle">h<tspan baseline-shift="sub" font-size="15">3</tspan></text><text class="tm-text2" x="460" y="60" font-size="17" text-anchor="middle">token 3</text><line class="tm-line" x1="460" y1="72" x2="460" y2="120" marker-end="url(#a)"/><line class="tm-line" x1="505" y1="152" x2="540" y2="152" marker-end="url(#a)"/><rect class="tm-accent" x="545" y="125" width="90" height="55" rx="12"/><text class="tm-text" x="590" y="159" font-size="23" text-anchor="middle">h<tspan baseline-shift="sub" font-size="15">4</tspan></text><text class="tm-text2" x="590" y="60" font-size="17" text-anchor="middle">token 4</text><line class="tm-line" x1="590" y1="72" x2="590" y2="120" marker-end="url(#a)"/><line class="tm-line" x1="635" y1="152" x2="670" y2="152" marker-end="url(#a)"/><rect class="tm-accent" x="675" y="125" width="90" height="55" rx="12"/><text class="tm-text" x="720" y="159" font-size="23" text-anchor="middle">h<tspan baseline-shift="sub" font-size="15">5</tspan></text><text class="tm-text2" x="720" y="60" font-size="17" text-anchor="middle">token 5</text><line class="tm-line" x1="720" y1="72" x2="720" y2="120" marker-end="url(#a)"/><text class="tm-text-accent" x="460" y="255" font-size="21" text-anchor="middle" direction="rtl">حداقل 5 گام پیدرپی؛ برای n token، حداقل n گام</text></svg>
```
batching زنجیرهٔ ترتیبی را حذف نمیکند
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path class="tm-arrow" d="M 0 0 L 10 5 L 0 10 z"/></marker></defs><text class="tm-text" x="400" y="60" font-size="23" text-anchor="middle" direction="rtl">دو جملهٔ پنج tokenی</text><text class="tm-text2" x="110" y="173" font-size="19" text-anchor="middle" direction="rtl">جملهٔ اول</text><text class="tm-text2" x="110" y="273" font-size="19" text-anchor="middle" direction="rtl">جملهٔ دوم</text><rect class="tm-soft" x="185" y="110" width="82" height="210" rx="14"/><rect class="tm-accent" x="195" y="140" width="62" height="55" rx="10"/><rect class="tm-accent" x="195" y="240" width="62" height="55" rx="10"/><text class="tm-text" x="226" y="174" font-size="21" text-anchor="middle">h₁</text><text class="tm-text" x="226" y="274" font-size="21" text-anchor="middle">h₁</text><text class="tm-text-accent" x="226" y="350" font-size="17" text-anchor="middle" direction="rtl">همزمان</text><path class="tm-line" d="M257 167 H290 M357 167 H390 M457 167 H490 M557 167 H590 M257 267 H290 M357 267 H390 M457 267 H490 M557 267 H590" marker-end="url(#arrow)"/><rect class="tm-surface" x="295" y="140" width="62" height="55" rx="10"/><rect class="tm-surface" x="395" y="140" width="62" height="55" rx="10"/><rect class="tm-surface" x="495" y="140" width="62" height="55" rx="10"/><rect class="tm-surface" x="595" y="140" width="62" height="55" rx="10"/><rect class="tm-surface" x="295" y="240" width="62" height="55" rx="10"/><rect class="tm-surface" x="395" y="240" width="62" height="55" rx="10"/><rect class="tm-surface" x="495" y="240" width="62" height="55" rx="10"/><rect class="tm-surface" x="595" y="240" width="62" height="55" rx="10"/><text class="tm-text" x="326" y="174" font-size="21" text-anchor="middle">h₂</text><text class="tm-text" x="426" y="174" font-size="21" text-anchor="middle">h₃</text><text class="tm-text" x="526" y="174" font-size="21" text-anchor="middle">h₄</text><text class="tm-text" x="626" y="174" font-size="21" text-anchor="middle">h₅</text><text class="tm-text" x="326" y="274" font-size="21" text-anchor="middle">h₂</text><text class="tm-text" x="426" y="274" font-size="21" text-anchor="middle">h₃</text><text class="tm-text" x="526" y="274" font-size="21" text-anchor="middle">h₄</text><text class="tm-text" x="626" y="274" font-size="21" text-anchor="middle">h₅</text><text class="tm-text-accent" x="465" y="405" font-size="22" text-anchor="middle" direction="rtl">هر جمله: حداقل پنج گام ترتیبی</text></svg>
```
از ورودی تا خروجی auto-regressive
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="arr" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto"><path class="tm-arrow" d="M0,0 L8,4 L0,8 Z"/></marker></defs><text class="tm-text" x="400" y="60" font-size="24" text-anchor="middle">encoder-decoder</text><rect class="tm-soft" x="100" y="140" width="230" height="100" rx="16"/><text class="tm-text" x="215" y="198" font-size="25" text-anchor="middle">encoder</text><rect class="tm-surface" x="470" y="140" width="230" height="100" rx="16"/><text class="tm-text" x="585" y="198" font-size="25" text-anchor="middle">decoder</text><line class="tm-line" x1="330" y1="190" x2="470" y2="190" marker-end="url(#arr)"/><text class="tm-text" x="215" y="300" font-size="23" text-anchor="middle">x₁ x₂ x₃</text><line class="tm-line" x1="215" y1="270" x2="215" y2="245" marker-end="url(#arr)"/><text class="tm-text-accent" x="400" y="166" font-size="20" text-anchor="middle">z₁ z₂ z₃</text><line class="tm-line" x1="585" y1="240" x2="585" y2="275" marker-end="url(#arr)"/><rect class="tm-accent" x="550" y="280" width="70" height="45" rx="10"/><text class="tm-text" x="585" y="310" font-size="23" text-anchor="middle">y₁</text><path class="tm-line-accent tm-flow" d="M620 302 L745 302 L745 190 L705 190" fill="none" marker-end="url(#arr)"/><line class="tm-line" x1="585" y1="325" x2="585" y2="365" marker-end="url(#arr)"/><rect class="tm-soft" x="550" y="370" width="70" height="45" rx="10"/><text class="tm-text" x="585" y="400" font-size="23" text-anchor="middle">y₂</text><text class="tm-text-accent" x="705" y="350" font-size="17" text-anchor="middle">auto-regressive</text></svg>
```
بهبود محاسبات، زنجیره را حذف نمیکند
```svg
<svg viewBox="0 0 800 450" xmlns="http://www.w3.org/2000/svg"><defs><marker id="a" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path class="tm-arrow" d="M 0 0 L 10 5 L 0 10 z"/></marker></defs><text class="tm-text2" x="400" y="45" text-anchor="middle" font-size="19" direction="rtl">factorization و conditional computation: بهبود هر گام</text><text class="tm-muted" x="400" y="90" text-anchor="middle" font-size="18" direction="rtl">وابستگی درون یک مثال</text><rect class="tm-accent" x="90" y="125" width="85" height="54" rx="10"/><rect class="tm-accent" x="225" y="125" width="85" height="54" rx="10"/><rect class="tm-soft" x="360" y="125" width="85" height="54" rx="10"/><rect class="tm-soft" x="495" y="125" width="85" height="54" rx="10"/><rect class="tm-soft" x="630" y="125" width="85" height="54" rx="10"/><text class="tm-text" x="132" y="158" text-anchor="middle" font-size="21">h<tspan baseline-shift="sub" font-size="14">1</tspan></text><text class="tm-text" x="267" y="158" text-anchor="middle" font-size="21">h<tspan baseline-shift="sub" font-size="14">2</tspan></text><text class="tm-text" x="402" y="158" text-anchor="middle" font-size="21">h<tspan baseline-shift="sub" font-size="14">3</tspan></text><text class="tm-text" x="537" y="158" text-anchor="middle" font-size="21">h<tspan baseline-shift="sub" font-size="14">4</tspan></text><text class="tm-text" x="672" y="158" text-anchor="middle" font-size="21">h<tspan baseline-shift="sub" font-size="14">5</tspan></text><path class="tm-line" d="M175 152 H225 M310 152 H360 M445 152 H495 M580 152 H630" marker-end="url(#a)"/><path class="tm-line-accent" d="M132 188 C200 260 600 260 672 188" marker-end="url(#a)"/><text class="tm-text-accent" x="400" y="248" text-anchor="middle" font-size="18" direction="rtl">attention در کنار recurrence</text><text class="tm-text2" x="400" y="300" text-anchor="middle" font-size="19" direction="rtl">Transformer: اتکا به attention، بدون recurrence</text><text class="tm-text-green" x="400" y="354" text-anchor="middle" font-size="20" direction="rtl">حداقل 5 گام ترتیبی برای هر مثال</text><text class="tm-text2" x="400" y="391" text-anchor="middle" font-size="18" direction="rtl">batch: موازی میان مثالها، نه میان حالتهای یک مثال</text></svg>
```
Back to course