Intro to NLP

Through the lens of computational ef ciency

Prof. Emma Strubell

fi
AWS credits
Last chance to request is TODAY.

• We already submitted a batch of requests for the students who submitted the
form on time. Thank you!

• We will submit again on Friday (tomorrow) for those who did not submit on time.
Unfortunately due to the way these are processed we cannot guarantee that late
submissions will get credits, but we will try.

• Form is on Piazza.
Keywords

• Feature representations.

• Input / output dimensionality.

• CNNs: Layers (norms, pooling, etc). 1D, 2D, 3D.

• Transformers: Self-attention, encoder vs decoder.

• Mapping, real valued control, …

Questions

• How much information does a pixel, a spectrogram, a word contain?

• How much do I care about parameter size vs model size?

• Does my task require a large vocabulary (why?), high resolution (why?), …

• What interactions are required in the input?

(Which pixels or words need to talk to each other?)
 Jacob Eisenstein: An Introduction To Natural Language Processing
Jacob Eisenstein: An Introduction To Natural Language Processing

Lab
The 1:classification
text
The text NLP using
classification Bag
problem
problem of Words
■ Given
Givenaatext
text w = (w1 , w2 , . . . , wT ) 2
Given a text w = (w , w , . . . , w ) 2 V ⇤
V ⇤
problems?
choose a label y 2 Y. 1 2 T
■ Choose
choose aa label
label y 2 Y.

■ The bag-of-words is a xed-length vector of word counts:

w = The drinks were strong but the sh tacos were bland
x = 0 … 1 1 … 1 … 0 1 2 1 … 2 … 0

r k d u t h o s e ng re e r
a n b s a c o t h o … e … t h
d v … b l a … … t tac s t r w z y
a r
a

■ Length of x is equal to the size of the vocabulary, V

5
fi
fi
fi
 Neural architectures for sequence data
How to represent text for classification?
Text is naturally viewed as a sequence of tokens, w , w , . . . , w 1 2
Another choice of input representation: word embeddings
I Context is lost when this sequence is converted to a
■ Text is naturally viewed as a sequence of tokens w1, w2, …, wT
bag-of-words.
■ Context is lost when this sequence is converted to a bag-of-words.
I Instead, a lookup layer can compute embeddings for ea
■ Instead, a lookup layer token,
can resulting
compute in
embeddingsa matrix X (0) = ⇥
(real-valued
(x!z) [e
vectors)wfor
1 , e w 2 , . . . ,
each type, resulting in a matrix
where X 2 R (0) K e ⇥M .
I Higher-order representations X(d>0) can then be comput
-1.36 0.28 from (0) , as2.24
0.71 X-1.3 shown -1.36on-0.76
the next
0.71 slide.
1.23
-0.23 0.54 1.43 0.39 -1.58 -0.23 1.66 1.43 -0.62
0.84 I For
0.07 0.11classification,
-1.14 -0.39 the top
0.84 -0.15layer
(D)
0.11X 1.13must Ke be converted i
… … vector,
… using
… a pooling
… … operation,
… … such as max-pooling:
…
-0.067 -2.32 -5.6 0.07 1.64 -0.067 0.52 -5.6 0.16

e k s re n g u t e s (D)re (D) d (D)

w = h ir n e h
zk t = max(x o e , x n, . . . , x ).
t w rt o b c w la
ta
k,1 k,2 k,M
d s b 6
 combination of x and y , such as,
(
How to represent fj (x, y )a
= word?
xwhale , if y = Fiction,
0, otherwise.
■ Until the ~2010s, in NLP words == atomic symbols.
This corresponds to the following
■ One-hot representations in vector space. formalization as a column vector,

■ f (x,
But leader and president arey orthogonal!
= 1) = [x; 0; 0; . . . ; 0] 0
| {z }
0 … 0 0 0 0 1 tacos
1 0 0 0 0 1 0 0 0 (K 1)⇥V 0 0 … 0 1 0 0 0 burritos
0 0 0 0 1 0 0 0 0
0 1 0 0 0 f (x, y = 2) = [0; 0; . . . ; 0; x; 00; 00; …
0 0 0 0 . . .0; 00] 1 0 0 drinks
0 0 0 1 0 0 0 0
| {z } | {z }
0
V (K0 …
2)⇥V
0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 leader
0 0 1 0 0f (x, y = K ) = [0; 0; . . . ; 0; x],0
0 0 1 0
1 … 0 0 0 0 0 president
… … … … … … … | {z }
… …
0 0 0 0 0 0 1 0 0 (K 1)⇥V
s
0 0 … 1 0 0 0 0 bland
e ks re ng ut e o re d
w = th rin we tro b h
t ac we lan
d s t b
7
How to represent a word?
■ Solution: encode word similarity (not just identity) in word
representations.

■ How to encode similarity?

■ Consider encountering a new word: tezgüino. context similarity?
1. A bottle of tezgüino is on the table. 1 2 3 4

tezgüino 1 1 1 1
2. Everybody likes tezgüino.
loud 0 0 0 0 0
3. Don’t have tezgüino before you drive. motor oil 2

term
1 0 0 1

tortillas 0 1 0 1 2
4. We make tezgüino out of corn.
choices 0 1 0 0 1
wine 1 1 1 0 3
8
 Distributional hypothesis
■ These representations encode distributional properties of each word.
■ The distributional hypothesis: words with similar meaning are used in similar
contexts.

“The meaning of a word it its use in the language.” [Wittgenstein 1943]

“If A and B have almost identical environments we say that they are synonyms.” [Harris 1954]

“You shall know a word by the company it keeps.” [Firth 1957]

9
Distributional hypothesis
■ These representations encode distributional properties of each word.
■ The distributional hypothesis: words with similar meaning are used in similar
contexts.

10
 Neural architectures for sequence data
How to represent text for classification?
Text is naturally viewed as a sequence of tokens, w , w , . . . , w 1 2
Another choice of input representation: word embeddings
I Context is lost when this sequence is converted to a
■ Text is naturally viewed as a sequence of tokens w1, w2, …, wT
bag-of-words.
■ Context is lost when this sequence is converted to a bag-of-words.
I Instead, a lookup layer can compute embeddings for ea
■ Instead, a lookup layer token,
can resulting
compute in a
embeddingsmatrix X (0) =
(real-valued⇥ (x!z) [e
vectors)wfor
1 , e w 2 , . . . ,
each type, resulting in a matrix
where X 2 R(0) K e ⇥M .
I Higher-order representations X(d>0) can then be comput
-1.36 -0.11 0.71 0.53 0.19(0) -1.36 1.27 0.71 0.17
from X , as shown on the next slide.
-0.23 -0.58
0.84 -0.32
1.43 -0.42 -0.33 -0.23 -0.86
0.11 I -0.27 -0.25 0.84
1.43
2.11the 0.11
-0.08
-1.04 X
implications
(D)
For classification, top layer must be converted i
… … … … … … … … … for e ciency?
-0.067 0.82 -5.6
vector,
1.94
using a pooling
-0.2 -0.067 -0.24 -5.6
operation,
-0.38
such as max-pooling:
e ks re n g u t e s re d
w = h n e th o e n
i rt o b
t (D) (D) (D)
r w azc = max(x
w la , x , . . . , x ).
d s t k b k,1 k,2 k,M
11
ffi
word2vec
■ Instead of counting how often each word w occurs near apricot
■ Train a classi er on a binary prediction task: Is w likely to show up near apricot?
■ Don’t actually care about performing this task, but we’ll take the learned classi er
weights as the word embeddings.

■ Training is self-supervised: no annotated data required, just raw text!

■ Two algorithms:
■ Context bag-of-words (CBOW): predict current word using context
P(wt | wt+1 , ..., wt+k , wt
<latexit sha1_base64="3WBJvUFBWiHqBTSXihrpkDx+O1M=">AAAEDXicfVJLj9MwEHa3PJby6sKRi6FUWtg2ahckuCCtgAMXxCKxD6muKseZpFb9CLbTbhXlN3Dlj3BDXPkN/BmEmxS02a4YKdbnb2Yyn2cmTAW3bjD41dhqXrl67fr2jdbNW7fv3G3v3Du2OjMMjpgW2pyG1ILgCo4cdwJOUwNUhgJOwtmblf9kDsZyrT65ZQpjSRPFY86o89Sk/bVLtPcbnkwdNUYv8hlXSYH7+CIvqSrw3ga90KXj1Ybjcwagitbh7mLiMJE8wotJ7vaGRQ8HQdCrbrOiAv0a3Z8VTybtziAYlIY3wXANOmhth5OdrTGJNMskKMcEtXY0HKRunFPjOBNQtEhmIaVsRhMYeaioBDvOyxYWuOuZCMfa+E85XLLnM3IqrV3K0EdK6qb2om9FXuYbZS5+Oc65SjMHilWF4kxgp/FqHjjiBpgTSw8oM9xrxWxKDWXOT63VPV9mCmIOrv4QJse5jcvqNUmh9HcDChZMSz+h6GlOYiq5WEYQ00y4Iic2/osva00vmvPUrrt0VrWpRQQ4ov2UuaJCQOzI6qjT5Q6Q8myRt+BnYeC9F/ghBUOdNl4JNYmkZ4WfTUIekhX8XyRX/yI9rD8rLwX4x6xaoFNQeVFCJrQFEiZGZ2lN8EZ+KdT/gMa+41U81NOqCL+Qw4vrtwmO94Phs2D/4/POwev1am6jB+gR2kVD9AIdoHfoEB0hhn43Hjf6jaD5pfmt+b35owrdaqxz7qOaNX/+AYyLYyE=</latexit>
1 , ..., wt k )
■ Skip-gram: predict each context word using current word
P(wt+1 , ..., wt+k , wt
<latexit sha1_base64="akpfjX8kBn0dvm8cm4RZpNlMonM=">AAAER3icjVJNb9NAELWbAsV8tIUjl4UoUktbKylI9IJUAQcuiCKRtlI3ROv12FllP8zuumlk+Qdy5cav4IY4srZTVDcVYiSv3r5545mdmSjjzNh+/4e/0lm9dfvO2t3g3v0HD9c3Nh8dG5VrCkOquNKnETHAmYShZZbDaaaBiIjDSTR9W/lPzkEbpuRnO89gJEgqWcIosY4ab3zvYeX8mqUTS7RWs2LKZFqiPXSdF0SWaGeJnqna8XrJ8TUHkGXQO9qajS3CgsVoNi7szqDcRWEY7ja3admAvRa9Ny23gyrw//WXGez2eKPbD/u1oWUwWICut7Cj8ebKCMeK5gKkpZwYczboZ3ZUEG0Z5VAGODeQETolKZw5KIkAMyrq5peo55gYJUq7T1pUs1cjCiKMmYvIKQWxE3PdV5E3+c5ymxyMCiaz3IKkTaIk58gqVE0SxUwDtXzuAKGauVoRnRBNqHXzDnpX00yAn4NtP4SKUWGSOnurpEi4uwYJM6qEm238vMAJEYzPY0hIzm1ZYJNc4ptasxufs8wsunTRtCnAHCxWbj+YJJxDYnF1tOl6e3B9BvgduFlo+OAK/JiBJlZpVwnRqSAXpZtNip/iCv5LyeRfpYPtZxV1Ae4xVQtUBrIoa0i5MoCjVKs8axW8FF8X6n5AEtfxRg/tsEbhFnJwff2WwfF+OHgR7n962T18s1jNNe+J98zb8gbeK+/Qe+8deUOP+gf+Fz/1J51vnZ+dX53fjXTFX8Q89lq26v8BLoh01w==</latexit>
1 , ..., wt k | wt )
12
fi
fi
 Language models as word representations
■ A problem with (most) word2vec-style word embeddings: single representation
per word type.
the new-look play area is due to be completed by early spring 2020
gerrymandered congressional districts favor representatives who play to the party base
the freshman then completed the three-point play for a 66-63 lead

■ Multiple senses entangled:

play = -1.36 -0.51 0.71 -0.29 -0.18 -1.36 -0.08 0.71 0.61 …

nearest neighbors: playing played Play verb

game plays football noun
games player multiplayer adjective
13
 Language models as word representations
■ A problem with (most) word2vec-style word embeddings: single representation
per word type.
the new-look play area is due to be completed by early spring 2020
gerrymandered congressional districts favor representatives who play to the party base
the freshman then completed the three-point play for a 66-63 lead
■ Instead, want contextualized word embeddings: token representations that differ
depending on the context.

■ But, still want to leverage self-supervised training on massive data.

■ Solution: use hidden representations from a neural language model.
14
Probabilistic language models
■ Goal: compute the probability of a sentence (or sequence of words):
P(w) = P(w1 , w2 , w3 , ..., wn )
<latexit sha1_base64="uV9El3eWQJrO9mCHFF9pZ71WvHw=">AAADm3icfVJbi9NAFJ5tvKz11tVHEaJlpSslNF1BX4TF9UFEsYLdXeiUMpmcpMPOJcxM2i0h/8Bf46v+Ef+Nk7TKZrs4kJMv55LznW9OlHFm7GDwe6fl3bh56/bunfbde/cfPOzsPToxKtcUxlRxpc8iYoAzCWPLLIezTAMREYfT6Py4ip8uQBum5De7ymAqSCpZwiixzjXrvBj1cCSKZXngv/VHveUs7PvL2bAyh30/CIIKyYNZpzsIBvXxt0G4AV20OaPZXmuKY0VzAdJSToyZhIPMTguiLaMcyjbODWSEnpMUJg5KIsBMi3qg0t93nthPlHaPtH7tvVxREGHMSkQuUxA7N1djlfO62CS3yZtpwWSWW5B03SjJuW+VX6njx0wDtXzlAKGaOa4+nRNNqHUatvcvt5kDX4BtDkLFtDBJ3b1BKRLuW4OEJVVCEBm/LHBCBOOrGBKSc1sW2CR/8XXS9OMFy8xGpYu1TG3MwWKlWcok4RwSiyvTdLvX3OLatvF7cHeh4bMj+CUDTazSjgnRqSAXpbubFD/DFfxfJpP/Mh1sjlXUBNwwlQQqA1mUNaRcGcBRqlWeNQhv1ddE3Q9I4hRf50OzbJ3hFjK8un7b4GQYhIfB8Our7tG7zWruoifoOeqhEL1GR+gDGqExoug7+oF+ol/eU+/Y++h9Wqe2djY1j1HjeOM/p0c0yw==</latexit>

■ Related task: probability of the next word:

P(w5 | w4 , w3 , w2 , w1 )
<latexit sha1_base64="jRmBI2p2WQtfZa//VkFBNhGeKo8=">AAADk3icfVLLattAFJ1YfaTuI05KV4Wi1gTSEozlpKTQTUiz6CbUhToJeIQZja7kIfMQMyM7RnjXr+m2/Zn+TUeSW6I4dEBXR/ehe+6ZG2WcGdvv/95oeffuP3i4+aj9+MnTZ1ud7Z1zo3JNYUQVV/oyIgY4kzCyzHK4zDQQEXG4iK4+lfGLGWjDlPxmFxmEgqSSJYwS61yTzqvh3nzy3seCxf58crjvzEFpBqUJ3k463X6vXx1/HQQr0EWrM5xst0IcK5oLkJZyYsw46Gc2LIi2jHJYtnFuICP0iqQwdlASASYsqkGW/q7zxH6itHuk9SvvzYqCCGMWInKZgtipuR0rnXfFxrlNPoQFk1luQdK6UZJz3yq/VMWPmQZq+cIBQjVzXH06JZpQ67Rr795sMwU+A9schIqwMEnVvUEpEu5bg4Q5VUIQGb8rcEIE44sYEpJzuyywSf7iu6TZj2csMyuVrmuZ2piDxUqzlEnCOSQWl6bpdq+pxZVt41Nwd6HhzBH8koEmVmnHhOhUkOulu5sUv8Yl/F8mk/8yHWyOVVQE3DClBCoDWSwrSLkygKNUqzxrEF6rr4i6H5DEKV7nQ7OsznALGdxev3VwPugFB73B18Pu8clqNTfRS/QG7aEAHaFj9BkN0QhR9B39QD/RL++F99E78U7r1NbGquY5ahzv7A9RbzKU</latexit>

■ A model that computes either of these is called a language model (or LM).

15
 Why language models?

■ Machine translation:
P(high winds tonight) > P(large winds tonight)

■ Spelling correction:
P( ve minutes late) > P( ve minuets late)

■ Speech recognition:
P(I saw a van) > P(eyes awe of an)

■ Since ~2018, state-of-the-art for many

language processing tasks.

16
fi
fi
 How to compute P(w)?
■ Want to compute the joint probability:
Y
P(w1 , w2 , w3 , ..., wn ) = P(wi | w1 , w2 , ..., wi 1)
<latexit sha1_base64="U8ZV/pD4Y1NJyPBat3IBopfzn6g=">AAAFNXicfVJNbxMxEN2GACV8tXDkYqiKuihE2bQILkht4cABRJBIi1RHkeP1Jla99sr2NolW+7s48js4cENc+QuMdzdpt61YKZM3M2/smecZJ4Ib2+3+XGvcaN68dXv9TuvuvfsPHm5sPjoyKtWUDagSSn8bE8MEl2xguRXsW6IZiceCHY9P37n88RnThiv51S4SNozJRPKIU2IhNNpc+7Hd35mNXiEc8xDNRnttMLvO9JwJ/BbkuTXgEMt0G3GARrWR1USahGgmLThTYn301nEPy5MOwEU40oRm/Z2DNjr0cwf83HEKH/IuABVVgcvMR0Ebzd3dc9dFp9NxSJZkSPrO9sorlu7uyi1KfShyYXkerY7J5MsgXw7kI0xDZeHYYrDyjFqcm1UQl9Of54y6nMOgzHn+gjrXETFI6Lec8MFK6uW4s3JcnGgVjrjj8OptltyKlnE3zmhjq9vpFh+6CoIKbHnV1x9tNoY4VDSNoTcqiDEnQTexw4xoy6lgeQunhiWEnpIJOwEoSczMMCtWLUfbEAlRpDT8YLYierEiI7Exi3gMzJjYqbmcc8Hrciepjd4MMy6T1DJJy4uiVCCrkNtbFHLNqBULAIRqDr0iOiWwXyCpaW1fvGbKxBmz9UFoPMxMVNxea2kcgw/PxGZUxTGR4YsMRyTmYhGyiKTC5hk20RJfJ007POOJqVSalzK1sGAWK80nXBIhWGSxM/Uw/E0tLmwLv2fwFpp9ggY/J0wTqzR0QvQkJvMc3maCn2IH/8fkcsUEWB8rKxqAYZwEKmEyywtIhTIMjydapUmt4Sv1RaNwAIlA8ZLP6mUlAxYyuLx+V8FRrxPsdnpf9rb2D6vVXPeeeM+8HS/wXnv73gev7w082nje+NgYNI6a35u/mr+bf0pqY62qeezVvubff3PQsgI=</latexit>
i

■ For example:
P(its, water , is, so, transparent, that) =
<latexit sha1_base64="yLMSiidNLBCPcyzUiGZ9zjUHbv8=">AAAEZXicfVLdbtMwFE63AiMw2ABxwwWGaVKDStV0Q3CDtA0uuEEUif1Ic1U5jtNac+zIdrZWIZfwjjwBtzwCx0k3lm3CUk6+8//5HEeZ4Mb2+79aS8vtW7fvrNz1791fffBwbf3RgVG5pmyfKqH0UUQME1yyfcutYEeZZiSNBDuMTj44/+Ep04Yr+c3OMzZKyUTyhFNiwTRe+7M57JyN3yCc8hidjbe7ILacGDgRBv6ww60BTCzTXcQBGtVFVhNpMqKZtKBMiQ3Qex9K7dWFdkFFONGEFsPObhftBaUDQeliKh38zgAZiwTnmY3DLpq51jNHotfrOSTrYHAGTg7qFufq1oVapQaQ5Mzyn3VRppCvwzLwx2sb/V6/Oug6CBdgw1uc4Xh9aYRjRfMU7koFMeY47Gd2VBBtORWs9HFuWEboCZmwY4CSpMyMimo1JdoES4wSpeGTFlXWyxkFSY2ZpxFEpsROzVWfM97kO85t8m5UcJnllklaN0pygaxCbs8o5ppRK+YACNUcuCI6JbAR2KPxNy+3mTJxymzzIjQdFSapujcoRSnosHZ2RlWaEhm/KnBCUi7mMUtILmxZYJOc45tG041PeWYWU5rVY/KxYBYrzSdcEiFYYrETTTP8phZX0scfGexCs89A8EvGNLFKAxOiJymZlbCbCX6BHfxfJJcXkQCb1yoqAnAZNwKVMVmUFaRCGYajiVZ51iB8Lb8iCgVIAhOv41kzrY6ABxlefX7XwcGgF271Bl+3N3b2Fk9zxXvmvfQ6Xui99Xa8T97Q2/do66D1vfWj9XP5d3u1/aT9tA5dai1yHnuN037+F79gbTA=</latexit>

P(its) · P(water | its) · P(is | its water ) · P(so | its water is) · P(tran <latexit sha1_base64="Ke4jlzshZRmzIj5Dxs23yKyAh9c=">AAAE7nicfVJLb9NAEHZCSouBPuDIZaGqVKMQxWkRXJDawoELIkj0IXWjaLNeJ6uud63ddZvI8t/ghrjylzjxV5i13YfbipUy+Wa+eXsmqeDG9vt/Wu0HnaWHyyuP/MdPnq6urW88OzIq05QdUiWUPpkQwwSX7NByK9hJqhlJJoIdT84+Ov74nGnDlfxuFykbJWQqecwpsWAab7TmW8Pti/FbhBMeoYvxbhfEjhMDJ8LAB55bAwqxTHcRB2hUF1lNpEmJZtKCMiM2QB+c70GVaR9UhGNNaD7c3u+ig6BwICicT6kD7wwQUQc4Zj4Ou2juas9dF71ezyFZOQMZODmoSlyqO1dqGRpAkDPLa2udJpdvwiLwy3kChGmkLGQt56pSNOzcXBlxNfw1Z9RtDsNirvkby7nPEcMGg/H6Zr/XLx+6C8IabHr1G4432iMcKZolkJUKYsxp2E/tKCfacipY4ePMsJTQMzJlpwAlSZgZ5eWNFGgLLBGKlYYfdFVab0bkJDFmkUzAMyF2Zm5zzngfd5rZ+P0o5zLNLJO0KhRnAlmF3MGhiGtGrVgAIFRz6BXRGYHDgGUYf+tmmRkT58w2B6HJKDdxWb3R0iQBHRbMLqhKEiKj1zmOScLFImIxyYQtcmziS3zfarrROU9NvaV5tSYfC2ax0nzKJRGCxRY70TTD38ziUvr4E4NvodkXaPBryjSxSkMnRE8TMi/g20zxS+zg/zy5vPIE2BwrLxuAYdwKVMpkXpSQCmUYnky1ytJGw3fiy0YhAYlh45U/a4ZVHnCQ4e3zuwuOBr1wpzf4tru5d1Cf5or3wnvlbXuh987b8z57Q+/Qo62/7aX2anutk3Z+dH52flWu7VYd89xrvM7vf1c8nFc=</latexit>

P(water | its) · P(is | its water ) · P(so | its water is) · P(transparent | its water is so)

17
 How to estimate P(wi)?
■ Can we just count and divide (relative frequency estimate)?
count(its water is so transparent that the)
P(the | its water is so transparent that) =
count(its water is so transparent that)
<latexit sha1_base64="cIuL3OLQ+AQY7cqAAH4BiMv069w=">AAAF43icnVRLb9NAEHZTAiW8WjhyMVSVahSiOC2CC1JbOHBBBInSSt0o2qzXyar7sHbXTSLLv4Ab4sq/4sJf4cSs7bRxWxDCUibfzHyz81p7lHBmbLf7c6WxeqN589ba7dadu/fuP1jfePjZqFQTekgUV/p4hA3lTNJDyyynx4mmWIw4PRqdvnH+ozOqDVPyk50ndCDwWLKYEWzBNNxoiK3+9nT4wkeCRf50uNsGseNEz4kwaIGfWQMKtlS3fQbQqLZvNZYmwZpKC8oE28B/7bgH5Un7oPoo1phk/e39tn8Q5A4EueMUOvidASKqAOeZDcO2P3O5Z66KTqfjkCzJ4Ayc7JUpFurOuVqEBhDkzPLCWh2Tyedhvmgo8BGJlIVji8bKM2p2Zs6NqOz+wmfUZR+CyVz4l6ZzHRHBCItCpq68ataLfqdlvyjRKhoyx2HVchbcipaxoh/INqF/yoKWF4UWi6o2gwS2Ey0yolJp8+1/inaSwjL/JzbIh+ub3U63ePyrIKzAplc9fbigAxQpkgo4hHBszEnYTewgw9oywmneQqmhCSaneExPAEosqBlkxYuR+1tgifxYafjBJgrrckSGhTFzMQKma8dc9jnjdb6T1MavBhmTSWqpJGWiOOW+Vb57y/yIaUosnwPARDOo1ScTDDOHAZnW1nKaCeVn1NYbIWKQmbjIXitpJECHedIpUUJgGT3LUIwF4/OIxjjlFvZi4gW+bjTt6IwlpprSrBxTC3FqkdJszCTmnMYWOVE3w9/EokK20FsKu9D0PRT4IaEaW6WhEqzHAs9y2M0YPUEO/o3J5DkTYL2trCigumQqoTLLC0i4MhSNxlqlSa3gK/FFoXAAjmHiJZ/Ww0oGXMjw8vW7Cj73OuFOp/dxd3PvoLqaa95j76m37YXeS2/Pe+f1vUOPNH40fq2uwteaNr80vza/ldTGShXzyKs9ze+/AdiD8Y0=</latexit>

■ No! Too sparse: we’ll never observe enough data to estimate.

■ High variance: Unobserved grammatical sentences will have probability 0.

18
Markov assumption
■ Markov assumption: conditional probability distribution of future states (words)
depends only on the present state, not the sequence of events that preceded it.
Y
P(w1 , w2 , w3 , ..., wn ) ⇡ P(wi | wi 1 , ..., wi k )
<latexit sha1_base64="Gv+gRtBmN+qrTZzF1Bn1zByDaug=">AAAGiHicnVRLb9NAEHYbCMW82nLkslBVqlGI4rSocEBqAwcuiCDRFqkbRZv1OlnV9lq76yaR5d/CFX4S/4ZZP9q4CRVgKZN5fLM7883YozjgSnc6v9bWG3fuNu9t3LcfPHz0+Mnm1vapEomk7ISKQMhvI6JYwCN2orkO2LdYMhKOAnY2unhv4meXTCouoq96HrNBSMYR9zklGlzDrcb2bn9vOnyNcMg9NB0etEDsG9E1wnVsiHOtwCCayRbioCrRQlqSSMVEskiDMSHaQe8MtlecdAwmwr4kNO3vHbdQz8mM4mQGk9sQNw7IKBNMZDZ0W2hm7p6ZKtrtttGiAgxBx8hucUVl7l+ZeaoDScYdXXvLY9LolZtVDTkIU09oODZvrDij5ufqyomL7q9jStyMYWDmOr7AziogBgrzQqamvJLrqt9p0S+OpfCG3GB4OZwKW8JSXvWjJ+xP1+DFSeFqUuVocEj0RIYpFUmks72/yjaSwTT/J7cY/79Wi0kMXMzQQuYyyr6FzOqAFZTmHC4yepE5w82dTruTP2hZcUtlxyqf/nBrfYA9QZMQyqEBUerc7cR6kBKpOQ1YZuNEsZjQCzJm56BGJGRqkOZvb4Z2weMhX0j4wbrk3sWMlIRKzcMRIA3l6mbMOFfFzhPtvxmkPIoTzSJaXOQnAdICmU8B8rhkVAdzUAiVHGpFdEJgL2Agyt5dvGbCgkum643QcJAqP7+9VtIoBBsmw6ZUhCGJvJcp9knIg7nHfJIEGnZH+ZW+ipqWd8ljVbI0K2iyccA0FpKPeUSCgPkaG1F3w99E41za+AODWUj2CQr8HDNJtJBQCZHjkMwymM0YP8dGvQ3JoyskqPW20ryA8kUQMYvSLFdpIBTDo7EUSVwreCk/LxQOID4wXuBZPa1AwEK6N9dvWTnttt39dvfLwc5Rr1zNDeuZ9cLas1zr0DqyPlp968SijXnje+NH42fTbnaah823BXR9rcx5atWeZu83OUMm2w==</latexit>
i

■ In other words, approximate each component of the product:

P(wi | wi
<latexit sha1_base64="y19Il2zgQALmijEY2ZIBzqzR+tY=">AAAG1XicnVRLb9NAEHYbCCW8WjhyWagqJShEcVoEF6S2cOCCKBJ9SN0o2qzXyar2rrW7bhJZviGu/Cv+A/+BK5yZ9aONm1ABljKZxzezM9+sPYwCrk23+31ltXbjZv3W2u3Gnbv37j9Y33h4pGWsKDukMpDqZEg0C7hgh4abgJ1EipFwGLDj4dkbGz8+Z0pzKT6ZWcT6IRkJ7nNKDLgGG7XB1kFzMniBcMg9NBnstEFsW9Gzwm01IM6NBoMYptqIg6plGxlFhI6IYsKAMSamhV5b7H5eaQ9MhH1FaHLQ3Guj/VZqlVZqMZkNceuAjCLBRqYDt42m9uyp7aLT6VhN5GAItqzs5UeU5vaFmaW2IMm6xaW3KJOI525aDtRCmHrSQNlssLxGxc/1hRPn01/GtLwaw8DMZXyOnWVADBRmjUxsewXX5byTfF4cKekNuMXwYjkltoAlvJzHjNmfjsHzm8LlporV4JCYsQoTKmNh0uZfZVvJYJv/k5uv/1+7xSQCLqZoLnMRdS2bZYUlnGYkzlN6BpRmALQUUbwWc01dX2uwvtntdLMHLSpuoWw6xXMw2FjtY0/SOITZaEC0PnW7keknRBlOA5Y2cKxZROgZGbFTUAUJme4n2acgRVvg8ZAvFfzg7mXe+YyEhFrPwiEg7f701Zh1LoudxsZ/1U+4iGLDBM0P8uMAGYnsdwV5XDFqghkohCoOvSI6JnDJYLu6sTV/zJgF58xUB6FhP9F+dnqlpWEINqyZTagMQyK8Zwn2SciDmcd8EgcGLqL2S30ZNW3vnEe6YGma09TAATNYKj7iggQB8w22ouqGv7HBmWzgtwx2odh7aPBDxBQxUkEnRI1CMk1hNyP8BFv1OiQXF0hQq2MlWQPFWyUjJpI0U2kgNcPDkZJxVGl4IT9rFAoQHxjP8ayaliPgQrpXr9+ictTruNud3sedzd394mquOY+dp07TcZ2Xzq7zzjlwDh1a+1b7UftZ+1U/rqf1z/UvOXR1pch55FSe+tffqDtBJg==</latexit>
1 , ..., w2 , w1 ) ⇡ P(wi | wi 1 , ..., wi k )

19
 Simplest case: Unigram model
■ Unigram model:
Y
P(w1 , w2 , ..., wn ) ⇡ P(wi )
<latexit sha1_base64="4ciLqBLwCLf1cKEiZEyIk78X24s=">AAAHBHicnVTLbtNAFHXbEEp4tbDsZqBEilGI4rQINkhtYcEGESTaInWiaDIeJ6PaHmtm3CSyvGXJl7BDbPkPPoQ9d2ynjZu0PCxlch/n3jn3YQ8inyvdbv9cWV2r3KjeXL9Vu33n7r37G5sPjpSIJWWHVPhCfhoQxXweskPNtc8+RZKRYOCz48Hpa+M/PmNScRF+1NOI9QIyDLnHKdFg6m+ufal3G+P+c4QD7qJxf7cJx445OuZw7Br4uVagEM1kE3EQlWgiLUmoIiJZqEEZEW2jVwZ7kGfaBxVhTxKadBv7TXRgp0awU4PJdPAbA0QUAcYz6TtNNDF3TwyLVqtlpDAHg9M2Zye/YqbunKtZqA1BxhxeWIs0SfjMSWcF2QhTV2hImxWW5yjZuTo34rz6C58Sl30YOnPhn+vOMiCGFmZExoZe0etZveO8XhxJ4fa5wfBiODNsAUv4rB49Ylddg+cnhWeTKkaDA6JHMkioiEOdNv4q2pwMpvk/sfn4/5UtJhH0YoLmIhdR13ZzlmFJT7Mmzrf0NC1ycbQUUrwXc6z+kGye15WU8jR2rb+x3W61swctCk4hbFvF0+1vrvawK2gcQCuoT5Q6cdqR7iVEak59ltZwrFhE6CkZshMQQxIw1UuyL0eK6mBxkSck/GBVM+t8REICpabBAJBm3OqyzxiX+U5i7b3sJTyMYs1Cml/kxT7SApnPEHK5ZFT7UxAIlRy4IjoisJOwDKpWn79mxPwzpsuF0KCXKC+7vURpEIAOW8HGVAQBCd2nCfZIwP2pyzwS+xr2VnkzeVlrmu4Zj1TRpUnephr2mcZC8iEPie8zT2NzlM3wN9I4O2v4DYNZSPYOCL6PmCRaSGBC5DAgkxRmM8SPsBGvQ/LwHAliuawkI1C8hCJiYZJmIvWFYngwlCKOSoQX4jOikIB40PEcz8phOQIW0rm8fovCUafl7LQ6H3a39w6K1Vy3tqzHVsNyrBfWnvXW6lqHFl37VdmqPKnUq5+rX6vfqt9z6OpKEfPQKj3VH78BpspNqQ==</latexit>
i

■ Example sentences generated by a unigram model trained on nancial news:

fth an of futures the an incorporated a a the in ation most dollars quarter in is mass

thrift did eighty said hard ‘m july bullish

that or limited the

20
fi
fl
fi
Bigram model
■ Bigram model: condition on the previous word

P(wi | wi
<latexit sha1_base64="BrkrYNNGGHjdfBmG2CVYMwdOnAI=">AAAHRXicnVRJb9NAFHbbEIrZWjhyGagq1ShEcVqWC1JbOHBBBIkuUieKJuNxMqrtsWbGTSLLv48jZ34EN8QV3thOGyehKljKy1u+tz+7Hwdc6Vbr+8rqWu1W/fb6HfvuvfsPHm5sPjpWIpGUHVERCHnaJ4oFPGJHmuuAncaSkbAfsJP++TtjP7lgUnERfdGTmHVDMoi4zynRoOptrn3b7uyMei8RDrmHRr29BpBdQ9qGuI4Ndq4VCEQz2UAcWCUaSEsSqZhIFmkQhkQ76K3BHhaRDkBE2JeEpp2dgwY6dDLDOJnB5DLYjQI8SgdjGffcBhqb3GNTRbPZNFxUgMHoGNouUkzF3Usxd3XAyaijK20ZJo1euNm0IQdh6gkNYfPGihgVPVeXSlx0f2VTYt6GYTJX9pnpLANiGGFeyMiUV8562u+o6BfHUng9bjC8XM4UW8JSPu1HD9nf0uDZTeHppsrV4JDooQxTKpJIZzs38jaUwTb/x7dY/79Wi0kMsxijGc9F1LXTnEZYMtN8iLMjPc/KWBwthZTvxUxVNwk2t7rFmoo4jv2vmStQx7Z7G1utZit/0CLjlsyWVT6d3uZqF3uCJiHMkgZEqTO3FetuSqTmNGCZjRPFYkLPyYCdARuRkKlumn96MrQNGg/5QsIPbj3XznqkJFRqEvYBae5FzduMcpntLNH+m27KozjRLKJFIj8JkBbIfMeQxyWjOpgAQ6jkUCuiQwJHDdek7O3ZNEMWXDBdbYSG3VT5efZKSf0QZDgrNqIiDEnkPU+xT0IeTDzmkyTQcPjKn/LLRtPwLnisyimNizHZOGAaC8kHPCJBwHyNDamq4W+ocU5t/J7BLiT7CAV+ipkkWkiohMhBSMYZ7GaAn2LDXofk0SUS2GpbaV5A+RaLmEVplrM0EIrh/kCKJK4UvOCfFwoBiA8TL/Cs6lYg4CDd+fNbZI7bTXe32f68t7V/WJ7muvXEembtWK712tq3Plgd68iitVc1XGM1v/61/qP+s/6rgK6ulD6PrcpT//0HMwBitg==</latexit>
1 , ..., w2 , w1 ) ⇡ P(wi | wi 1)

■ Example sentences generated by a bigram model trained on nancial news:

texaco rose one in this issue is pursuing growth in a boiler house said mr. gurria
mexico ’s motion control proposal without permission from ve hundred fty ve yen

outside new car parking lot of the agreement reached

this would be a record november

21

fi
fi
fi
fi
 N-gram models
■ Can extend to trigrams, 4-grams, 5-grams, …
■ In general, this is an insuf cient model of language.
■ Language has long-distance dependencies, e.g.:
subject-verb agreement?

The computer(s) that I just put into the machine room on the fth oor is (are) crashing.

■ But, n-grams require accounting for Vn events. V=104, n=7 → 1028 events.

■ Another example of bias-variance tradeo

22
fi
f
fi
fl
 <s> (according to its bigram probability). Let’s say the second word of that bigram
is w. We next chose a random bigram starting with w (again, drawn according to its
bigram probability), and so on.
N-gram models To give an intuition for the increasing power of higher-order n-grams, Fig. 3.3
shows random sentences generated from unigram, bigram, trigram, and 4-gram
models trained on Shakespeare’s works.
■ Can extend to trigrams, 4-grams, 5-grams, …
–To him swallowed confess hear both. Which. Of save on trail for are ay device and
1gram rote life have
–Hill he late speaks; or! a more to leg less first you enter
–Why dost stand forth thy canopy, forsooth; he is this palpable hit the King Henry. Live
2gram king. Follow.
–What means, sir. I confess she? then all sorts, he is trim, captain.
–Fly, and will rid me these news of price. Therefore the sadness of parting, as they say,
3gram ’tis done.
–This shall forbid it should be branded, if renown made it empty.
–King Henry. What! I will go seek the traitor Gloucester. Exeunt some of the watch. A
4gram great banquet serv’d in;
–It cannot be but so.
Figure 3.3 Eight sentences randomly generated from four n-grams computed from Shakespeare’s works. All
Is one marks
characters were mapped to lower-case and punctuation of these su ascient
were treated for your
words. Output use case?
is hand-corrected 23
ffi
 Neural language models
■ Don’t count, predict!
■ Input: word embeddings x1, x2, …, xN
aardvark zyzzyva
■ Output: P(wi | wi 1 , ..., w2 , w1 ) ⇡ P(wi | wi 1 )
<latexit sha1_base64="BrkrYNNGGHjdfBmG2CVYMwdOnAI=">AAAHRXicnVRJb9NAFHbbEIrZWjhyGagq1ShEcVqWC1JbOHBBBIkuUieKJuNxMqrtsWbGTSLLv48jZ34EN8QV3thOGyehKljKy1u+tz+7Hwdc6Vbr+8rqWu1W/fb6HfvuvfsPHm5sPjpWIpGUHVERCHnaJ4oFPGJHmuuAncaSkbAfsJP++TtjP7lgUnERfdGTmHVDMoi4zynRoOptrn3b7uyMei8RDrmHRr29BpBdQ9qGuI4Ndq4VCEQz2UAcWCUaSEsSqZhIFmkQhkQ76K3BHhaRDkBE2JeEpp2dgwY6dDLDOJnB5DLYjQI8SgdjGffcBhqb3GNTRbPZNFxUgMHoGNouUkzF3Usxd3XAyaijK20ZJo1euNm0IQdh6gkNYfPGihgVPVeXSlx0f2VTYt6GYTJX9pnpLANiGGFeyMiUV8562u+o6BfHUng9bjC8XM4UW8JSPu1HD9nf0uDZTeHppsrV4JDooQxTKpJIZzs38jaUwTb/x7dY/79Wi0kMsxijGc9F1LXTnEZYMtN8iLMjPc/KWBwthZTvxUxVNwk2t7rFmoo4jv2vmStQx7Z7G1utZit/0CLjlsyWVT6d3uZqF3uCJiHMkgZEqTO3FetuSqTmNGCZjRPFYkLPyYCdARuRkKlumn96MrQNGg/5QsIPbj3XznqkJFRqEvYBae5FzduMcpntLNH+m27KozjRLKJFIj8JkBbIfMeQxyWjOpgAQ6jkUCuiQwJHDdek7O3ZNEMWXDBdbYSG3VT5efZKSf0QZDgrNqIiDEnkPU+xT0IeTDzmkyTQcPjKn/LLRtPwLnisyimNizHZOGAaC8kHPCJBwHyNDamq4W+ocU5t/J7BLiT7CAV+ipkkWkiohMhBSMYZ7GaAn2LDXofk0SUS2GpbaV5A+RaLmEVplrM0EIrh/kCKJK4UvOCfFwoBiA8TL/Cs6lYg4CDd+fNbZI7bTXe32f68t7V/WJ7muvXEembtWK712tq3Plgd68iitVc1XGM1v/61/qP+s/6rgK6ulD6PrcpT//0HMwBitg==</latexit>
wi
<latexit sha1_base64="q3G0ZobyNR/xFER1uLu8ddVBM3k=">AAADeHicfVJLb9NAEN7GPIp5tIUjF0NU8RCKklCpHCvgwAVRJNJWylrReD12Vt2HtbtOG1n+DVzhp/FXOLF2AqqbipG8/jwPzzffTlIIbt1w+GurF9y6fefu9r3w/oOHj3Z29x6fWF0ahhOmhTZnCVgUXOHEcSfwrDAIMhF4mpx/aOKnCzSWa/XNLQuMJeSKZ5yB867Jxazi9Wy3PxwMW4s2wWgN+mRtx7O9XkxTzUqJyjEB1k5Hw8LFFRjHmcA6pKXFAtg55Dj1UIFEG1ct2zra9540yrTxj3JR671aUYG0dikTnynBze31WOO8KTYtXfYurrgqSoeKrRplpYicjprRo5QbZE4sPQBmuOcasTkYYM4LFO5fbTNHsUDXHYTJuLJZ271DKZH+26DCC6alBJW+rmgGkotlihmUwtUVtdlffJM0b9IFL+xapcuVTCEV6Kg2POcKhMDM0ebouv1r7mh7hvQj+rsw+NkT/FKgAaeNZwIml3BZ+7vJ6TPawP9lcvUv08PuWFVLwA/TSKALVFXdQia0RZrkRpdFh/BGfUvU/wAyr/gqH7tlqwy/kKPr67cJTsaD0dvB+OtB/+j9ejW3yVPynLwkI3JIjsgnckwmhBFOvpMf5GfvdxAFL4JXq9Te1rrmCelYMP4DNGgsFw==</latexit>

softmax(
softmax(
<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>

<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>
))

neural network

<latexit sha1_base64="4Ef2fl86UdpH4dmLe8oJRQ2pHaQ=">AAADenicfVJLb9NAEN4mPEp4teXIxTSqxKtRUiKVYwUcuCCKRJpKWSsar8fOKvuwdtchkeUfwRV+Gf+FA2snoLqpGMnrz/PwfPPtRJng1vX7v3Za7Vu379zdvde5/+Dho8d7+wcXVueG4Yhpoc1lBBYFVzhy3Am8zAyCjASOo/n7Kj5eoLFcq69ulWEoIVU84Qycd42X04IfD8vpXrff69cWbIPBBnTJxs6n+62QxprlEpVjAqydDPqZCwswjjOBZYfmFjNgc0hx4qECiTYsar5lcOQ9cZBo4x/lgtp7taIAae1KRj5TgpvZ67HKeVNskrvkbVhwleUOFVs3SnIROB1UwwcxN8icWHkAzHDPNWAzMMCcl6hzdLXNDMUCXXMQJsPCJnX3BqVI+m+DCr8xLSWo+GVBE5BcrGJMIBeuLKhN/uKbpHkdL3hmNyot1zJ1qEBHteEpVyAEJo5WR9PtXzNH67NDP6C/C4OfPMHPGRpw2ngmYFIJy9LfTUqf0Qr+L5Orf5keNscqagJ+mEoCnaEqyhoyoS3SKDU6zxqEt+prov4HkHjF1/nYLFtn+IUcXF+/bXBx0hu86Z18GXbP3m1Wc5c8JYfkORmQU3JGPpJzMiKMzMl38oP8bP1uH7ZftF+tU1s7m5onpGHt4R+xkSyN</latexit>
xi 4 <latexit sha1_base64="nB8pK9PGEwV+LmMCOwKc8YP2Au8=">AAADenicfVJLb9NAEN4mPEp4tXDkYhpV4hklbSU4VsCBC6JIpKmUtaLxeuyssg9rdx0SWf4RXOGX8V84sHYMqpuKkbz+PA/PN99OlAlu3XD4a6fTvXHz1u3dO7279+4/eLi3/+jc6twwHDMttLmIwKLgCseOO4EXmUGQkcBJtHhfxSdLNJZr9dWtMwwlpIonnIHzrslqVvDXx+Vsrz8cDGsLtsGoAX3S2NlsvxPSWLNconJMgLXT0TBzYQHGcSaw7NHcYgZsASlOPVQg0YZFzbcMDr0nDhJt/KNcUHsvVxQgrV3LyGdKcHN7NVY5r4tNc5e8DQuustyhYptGSS4Cp4Nq+CDmBpkTaw+AGe65BmwOBpjzEvUOL7eZo1iiaw/CZFjYpO7eohRJ/21Q4TempQQVvyhoApKLdYwJ5MKVBbXJX3ydNK/iJc9so9JqI1OPCnRUG55yBUJg4mh1tN3+NXe0Pnv0A/q7MPjJE/ycoQGnjWcCJpWwKv3dpPQpreD/Mrn6l+lhe6yiJuCHqSTQGaqirCET2iKNUqPzrEV4q74m6n8AiVd8k4/tsk2GX8jR1fXbBudHg9Hx4OjLSf/0XbOau+QJOSDPyIi8IafkIzkjY8LIgnwnP8jPzu/uQfd59+UmtbPT1DwmLeue/AGugCyM</latexit>
xi 3 xi 2
<latexit sha1_base64="Lw5rWS0utPfv8sAZFj6BQfU2/QM=">AAADenicfVJLb9NAEN4mPEp4teXIxTSqxKtREiqVYwUcuCCKRJpKWSsar8fOKvuwdtchkeUfwRV+Gf+FA2snoLqpGMnrz/PwfPPtRJng1vX7v3Za7Vu379zdvde5/+Dho8d7+wcXVueG4Yhpoc1lBBYFVzhy3Am8zAyCjASOo/n7Kj5eoLFcq69ulWEoIVU84Qycd42X04IfD8vpXrff69cWbIPBBnTJxs6n+62QxprlEpVjAqydDPqZCwswjjOBZYfmFjNgc0hx4qECiTYsar5lcOQ9cZBo4x/lgtp7taIAae1KRj5TgpvZ67HKeVNskrvkbVhwleUOFVs3SnIROB1UwwcxN8icWHkAzHDPNWAzMMCcl6hzdLXNDMUCXXMQJsPCJnX3BqVI+m+DCr8xLSWo+GVBE5BcrGJMIBeuLKhN/uKbpHkdL3hmNyot1zJ1qEBHteEpVyAEJo5WR9PtXzNH67NDP6C/C4OfPMHPGRpw2ngmYFIJy9LfTUqf0Qr+L5Orf5keNscqagJ+mEoCnaEqyhoyoS3SKDU6zxqEt+prov4HkHjF1/nYLFtn+IUcXF+/bXAx7A3e9IZfTrpn7zaruUuekkPynAzIKTkjH8k5GRFG5uQ7+UF+tn63D9sv2q/Wqa2dTc0T0rD2yR+rbyyL</latexit>
<latexit sha1_base64="0VBkd1Rw+AkOCuCd8jR6pjBnSJo=">AAADenicfVJLb9NAEN4mPEp4tXDkYhpV4tUoLpXgWAEHLogikaZS1orG67Gzyj6s3XVIZPlHcIVfxn/hwNoJqG4qRvL68zw833w7cS64dcPhr51O98bNW7d37/Tu3rv/4OHe/qNzqwvDcMS00OYiBouCKxw57gRe5AZBxgLH8fx9HR8v0Fiu1Ve3yjGSkCmecgbOu8bLacmPwmq61x8Oho0F2yDcgD7Z2Nl0vxPRRLNConJMgLWTcJi7qATjOBNY9WhhMQc2hwwnHiqQaKOy4VsFh96TBKk2/lEuaLyXK0qQ1q5k7DMluJm9Gqud18UmhUvfRiVXeeFQsXWjtBCB00E9fJBwg8yJlQfADPdcAzYDA8x5iXqHl9vMUCzQtQdhMipt2nRvUYql/zao8BvTUoJKXpQ0BcnFKsEUCuGqktr0L75OmlfJgud2o9JyLVOPCnRUG55xBUJg6mh9tN3+NXO0OXv0A/q7MPjJE/ycowGnjWcCJpOwrPzdZPQpreH/Mrn6l+lhe6yyIeCHqSXQOaqyaiAT2iKNM6OLvEV4q74h6n8AqVd8nY/tsnWGX8jw6vptg/PjQfh6cPzlpH/6brOau+QJOSDPSEjekFPykZyREWFkTr6TH+Rn53f3oPu8+3Kd2tnZ1DwmLeue/AGoXiyK</latexit>
xi 1
… breathless runners approached the
wi <latexit sha1_base64="v4ajhT6LWKsgSBJtJY78BKo4WNs=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1OK+b1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4Dyo6ZxQ==</latexit>
4 wi <latexit sha1_base64="tqdvN1Co9lflZKGA9FYLbBE3SQc=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1OC+X1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4Dx3yZxA==</latexit>
3 wi <latexit sha1_base64="cCatZM/QvTiGjI99S0wQVTzzBoU=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1O6+P1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DxGqZww==</latexit>
2 wi
<latexit sha1_base64="CH1jSvOpKNyBxULr16kKG8gaFmQ=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1Oy+L1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DwViZwg==</latexit>
1 24
 A feed-forward neural language model
■ Fixed-size input (here: 4)

wi
<latexit sha1_base64="q3G0ZobyNR/xFER1uLu8ddVBM3k=">AAADeHicfVJLb9NAEN7GPIp5tIUjF0NU8RCKklCpHCvgwAVRJNJWylrReD12Vt2HtbtOG1n+DVzhp/FXOLF2AqqbipG8/jwPzzffTlIIbt1w+GurF9y6fefu9r3w/oOHj3Z29x6fWF0ahhOmhTZnCVgUXOHEcSfwrDAIMhF4mpx/aOKnCzSWa/XNLQuMJeSKZ5yB867Jxazi9Wy3PxwMW4s2wWgN+mRtx7O9XkxTzUqJyjEB1k5Hw8LFFRjHmcA6pKXFAtg55Dj1UIFEG1ct2zra9540yrTxj3JR671aUYG0dikTnynBze31WOO8KTYtXfYurrgqSoeKrRplpYicjprRo5QbZE4sPQBmuOcasTkYYM4LFO5fbTNHsUDXHYTJuLJZ271DKZH+26DCC6alBJW+rmgGkotlihmUwtUVtdlffJM0b9IFL+xapcuVTCEV6Kg2POcKhMDM0ebouv1r7mh7hvQj+rsw+NkT/FKgAaeNZwIml3BZ+7vJ6TPawP9lcvUv08PuWFVLwA/TSKALVFXdQia0RZrkRpdFh/BGfUvU/wAyr/gqH7tlqwy/kKPr67cJTsaD0dvB+OtB/+j9ejW3yVPynLwkI3JIjsgnckwmhBFOvpMf5GfvdxAFL4JXq9Te1rrmCelYMP4DNGgsFw==</latexit>

softmax(
softmax(
<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>

<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>
))

neural network

<latexit sha1_base64="4Ef2fl86UdpH4dmLe8oJRQ2pHaQ=">AAADenicfVJLb9NAEN4mPEp4teXIxTSqxKtRUiKVYwUcuCCKRJpKWSsar8fOKvuwdtchkeUfwRV+Gf+FA2snoLqpGMnrz/PwfPPtRJng1vX7v3Za7Vu379zdvde5/+Dho8d7+wcXVueG4Yhpoc1lBBYFVzhy3Am8zAyCjASOo/n7Kj5eoLFcq69ulWEoIVU84Qycd42X04IfD8vpXrff69cWbIPBBnTJxs6n+62QxprlEpVjAqydDPqZCwswjjOBZYfmFjNgc0hx4qECiTYsar5lcOQ9cZBo4x/lgtp7taIAae1KRj5TgpvZ67HKeVNskrvkbVhwleUOFVs3SnIROB1UwwcxN8icWHkAzHDPNWAzMMCcl6hzdLXNDMUCXXMQJsPCJnX3BqVI+m+DCr8xLSWo+GVBE5BcrGJMIBeuLKhN/uKbpHkdL3hmNyot1zJ1qEBHteEpVyAEJo5WR9PtXzNH67NDP6C/C4OfPMHPGRpw2ngmYFIJy9LfTUqf0Qr+L5Orf5keNscqagJ+mEoCnaEqyhoyoS3SKDU6zxqEt+prov4HkHjF1/nYLFtn+IUcXF+/bXBx0hu86Z18GXbP3m1Wc5c8JYfkORmQU3JGPpJzMiKMzMl38oP8bP1uH7ZftF+tU1s7m5onpGHt4R+xkSyN</latexit>
xi 4 <latexit sha1_base64="nB8pK9PGEwV+LmMCOwKc8YP2Au8=">AAADenicfVJLb9NAEN4mPEp4tXDkYhpV4hklbSU4VsCBC6JIpKmUtaLxeuyssg9rdx0SWf4RXOGX8V84sHYMqpuKkbz+PA/PN99OlAlu3XD4a6fTvXHz1u3dO7279+4/eLi3/+jc6twwHDMttLmIwKLgCseOO4EXmUGQkcBJtHhfxSdLNJZr9dWtMwwlpIonnIHzrslqVvDXx+Vsrz8cDGsLtsGoAX3S2NlsvxPSWLNconJMgLXT0TBzYQHGcSaw7NHcYgZsASlOPVQg0YZFzbcMDr0nDhJt/KNcUHsvVxQgrV3LyGdKcHN7NVY5r4tNc5e8DQuustyhYptGSS4Cp4Nq+CDmBpkTaw+AGe65BmwOBpjzEvUOL7eZo1iiaw/CZFjYpO7eohRJ/21Q4TempQQVvyhoApKLdYwJ5MKVBbXJX3ydNK/iJc9so9JqI1OPCnRUG55yBUJg4mh1tN3+NXe0Pnv0A/q7MPjJE/ycoQGnjWcCJpWwKv3dpPQpreD/Mrn6l+lhe6yiJuCHqSTQGaqirCET2iKNUqPzrEV4q74m6n8AiVd8k4/tsk2GX8jR1fXbBudHg9Hx4OjLSf/0XbOau+QJOSDPyIi8IafkIzkjY8LIgnwnP8jPzu/uQfd59+UmtbPT1DwmLeue/AGugCyM</latexit>
xi 3 xi 2
<latexit sha1_base64="Lw5rWS0utPfv8sAZFj6BQfU2/QM=">AAADenicfVJLb9NAEN4mPEp4teXIxTSqxKtREiqVYwUcuCCKRJpKWSsar8fOKvuwdtchkeUfwRV+Gf+FA2snoLqpGMnrz/PwfPPtRJng1vX7v3Za7Vu379zdvde5/+Dho8d7+wcXVueG4Yhpoc1lBBYFVzhy3Am8zAyCjASOo/n7Kj5eoLFcq69ulWEoIVU84Qycd42X04IfD8vpXrff69cWbIPBBnTJxs6n+62QxprlEpVjAqydDPqZCwswjjOBZYfmFjNgc0hx4qECiTYsar5lcOQ9cZBo4x/lgtp7taIAae1KRj5TgpvZ67HKeVNskrvkbVhwleUOFVs3SnIROB1UwwcxN8icWHkAzHDPNWAzMMCcl6hzdLXNDMUCXXMQJsPCJnX3BqVI+m+DCr8xLSWo+GVBE5BcrGJMIBeuLKhN/uKbpHkdL3hmNyot1zJ1qEBHteEpVyAEJo5WR9PtXzNH67NDP6C/C4OfPMHPGRpw2ngmYFIJy9LfTUqf0Qr+L5Orf5keNscqagJ+mEoCnaEqyhoyoS3SKDU6zxqEt+prov4HkHjF1/nYLFtn+IUcXF+/bXAx7A3e9IZfTrpn7zaruUuekkPynAzIKTkjH8k5GRFG5uQ7+UF+tn63D9sv2q/Wqa2dTc0T0rD2yR+rbyyL</latexit>
<latexit sha1_base64="0VBkd1Rw+AkOCuCd8jR6pjBnSJo=">AAADenicfVJLb9NAEN4mPEp4tXDkYhpV4tUoLpXgWAEHLogikaZS1orG67Gzyj6s3XVIZPlHcIVfxn/hwNoJqG4qRvL68zw833w7cS64dcPhr51O98bNW7d37/Tu3rv/4OHe/qNzqwvDcMS00OYiBouCKxw57gRe5AZBxgLH8fx9HR8v0Fiu1Ve3yjGSkCmecgbOu8bLacmPwmq61x8Oho0F2yDcgD7Z2Nl0vxPRRLNConJMgLWTcJi7qATjOBNY9WhhMQc2hwwnHiqQaKOy4VsFh96TBKk2/lEuaLyXK0qQ1q5k7DMluJm9Gqud18UmhUvfRiVXeeFQsXWjtBCB00E9fJBwg8yJlQfADPdcAzYDA8x5iXqHl9vMUCzQtQdhMipt2nRvUYql/zao8BvTUoJKXpQ0BcnFKsEUCuGqktr0L75OmlfJgud2o9JyLVOPCnRUG55xBUJg6mh9tN3+NXO0OXv0A/q7MPjJE/ycowGnjWcCJpOwrPzdZPQpreH/Mrn6l+lhe6yyIeCHqSXQOaqyaiAT2iKNM6OLvEV4q74h6n8AqVd8nY/tsnWGX8jw6vptg/PjQfh6cPzlpH/6brOau+QJOSDPSEjekFPykZyREWFkTr6TH+Rn53f3oPu8+3Kd2tnZ1DwmLeue/AGoXiyK</latexit>
xi 1
… breathless runners approached the
wi <latexit sha1_base64="v4ajhT6LWKsgSBJtJY78BKo4WNs=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1OK+b1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4Dyo6ZxQ==</latexit>
4 wi <latexit sha1_base64="tqdvN1Co9lflZKGA9FYLbBE3SQc=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1OC+X1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4Dx3yZxA==</latexit>
3 wi <latexit sha1_base64="cCatZM/QvTiGjI99S0wQVTzzBoU=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1O6+P1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DxGqZww==</latexit>
2 wi
<latexit sha1_base64="CH1jSvOpKNyBxULr16kKG8gaFmQ=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1Oy+L1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DwViZwg==</latexit>
1 25
 A feed-forward neural language model
■ Concatenate k word embeddings: x = [xi 4 ; xi 3 ; xi 2 ; xi 1 ] <latexit sha1_base64="GCeRbBMVwxUC2bgQlz05ZqIBrJg=">AAADonicfVLbbtNAEN3GXEq4NIVHXgyhEkI0stNKICGkCnhASIhWkLZS1orW67Gz6l6s3XVIZPkr+Bpe4Sv4G9ZOguqmYiSvj+fiOXN24pwzY4Pgz1bHu3Hz1u3tO9279+4/2OntPjw1qtAURlRxpc9jYoAzCSPLLIfzXAMRMYez+OJ9HT+bgTZMyW92kUMkSCZZyiixzjXp7eNYlPPKf+uP55OS7R9Wb/wGHKzBcA3CKpr0+sEgaMzfBOEK9NHKjie7nQgnihYCpKWcGDMOg9xGJdGWUQ5VFxcGckIvSAZjByURYKKymavy95wn8VOl3SOt33gvV5REGLMQscsUxE7N1VjtvC42Lmz6OiqZzAsLki4bpQX3rfJrkfyEaaCWLxwgVDPH1adTogm1Tsru3uU2U+AzsO1BqIhKkzbdW5Ri4b41SPhOlRBEJi9KnBLB+CKBlBTcViU26RpfJ83LZMZys1JpvpSpizlYrDTLmCScQ2pxfbTd7jW1uDm7+AO4u9Dw2RH8koMmVmnHhOhMkHnl7ibDT3AN/5fJ5L9MB9tjlQ0BN0wtgcpBllUDKVcGcJxpVeQtwhv1DVH3A5I6xZf50C5bZriFDK+u3yY4HQ7Cg8Hw5LB/9G61mtvoMXqKnqMQvUJH6CM6RiNE0Q/0E/1Cv71n3ifvxPu6TO1srWoeoZZ5+C/9kDnh</latexit>

wi
<latexit sha1_base64="q3G0ZobyNR/xFER1uLu8ddVBM3k=">AAADeHicfVJLb9NAEN7GPIp5tIUjF0NU8RCKklCpHCvgwAVRJNJWylrReD12Vt2HtbtOG1n+DVzhp/FXOLF2AqqbipG8/jwPzzffTlIIbt1w+GurF9y6fefu9r3w/oOHj3Z29x6fWF0ahhOmhTZnCVgUXOHEcSfwrDAIMhF4mpx/aOKnCzSWa/XNLQuMJeSKZ5yB867Jxazi9Wy3PxwMW4s2wWgN+mRtx7O9XkxTzUqJyjEB1k5Hw8LFFRjHmcA6pKXFAtg55Dj1UIFEG1ct2zra9540yrTxj3JR671aUYG0dikTnynBze31WOO8KTYtXfYurrgqSoeKrRplpYicjprRo5QbZE4sPQBmuOcasTkYYM4LFO5fbTNHsUDXHYTJuLJZ271DKZH+26DCC6alBJW+rmgGkotlihmUwtUVtdlffJM0b9IFL+xapcuVTCEV6Kg2POcKhMDM0ebouv1r7mh7hvQj+rsw+NkT/FKgAaeNZwIml3BZ+7vJ6TPawP9lcvUv08PuWFVLwA/TSKALVFXdQia0RZrkRpdFh/BGfUvU/wAyr/gqH7tlqwy/kKPr67cJTsaD0dvB+OtB/+j9ejW3yVPynLwkI3JIjsgnckwmhBFOvpMf5GfvdxAFL4JXq9Te1rrmCelYMP4DNGgsFw==</latexit>

softmax(
softmax(
<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>

<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>
))

neural network

<latexit sha1_base64="4Ef2fl86UdpH4dmLe8oJRQ2pHaQ=">AAADenicfVJLb9NAEN4mPEp4teXIxTSqxKtRUiKVYwUcuCCKRJpKWSsar8fOKvuwdtchkeUfwRV+Gf+FA2snoLqpGMnrz/PwfPPtRJng1vX7v3Za7Vu379zdvde5/+Dho8d7+wcXVueG4Yhpoc1lBBYFVzhy3Am8zAyCjASOo/n7Kj5eoLFcq69ulWEoIVU84Qycd42X04IfD8vpXrff69cWbIPBBnTJxs6n+62QxprlEpVjAqydDPqZCwswjjOBZYfmFjNgc0hx4qECiTYsar5lcOQ9cZBo4x/lgtp7taIAae1KRj5TgpvZ67HKeVNskrvkbVhwleUOFVs3SnIROB1UwwcxN8icWHkAzHDPNWAzMMCcl6hzdLXNDMUCXXMQJsPCJnX3BqVI+m+DCr8xLSWo+GVBE5BcrGJMIBeuLKhN/uKbpHkdL3hmNyot1zJ1qEBHteEpVyAEJo5WR9PtXzNH67NDP6C/C4OfPMHPGRpw2ngmYFIJy9LfTUqf0Qr+L5Orf5keNscqagJ+mEoCnaEqyhoyoS3SKDU6zxqEt+prov4HkHjF1/nYLFtn+IUcXF+/bXBx0hu86Z18GXbP3m1Wc5c8JYfkORmQU3JGPpJzMiKMzMl38oP8bP1uH7ZftF+tU1s7m5onpGHt4R+xkSyN</latexit>
xi 4 <latexit sha1_base64="nB8pK9PGEwV+LmMCOwKc8YP2Au8=">AAADenicfVJLb9NAEN4mPEp4tXDkYhpV4hklbSU4VsCBC6JIpKmUtaLxeuyssg9rdx0SWf4RXOGX8V84sHYMqpuKkbz+PA/PN99OlAlu3XD4a6fTvXHz1u3dO7279+4/eLi3/+jc6twwHDMttLmIwKLgCseOO4EXmUGQkcBJtHhfxSdLNJZr9dWtMwwlpIonnIHzrslqVvDXx+Vsrz8cDGsLtsGoAX3S2NlsvxPSWLNconJMgLXT0TBzYQHGcSaw7NHcYgZsASlOPVQg0YZFzbcMDr0nDhJt/KNcUHsvVxQgrV3LyGdKcHN7NVY5r4tNc5e8DQuustyhYptGSS4Cp4Nq+CDmBpkTaw+AGe65BmwOBpjzEvUOL7eZo1iiaw/CZFjYpO7eohRJ/21Q4TempQQVvyhoApKLdYwJ5MKVBbXJX3ydNK/iJc9so9JqI1OPCnRUG55yBUJg4mh1tN3+NXe0Pnv0A/q7MPjJE/ycoQGnjWcCJpWwKv3dpPQpreD/Mrn6l+lhe6yiJuCHqSTQGaqirCET2iKNUqPzrEV4q74m6n8AiVd8k4/tsk2GX8jR1fXbBudHg9Hx4OjLSf/0XbOau+QJOSDPyIi8IafkIzkjY8LIgnwnP8jPzu/uQfd59+UmtbPT1DwmLeue/AGugCyM</latexit>
xi 3 xi 2
<latexit sha1_base64="Lw5rWS0utPfv8sAZFj6BQfU2/QM=">AAADenicfVJLb9NAEN4mPEp4teXIxTSqxKtREiqVYwUcuCCKRJpKWSsar8fOKvuwdtchkeUfwRV+Gf+FA2snoLqpGMnrz/PwfPPtRJng1vX7v3Za7Vu379zdvde5/+Dho8d7+wcXVueG4Yhpoc1lBBYFVzhy3Am8zAyCjASOo/n7Kj5eoLFcq69ulWEoIVU84Qycd42X04IfD8vpXrff69cWbIPBBnTJxs6n+62QxprlEpVjAqydDPqZCwswjjOBZYfmFjNgc0hx4qECiTYsar5lcOQ9cZBo4x/lgtp7taIAae1KRj5TgpvZ67HKeVNskrvkbVhwleUOFVs3SnIROB1UwwcxN8icWHkAzHDPNWAzMMCcl6hzdLXNDMUCXXMQJsPCJnX3BqVI+m+DCr8xLSWo+GVBE5BcrGJMIBeuLKhN/uKbpHkdL3hmNyot1zJ1qEBHteEpVyAEJo5WR9PtXzNH67NDP6C/C4OfPMHPGRpw2ngmYFIJy9LfTUqf0Qr+L5Orf5keNscqagJ+mEoCnaEqyhoyoS3SKDU6zxqEt+prov4HkHjF1/nYLFtn+IUcXF+/bXAx7A3e9IZfTrpn7zaruUuekkPynAzIKTkjH8k5GRFG5uQ7+UF+tn63D9sv2q/Wqa2dTc0T0rD2yR+rbyyL</latexit>
<latexit sha1_base64="0VBkd1Rw+AkOCuCd8jR6pjBnSJo=">AAADenicfVJLb9NAEN4mPEp4tXDkYhpV4tUoLpXgWAEHLogikaZS1orG67Gzyj6s3XVIZPlHcIVfxn/hwNoJqG4qRvL68zw833w7cS64dcPhr51O98bNW7d37/Tu3rv/4OHe/qNzqwvDcMS00OYiBouCKxw57gRe5AZBxgLH8fx9HR8v0Fiu1Ve3yjGSkCmecgbOu8bLacmPwmq61x8Oho0F2yDcgD7Z2Nl0vxPRRLNConJMgLWTcJi7qATjOBNY9WhhMQc2hwwnHiqQaKOy4VsFh96TBKk2/lEuaLyXK0qQ1q5k7DMluJm9Gqud18UmhUvfRiVXeeFQsXWjtBCB00E9fJBwg8yJlQfADPdcAzYDA8x5iXqHl9vMUCzQtQdhMipt2nRvUYql/zao8BvTUoJKXpQ0BcnFKsEUCuGqktr0L75OmlfJgud2o9JyLVOPCnRUG55xBUJg6mh9tN3+NXO0OXv0A/q7MPjJE/ycowGnjWcCJpOwrPzdZPQpreH/Mrn6l+lhe6yyIeCHqSXQOaqyaiAT2iKNM6OLvEV4q74h6n8AqVd8nY/tsnWGX8jw6vptg/PjQfh6cPzlpH/6brOau+QJOSDPSEjekFPykZyREWFkTr6TH+Rn53f3oPu8+3Kd2tnZ1DwmLeue/AGoXiyK</latexit>
xi 1
breathless runners approached the
wi <latexit sha1_base64="v4ajhT6LWKsgSBJtJY78BKo4WNs=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1OK+b1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4Dyo6ZxQ==</latexit>
4 wi <latexit sha1_base64="tqdvN1Co9lflZKGA9FYLbBE3SQc=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1OC+X1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4Dx3yZxA==</latexit>
3 wi <latexit sha1_base64="cCatZM/QvTiGjI99S0wQVTzzBoU=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1O6+P1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DxGqZww==</latexit>
2 wi
<latexit sha1_base64="CH1jSvOpKNyBxULr16kKG8gaFmQ=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1Oy+L1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DwViZwg==</latexit>
1 26
 A feed-forward neural language model
■ Concatenate k word embeddings: x = [xi 4 ; xi 3 ; xi 2 ; xi 1 ] <latexit sha1_base64="GCeRbBMVwxUC2bgQlz05ZqIBrJg=">AAADonicfVLbbtNAEN3GXEq4NIVHXgyhEkI0stNKICGkCnhASIhWkLZS1orW67Gz6l6s3XVIZPkr+Bpe4Sv4G9ZOguqmYiSvj+fiOXN24pwzY4Pgz1bHu3Hz1u3tO9279+4/2OntPjw1qtAURlRxpc9jYoAzCSPLLIfzXAMRMYez+OJ9HT+bgTZMyW92kUMkSCZZyiixzjXp7eNYlPPKf+uP55OS7R9Wb/wGHKzBcA3CKpr0+sEgaMzfBOEK9NHKjie7nQgnihYCpKWcGDMOg9xGJdGWUQ5VFxcGckIvSAZjByURYKKymavy95wn8VOl3SOt33gvV5REGLMQscsUxE7N1VjtvC42Lmz6OiqZzAsLki4bpQX3rfJrkfyEaaCWLxwgVDPH1adTogm1Tsru3uU2U+AzsO1BqIhKkzbdW5Ri4b41SPhOlRBEJi9KnBLB+CKBlBTcViU26RpfJ83LZMZys1JpvpSpizlYrDTLmCScQ2pxfbTd7jW1uDm7+AO4u9Dw2RH8koMmVmnHhOhMkHnl7ibDT3AN/5fJ5L9MB9tjlQ0BN0wtgcpBllUDKVcGcJxpVeQtwhv1DVH3A5I6xZf50C5bZriFDK+u3yY4HQ7Cg8Hw5LB/9G61mtvoMXqKnqMQvUJH6CM6RiNE0Q/0E/1Cv71n3ifvxPu6TO1srWoeoZZ5+C/9kDnh</latexit>

w
■ Hidden layer: f (⇥ (x!z)
x) i
<latexit sha1_base64="2f/0bqNiHOpTM4YcPjA05cCkJt4=">AAAD+3icfVLNbhMxEHYafsryl5ZjOSxElVIEUZJWAgkhVcCBC6JITVspXiKvM5tY9c/K9oYEa6+8CDfElYfhHXgIvJsEdZvCrNb77XjG883niVPOjO10ftU26teu37i5eSu4fefuvfuNre0TozJNoU8VV/osJgY4k9C3zHI4SzUQEXM4jc/fFPunU9CGKXls5ylEgowlSxgl1ruGja+7OBZuloevwsFs6Nizg/xlWIL9FeitQDePgl18PAFLPrnWDFsVftnLvUsQO9HCGZVYQWZ5C4f/evaCpLV2wmwvGDaanXantHAddJegiZZ2NNzaiPBI0UyAtJQTYwbdTmojR7RllEMe4MxASug5GcPAQ0kEmMiVguXhrveMwkRp/0oblt6LGY4IY+Yi9pFFb+byXuG8am+Q2eRF5JhMMwuSLgolGQ99o4X64YhpoJbPPSBUM881pBOiCbX+jrySF8pMgE/BVhuhInImKatXKMXC/2uQ8JkqIYgcPXE4IYLx+QgSknGbO2ySFb5KmqejKUvNUqXZQqYAc7BYaTZmknAOicXFUnX7z8Ticg3wW/B3oeG9J/ghBU2s0p4J0eNiLPzdjPEjXMD/RTL5N9LDaluuJOCbKSRQKUiXl5ByZQDHY62ytEJ4Lb8k6g8giVd8EQ/VtEWEH8ju5fFbBye9dne/3ft40Dx8vRzNTbSDHqMW6qLn6BC9Q0eojyj6Xduu7dQe1vP6t/r3+o9F6EZtmfMAVaz+8w+RO1Y0</latexit>
<latexit sha1_base64="q3G0ZobyNR/xFER1uLu8ddVBM3k=">AAADeHicfVJLb9NAEN7GPIp5tIUjF0NU8RCKklCpHCvgwAVRJNJWylrReD12Vt2HtbtOG1n+DVzhp/FXOLF2AqqbipG8/jwPzzffTlIIbt1w+GurF9y6fefu9r3w/oOHj3Z29x6fWF0ahhOmhTZnCVgUXOHEcSfwrDAIMhF4mpx/aOKnCzSWa/XNLQuMJeSKZ5yB867Jxazi9Wy3PxwMW4s2wWgN+mRtx7O9XkxTzUqJyjEB1k5Hw8LFFRjHmcA6pKXFAtg55Dj1UIFEG1ct2zra9540yrTxj3JR671aUYG0dikTnynBze31WOO8KTYtXfYurrgqSoeKrRplpYicjprRo5QbZE4sPQBmuOcasTkYYM4LFO5fbTNHsUDXHYTJuLJZ271DKZH+26DCC6alBJW+rmgGkotlihmUwtUVtdlffJM0b9IFL+xapcuVTCEV6Kg2POcKhMDM0ebouv1r7mh7hvQj+rsw+NkT/FKgAaeNZwIml3BZ+7vJ6TPawP9lcvUv08PuWFVLwA/TSKALVFXdQia0RZrkRpdFh/BGfUvU/wAyr/gqH7tlqwy/kKPr67cJTsaD0dvB+OtB/+j9ejW3yVPynLwkI3JIjsgnckwmhBFOvpMf5GfvdxAFL4JXq9Te1rrmCelYMP4DNGgsFw==</latexit>

dims:

softmax( )
softmax( ) [V]
■ Output layer: SoftMax(⇥(z!y ) z)
<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>

<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>

(z!y ) [V x dz]
<latexit sha1_base64="UxC4Ej1oFjZxmUwU7TnbcbVZItM=">AAAEIXicfVJbb9MwFE5XLiPcuvHIS6Cq1CKo2m4SkxDSGDzwMjHEblIdKsc9aa35EtlOaWvl1/DKH+EN8Yb4Mzhpi5Z14CjOl+PvHH/nEiWMatPp/KpsVG/cvHV7845/9979Bw9rW9unWqaKwAmRTKrzCGtgVMCJoYbBeaIA84jBWXTxNj8/m4DSVIpjM0sg5HgkaEwJNs40qH1roIjbaRa8DvrTgaUvdrNXQQF2VqC3At0s9BvoeAwGf7bNKTIymLcyZ+LYjBW3WsaG42nWRMG/npbfiJtrIaYtfxXjk4txWMRYkeY5adbK5i1/UKt32p1iBeuguwR1b7mOBlsbIRpKknIQhjCsdb/bSUxosTKUMMh8lGpIMLnAI+g7KDAHHdqirFnQcJZhEEvlXmGCwnrZw2Ku9YxHjpmr11fPcuN1Z/3UxHuhpSJJDQiyuChOWeASzXsUDKkCYtjMAUwUdVoDMsYKE+M66ep96ZoxsAmYciKEh1bHxe0lSRF3/woEfCGScyyGzyyKMadsNoQYp8xkFul4ha8rzfPhhCZ6WaXpokw+YmCQVHREBWYMYoPyrWx2n7FBxe6jd+B6oeDQCfyQgMJGKqcEq1E+PK43I/QE5fB/TCr+Mh0sp2ULAS6ZvAQyAWGzAhImNaBopGSalASv+RdCXQAcu4ov+FB2WzDcQHavjt86OO21uzvt3sfd+v7BcjQ3vcfeU6/pdb2X3r733jvyTjxS2a7sVd5UDqpfq9+rP6o/F9SNytLnkVda1d9/AELgZJU=</latexit>

⇥
<latexit sha1_base64="E8I2t4Oi7kYatxSgWbJwMVzGCPk=">AAAD93icfVLLbhMxFHUSHiVASWEBEpuBKFKKIErSSiAhpApYsEEUqS8pHiKPcyex6sfI9oSk1mz4EXaILZ/DH/AZeCZJ1WkKHo195vqe8bnHN0o4M7bb/V2p1q5dv3Fz41b99p27m/caW/ePjEo1hUOquNInETHAmYRDyyyHk0QDERGH4+j0Xb5/PAVtmJIHdp5AKMhYsphRYn1o2PjWwpFwsyx4EwxmQ8de7GavgwLsrEB/BXpZWG/hgwlY8sW1Z9iq4Gw78yFB7EQLZ1RsBZllbRz869mun/PPcv7c84eNZrfTLUawDnpL0ETLsT/cqoZ4pGgqQFrKiTGDXjexoSPaMsohq+PUQELoKRnDwENJBJjQFWZlQctHRkGstH+lDYroRYYjwpi5iHxmXpe5vJcHr9obpDZ+FTomk9SCpIuD4pQHvszc+WDENFDL5x4QqpnXGtAJ0YRafz/exQvHTIBPwZYLoSJ0Ji5OL0mKhP/WIOErVUIQOXrmcEwE4/MRxCTlNnPYxCt8lTXPR1OWmKVLs4VNdczBYqXZmEnCOcQW51M57JeJxcVcx+/B34WGj17gpwQ0sUp7JUSP85bwdzPGT3AO/5fJ5Hmmh+WyXCHAF5NboBKQLisg5coAjsZapUlJ8Bq/EOp/QGLv+CIfyrRFhm/I3uX2WwdH/U5vp9P/vNvce7tszQ30GD1FbdRDL9Ee+oD20SGi6E9ls/Kw8qg2r32v/aj9XKRWK0vOA1QatV9/Aa5aVN4=</latexit>

(z!y )
exp(✓j · z)
z
<latexit sha1_base64="OAPHFDGngIAcmszwYq5hmx7mWO4=">AAAEJHicfVJbb9MwFE5XLiPcNhBPvASqSi1iU9tNAmlCmmBIvEwMsW6T6lA57klrzZfIdkpbK3+HV/4Ib4gHXvgtOGkLyzpwFOfL8XeOv3OJEka1abV+Vtaq167fuLl+y7995+69+xubD060TBWBLpFMqrMIa2BUQNdQw+AsUYB5xOA0On+Tn5+OQWkqxbGZJhByPBQ0pgQbZ+pvfK2jiNtJFrwKepO+pVu72V5QgJ0l6CxBOwv9OjoegcGfbGOCjAxmzcyZODYjxa2WseF4kjVQ8K+n6dfjxkqISfNvkI8uyGERZMma5axpM5s1/Znf36i1tlvFClZBewFq3mId9TfXQjSQJOUgDGFY6167lZjQYmUoYZD5KNWQYHKOh9BzUGAOOrRFZbOg7iyDIJbKvcIEhfWih8Vc6ymPHDPXry+f5carznqpiV+GlookNSDI/KI4ZYFLNW9TMKAKiGFTBzBR1GkNyAgrTIxrpqvWhWtGwMZgyokQHlodF7eXJEXc/SsQ8JlIzrEYPLMoxpyy6QBinDKTWaTjJb6qNM8HY5roRZUm8zL5iIFBUtEhFZgxiA3Kt7LZfUYGFbuPDsD1QsGhE/g+AYWNVE4JVsN8flxvhugJyuH/mFT8YTpYTssWAlwyeQlkAsJmBSRMakDRUMk0KQle8S+EugA4dhWf86HsNme4gWxfHr9VcNLZbu9sdz7s1vZfL0Zz3XvsPfUaXtt74e1777wjr+uRyqPKXuWg8rb6pfqt+r36Y05dqyx8HnqlVf31G2ObZVw=</latexit>

[dz]
P(wi = j | z) = P (x!z)
(z!y )
exp(✓j 0 · z) ⇥
<latexit sha1_base64="oCCC6lR82ch2B7kQhLCX1aCX53M=">AAADtXicfVLbbtNAEN3GXIq5pfDIiyGKVBBEcVqpSAgpAh54QRSpaStlTbRej5NV92LtrkOC5d/ha3gFib9h7SSobipG8vp4Lp4zZyfOODO23/+z0/Ju3Lx1e/eOf/fe/QcP23uPTo3KNYURVVzp85gY4EzCyDLL4TzTQETM4Sy+eF/Fz+agDVPyxC4ziASZSpYySqxzTdrDLo5FsSiDt8F4MSnYq8PyTVCDgw0YbEBYRj4+mYElX4v9BbYq+P68nLQ7/V6/tmAbhGvQQWs7nuy1IpwomguQlnJizDjsZzYqiLaMcih9nBvICL0gUxg7KIkAExX1qGXQdZ4kSJV2j7RB7b1cURBhzFLELlMQOzNXY5Xzutg4t+nrqGAyyy1IumqU5jxwU1a6BQnTQC1fOkCoZo5rQGdEE2qdun73cpsZ8DnY5iBURIVJ6+4NSrFw3xokfKNKCCKTFwVOiWB8mUBKcm7LApt0g6+T5mUyZ5lZq7RYyeRjDhYrzaZMEs4htbg6mm73mllcnz7+AO4uNHxyBD9noIlV2jEheirIonR3M8VPcQX/l8nkv0wHm2MVNQE3TCWBykAWZQ0pVwZwPNUqzxqEt+prou4HJHWKr/KhWbbKcAsZXl2/bXA66IUHvcGXw87w3Xo1d9ET9AztoxAdoSH6iI7RCFH0A/1Ev9Bv78iLvMRLV6mtnXXNY9QwT/0FBYlBGA==</latexit>
[dz x 4dword]
j 0 2V
<latexit sha1_base64="RIydrX8x0WCAtU63ar9UQ6SJBu4=">AAAEsnicfVJbb9MwFE5HgRFuGzzyEpgqWgRTu00CCZAm4IGXiSF2QcylcpyT1psdR7azpbXya/g1vMIT/4aTtIV2LTiK8+X4O8ffuYSp4Ma2279qK1fqV69dX73h37x1+87dtfV7R0ZlmsEhU0LpzyE1IHgCh5ZbAZ9TDVSGAo7Ds7fl+fE5aMNVcmCHKXQl7Sc85oxaNPXWa68aJJQuL4LXwUnec/zZTvEyqMD2FGxNQafo+g1yMABLv7pmTqwKRq0CTZLagZbOqNhKmhdNEvzrafmNuLkQIm/9DfIJg+xVQaasUckatooRskZ+I1+mYb950eOYw2lAJI/QhJjEmjJHIE+bxJYOvdOZaIRFypa+jphMEsElt6bnTh8TngRHxawbGotlnn5vbaO92a5WsAg6E7DhTdZ+b32lSyLFMgmJZYIac9Jpp7brqLacCSh8khlIKTujfThBmFAJpuuqPhdBAy1RECuNb2KDyjrr4ag0ZihDZJbFNJfPSuOys5PMxi+6jidpZiFh44viTASYbzk0QcQ1MCuGCCjTHLUGbECxuhZHC9sxc80AxDnY+USY7DoTV7fPSQol/mtI4IIpKWkSPXEkppKLYQQxzYQtexNP8bLSPI3OeWomVcrHZfKJAEuU5n2eUCEgtqTc5s34GVhS7T55B9gLDXso8EMKmlqlUQnV/XKYsTd98pCU8H9MnvxhIpxPy1UCMJmyBCqFxBUVZEIZIGFfqyydE7zgXwnFADTGio/5MO82ZuBAdi6P3yI42trsbG9ufdzZ2H0zGc1V74H3yGt6He+5t+u99/a9Q4/VvtW+137UftZ36l/qtM7G1JXaxOe+N7fq4jcnBZak</latexit>

<latexit sha1_base64="yS9RiScDddTxBdObrErFbfzwarQ=">AAAEJ3icfVJbb9MwFE5XLiPcNnhB4iVQVWoRm9ptEkgINAESe5kYYt0m1aFy3JPWmi+R7Yy0Vn4Qr/wR3hA88k9w0haWdeAozpfj7xx/5xIljGrT6fysrdSvXL12ffWGf/PW7Tt319bvHWmZKgI9IplUJxHWwKiAnqGGwUmiAPOIwXF0+qY4Pz4DpakUh2aSQMjxSNCYEmycabD2tYkibrM8eBn0s4GlGzv5i6AE2wuwtQDdPPSb6HAMBn+yrQwZGUzbuTNxbMaKWy1jw3GWt1Dwr6ftN+PWUois/TfIRxdkvwyyYE0L1qSdTx1r6mf+YK3R2eyUK1gG3TloePN1MFhfCdFQkpSDMIRhrfvdTmJCi5WhhEHuo1RDgskpHkHfQYE56NCWxc2DprMMg1gq9woTlNbzHhZzrSc8cswiBX3xrDBedtZPTfw8tFQkqQFBZhfFKQtctkWngiFVQAybOICJok5rQMZYYWJcP13Bzl0zBnYGppoI4aHVcXl7RVLE3b8CAZ+J5ByL4ROLYswpmwwhxikzuUU6XuDLSvN0eEYTPa9SNiuTjxgYJBUdUYEZg9igYqua3WdsULn76C24XijYdwLfJ6CwkcopwWpUjJDrzQg9QgX8H5OKP0wHq2nZUoBLpiiBTEDYvISESQ0oGimZJhXBS/6lUBcAx67iMz5U3WYMN5Ddi+O3DI62Nrvbm1sfdhq7r+ejueo99B57La/rPfN2vT3vwOt5pPag9qr2rrZX/1L/Vv9e/zGjrtTmPve9yqr/+g2IbWYh</latexit>
x [4dword]
<latexit sha1_base64="4Ef2fl86UdpH4dmLe8oJRQ2pHaQ=">AAADenicfVJLb9NAEN4mPEp4teXIxTSqxKtRUiKVYwUcuCCKRJpKWSsar8fOKvuwdtchkeUfwRV+Gf+FA2snoLqpGMnrz/PwfPPtRJng1vX7v3Za7Vu379zdvde5/+Dho8d7+wcXVueG4Yhpoc1lBBYFVzhy3Am8zAyCjASOo/n7Kj5eoLFcq69ulWEoIVU84Qycd42X04IfD8vpXrff69cWbIPBBnTJxs6n+62QxprlEpVjAqydDPqZCwswjjOBZYfmFjNgc0hx4qECiTYsar5lcOQ9cZBo4x/lgtp7taIAae1KRj5TgpvZ67HKeVNskrvkbVhwleUOFVs3SnIROB1UwwcxN8icWHkAzHDPNWAzMMCcl6hzdLXNDMUCXXMQJsPCJnX3BqVI+m+DCr8xLSWo+GVBE5BcrGJMIBeuLKhN/uKbpHkdL3hmNyot1zJ1qEBHteEpVyAEJo5WR9PtXzNH67NDP6C/C4OfPMHPGRpw2ngmYFIJy9LfTUqf0Qr+L5Orf5keNscqagJ+mEoCnaEqyhoyoS3SKDU6zxqEt+prov4HkHjF1/nYLFtn+IUcXF+/bXBx0hu86Z18GXbP3m1Wc5c8JYfkORmQU3JGPpJzMiKMzMl38oP8bP1uH7ZftF+tU1s7m5onpGHt4R+xkSyN</latexit>
xi 4 <latexit sha1_base64="nB8pK9PGEwV+LmMCOwKc8YP2Au8=">AAADenicfVJLb9NAEN4mPEp4tXDkYhpV4hklbSU4VsCBC6JIpKmUtaLxeuyssg9rdx0SWf4RXOGX8V84sHYMqpuKkbz+PA/PN99OlAlu3XD4a6fTvXHz1u3dO7279+4/eLi3/+jc6twwHDMttLmIwKLgCseOO4EXmUGQkcBJtHhfxSdLNJZr9dWtMwwlpIonnIHzrslqVvDXx+Vsrz8cDGsLtsGoAX3S2NlsvxPSWLNconJMgLXT0TBzYQHGcSaw7NHcYgZsASlOPVQg0YZFzbcMDr0nDhJt/KNcUHsvVxQgrV3LyGdKcHN7NVY5r4tNc5e8DQuustyhYptGSS4Cp4Nq+CDmBpkTaw+AGe65BmwOBpjzEvUOL7eZo1iiaw/CZFjYpO7eohRJ/21Q4TempQQVvyhoApKLdYwJ5MKVBbXJX3ydNK/iJc9so9JqI1OPCnRUG55yBUJg4mh1tN3+NXe0Pnv0A/q7MPjJE/ycoQGnjWcCJpWwKv3dpPQpreD/Mrn6l+lhe6yiJuCHqSTQGaqirCET2iKNUqPzrEV4q74m6n8AiVd8k4/tsk2GX8jR1fXbBudHg9Hx4OjLSf/0XbOau+QJOSDPyIi8IafkIzkjY8LIgnwnP8jPzu/uQfd59+UmtbPT1DwmLeue/AGugCyM</latexit>
xi 3 <latexit sha1_base64="Lw5rWS0utPfv8sAZFj6BQfU2/QM=">AAADenicfVJLb9NAEN4mPEp4teXIxTSqxKtREiqVYwUcuCCKRJpKWSsar8fOKvuwdtchkeUfwRV+Gf+FA2snoLqpGMnrz/PwfPPtRJng1vX7v3Za7Vu379zdvde5/+Dho8d7+wcXVueG4Yhpoc1lBBYFVzhy3Am8zAyCjASOo/n7Kj5eoLFcq69ulWEoIVU84Qycd42X04IfD8vpXrff69cWbIPBBnTJxs6n+62QxprlEpVjAqydDPqZCwswjjOBZYfmFjNgc0hx4qECiTYsar5lcOQ9cZBo4x/lgtp7taIAae1KRj5TgpvZ67HKeVNskrvkbVhwleUOFVs3SnIROB1UwwcxN8icWHkAzHDPNWAzMMCcl6hzdLXNDMUCXXMQJsPCJnX3BqVI+m+DCr8xLSWo+GVBE5BcrGJMIBeuLKhN/uKbpHkdL3hmNyot1zJ1qEBHteEpVyAEJo5WR9PtXzNH67NDP6C/C4OfPMHPGRpw2ngmYFIJy9LfTUqf0Qr+L5Orf5keNscqagJ+mEoCnaEqyhoyoS3SKDU6zxqEt+prov4HkHjF1/nYLFtn+IUcXF+/bXAx7A3e9IZfTrpn7zaruUuekkPynAzIKTkjH8k5GRFG5uQ7+UF+tn63D9sv2q/Wqa2dTc0T0rD2yR+rbyyL</latexit>
xi 2 <latexit sha1_base64="0VBkd1Rw+AkOCuCd8jR6pjBnSJo=">AAADenicfVJLb9NAEN4mPEp4tXDkYhpV4tUoLpXgWAEHLogikaZS1orG67Gzyj6s3XVIZPlHcIVfxn/hwNoJqG4qRvL68zw833w7cS64dcPhr51O98bNW7d37/Tu3rv/4OHe/qNzqwvDcMS00OYiBouCKxw57gRe5AZBxgLH8fx9HR8v0Fiu1Ve3yjGSkCmecgbOu8bLacmPwmq61x8Oho0F2yDcgD7Z2Nl0vxPRRLNConJMgLWTcJi7qATjOBNY9WhhMQc2hwwnHiqQaKOy4VsFh96TBKk2/lEuaLyXK0qQ1q5k7DMluJm9Gqud18UmhUvfRiVXeeFQsXWjtBCB00E9fJBwg8yJlQfADPdcAzYDA8x5iXqHl9vMUCzQtQdhMipt2nRvUYql/zao8BvTUoJKXpQ0BcnFKsEUCuGqktr0L75OmlfJgud2o9JyLVOPCnRUG55xBUJg6mh9tN3+NXO0OXv0A/q7MPjJE/ycowGnjWcCJpOwrPzdZPQpreH/Mrn6l+lhe6yyIeCHqSXQOaqyaiAT2iKNM6OLvEV4q74h6n8AqVd8nY/tsnWGX8jw6vptg/PjQfh6cPzlpH/6brOau+QJOSDPSEjekFPykZyREWFkTr6TH+Rn53f3oPu8+3Kd2tnZ1DwmLeue/AGoXiyK</latexit>
xi 1

breathless runners approached the

wi
<latexit sha1_base64="v4ajhT6LWKsgSBJtJY78BKo4WNs=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1OK+b1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4Dyo6ZxQ==</latexit>
4 wi
<latexit sha1_base64="tqdvN1Co9lflZKGA9FYLbBE3SQc=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1OC+X1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4Dx3yZxA==</latexit>
3 wi
<latexit sha1_base64="cCatZM/QvTiGjI99S0wQVTzzBoU=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1O6+P1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DxGqZww==</latexit>
2 wi
<latexit sha1_base64="CH1jSvOpKNyBxULr16kKG8gaFmQ=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1Oy+L1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DwViZwg==</latexit>
1 27
 A feed-forward neural language model
Good:
■ Input size wi
dims:
<latexit sha1_base64="q3G0ZobyNR/xFER1uLu8ddVBM3k=">AAADeHicfVJLb9NAEN7GPIp5tIUjF0NU8RCKklCpHCvgwAVRJNJWylrReD12Vt2HtbtOG1n+DVzhp/FXOLF2AqqbipG8/jwPzzffTlIIbt1w+GurF9y6fefu9r3w/oOHj3Z29x6fWF0ahhOmhTZnCVgUXOHEcSfwrDAIMhF4mpx/aOKnCzSWa/XNLQuMJeSKZ5yB867Jxazi9Wy3PxwMW4s2wWgN+mRtx7O9XkxTzUqJyjEB1k5Hw8LFFRjHmcA6pKXFAtg55Dj1UIFEG1ct2zra9540yrTxj3JR671aUYG0dikTnynBze31WOO8KTYtXfYurrgqSoeKrRplpYicjprRo5QbZE4sPQBmuOcasTkYYM4LFO5fbTNHsUDXHYTJuLJZ271DKZH+26DCC6alBJW+rmgGkotlihmUwtUVtdlffJM0b9IFL+xapcuVTCEV6Kg2POcKhMDM0ebouv1r7mh7hvQj+rsw+NkT/FKgAaeNZwIml3BZ+7vJ6TPawP9lcvUv08PuWFVLwA/TSKALVFXdQia0RZrkRpdFh/BGfUvU/wAyr/gqH7tlqwy/kKPr67cJTsaD0dvB+OtB/+j9ejW3yVPynLwkI3JIjsgnckwmhBFOvpMf5GfvdxAFL4JXq9Te1rrmCelYMP4DNGgsFw==</latexit>

■ Sparsity (lack thereof) softmax(

<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>
)
softmax(
<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>
) [V]
■ Some sharing of ⇥ (z!y ) [V x dz]
representations across
<latexit sha1_base64="E8I2t4Oi7kYatxSgWbJwMVzGCPk=">AAAD93icfVLLbhMxFHUSHiVASWEBEpuBKFKKIErSSiAhpApYsEEUqS8pHiKPcyex6sfI9oSk1mz4EXaILZ/DH/AZeCZJ1WkKHo195vqe8bnHN0o4M7bb/V2p1q5dv3Fz41b99p27m/caW/ePjEo1hUOquNInETHAmYRDyyyHk0QDERGH4+j0Xb5/PAVtmJIHdp5AKMhYsphRYn1o2PjWwpFwsyx4EwxmQ8de7GavgwLsrEB/BXpZWG/hgwlY8sW1Z9iq4Gw78yFB7EQLZ1RsBZllbRz869mun/PPcv7c84eNZrfTLUawDnpL0ETLsT/cqoZ4pGgqQFrKiTGDXjexoSPaMsohq+PUQELoKRnDwENJBJjQFWZlQctHRkGstH+lDYroRYYjwpi5iHxmXpe5vJcHr9obpDZ+FTomk9SCpIuD4pQHvszc+WDENFDL5x4QqpnXGtAJ0YRafz/exQvHTIBPwZYLoSJ0Ji5OL0mKhP/WIOErVUIQOXrmcEwE4/MRxCTlNnPYxCt8lTXPR1OWmKVLs4VNdczBYqXZmEnCOcQW51M57JeJxcVcx+/B34WGj17gpwQ0sUp7JUSP85bwdzPGT3AO/5fJ5Hmmh+WyXCHAF5NboBKQLisg5coAjsZapUlJ8Bq/EOp/QGLv+CIfyrRFhm/I3uX2WwdH/U5vp9P/vNvce7tszQ30GD1FbdRDL9Ee+oD20SGi6E9ls/Kw8qg2r32v/aj9XKRWK0vOA1QatV9/Aa5aVN4=</latexit>

words z
<latexit sha1_base64="OAPHFDGngIAcmszwYq5hmx7mWO4=">AAAEJHicfVJbb9MwFE5XLiPcNhBPvASqSi1iU9tNAmlCmmBIvEwMsW6T6lA57klrzZfIdkpbK3+HV/4Ib4gHXvgtOGkLyzpwFOfL8XeOv3OJEka1abV+Vtaq167fuLl+y7995+69+xubD060TBWBLpFMqrMIa2BUQNdQw+AsUYB5xOA0On+Tn5+OQWkqxbGZJhByPBQ0pgQbZ+pvfK2jiNtJFrwKepO+pVu72V5QgJ0l6CxBOwv9OjoegcGfbGOCjAxmzcyZODYjxa2WseF4kjVQ8K+n6dfjxkqISfNvkI8uyGERZMma5axpM5s1/Znf36i1tlvFClZBewFq3mId9TfXQjSQJOUgDGFY6167lZjQYmUoYZD5KNWQYHKOh9BzUGAOOrRFZbOg7iyDIJbKvcIEhfWih8Vc6ymPHDPXry+f5carznqpiV+GlookNSDI/KI4ZYFLNW9TMKAKiGFTBzBR1GkNyAgrTIxrpqvWhWtGwMZgyokQHlodF7eXJEXc/SsQ8JlIzrEYPLMoxpyy6QBinDKTWaTjJb6qNM8HY5roRZUm8zL5iIFBUtEhFZgxiA3Kt7LZfUYGFbuPDsD1QsGhE/g+AYWNVE4JVsN8flxvhugJyuH/mFT8YTpYTssWAlwyeQlkAsJmBSRMakDRUMk0KQle8S+EugA4dhWf86HsNme4gWxfHr9VcNLZbu9sdz7s1vZfL0Zz3XvsPfUaXtt74e1777wjr+uRyqPKXuWg8rb6pfqt+r36Y05dqyx8HnqlVf31G2ObZVw=</latexit>

[dz]
Needs improvement: ⇥ (x!z) [dz x 4dword]
■ Still not enough context:
<latexit sha1_base64="oCCC6lR82ch2B7kQhLCX1aCX53M=">AAADtXicfVLbbtNAEN3GXIq5pfDIiyGKVBBEcVqpSAgpAh54QRSpaStlTbRej5NV92LtrkOC5d/ha3gFib9h7SSobipG8vp4Lp4zZyfOODO23/+z0/Ju3Lx1e/eOf/fe/QcP23uPTo3KNYURVVzp85gY4EzCyDLL4TzTQETM4Sy+eF/Fz+agDVPyxC4ziASZSpYySqxzTdrDLo5FsSiDt8F4MSnYq8PyTVCDgw0YbEBYRj4+mYElX4v9BbYq+P68nLQ7/V6/tmAbhGvQQWs7nuy1IpwomguQlnJizDjsZzYqiLaMcih9nBvICL0gUxg7KIkAExX1qGXQdZ4kSJV2j7RB7b1cURBhzFLELlMQOzNXY5Xzutg4t+nrqGAyyy1IumqU5jxwU1a6BQnTQC1fOkCoZo5rQGdEE2qdun73cpsZ8DnY5iBURIVJ6+4NSrFw3xokfKNKCCKTFwVOiWB8mUBKcm7LApt0g6+T5mUyZ5lZq7RYyeRjDhYrzaZMEs4htbg6mm73mllcnz7+AO4uNHxyBD9noIlV2jEheirIonR3M8VPcQX/l8nkv0wHm2MVNQE3TCWBykAWZQ0pVwZwPNUqzxqEt+prou4HJHWKr/KhWbbKcAsZXl2/bXA66IUHvcGXw87w3Xo1d9ET9AztoxAdoSH6iI7RCFH0A/1Ev9Bv78iLvMRLV6mtnXXNY9QwT/0FBYlBGA==</latexit>

larger context grows ⇥ (x!z) <latexit sha1_base64="yS9RiScDddTxBdObrErFbfzwarQ=">AAAEJ3icfVJbb9MwFE5XLiPcNnhB4iVQVWoRm9ptEkgINAESe5kYYt0m1aFy3JPWmi+R7Yy0Vn4Qr/wR3hA88k9w0haWdeAozpfj7xx/5xIljGrT6fysrdSvXL12ffWGf/PW7Tt319bvHWmZKgI9IplUJxHWwKiAnqGGwUmiAPOIwXF0+qY4Pz4DpakUh2aSQMjxSNCYEmycabD2tYkibrM8eBn0s4GlGzv5i6AE2wuwtQDdPPSb6HAMBn+yrQwZGUzbuTNxbMaKWy1jw3GWt1Dwr6ftN+PWUois/TfIRxdkvwyyYE0L1qSdTx1r6mf+YK3R2eyUK1gG3TloePN1MFhfCdFQkpSDMIRhrfvdTmJCi5WhhEHuo1RDgskpHkHfQYE56NCWxc2DprMMg1gq9woTlNbzHhZzrSc8cswiBX3xrDBedtZPTfw8tFQkqQFBZhfFKQtctkWngiFVQAybOICJok5rQMZYYWJcP13Bzl0zBnYGppoI4aHVcXl7RVLE3b8CAZ+J5ByL4ROLYswpmwwhxikzuUU6XuDLSvN0eEYTPa9SNiuTjxgYJBUdUYEZg9igYqua3WdsULn76C24XijYdwLfJ6CwkcopwWpUjJDrzQg9QgX8H5OKP0wHq2nZUoBLpiiBTEDYvISESQ0oGimZJhXBS/6lUBcAx67iMz5U3WYMN5Ddi+O3DI62Nrvbm1sfdhq7r+ejueo99B57La/rPfN2vT3vwOt5pPag9qr2rrZX/1L/Vv9e/zGjrtTmPve9yqr/+g2IbWYh</latexit>

x [4dword]
<latexit sha1_base64="bCdKuwGdUrZ6ar5XAT3GIHrKwSY=">AAAEOnicfVLLbtNAFHUaHsW8WliyMVSREh5R0lYCCSFVwIJNRSuatiJjovHkOhl1HtbMODgZ+c/Y8hFs2SG2fABjJ4G6KYzl8fGdc++c+4gSRrXpdL7V1upXrl67vn7Dv3nr9p27G5v3jrVMFYEekUyq0whrYFRAz1DD4DRRgHnE4CQ6e1Ocn0xAaSrFkZkmEHI8EjSmBBtnGmx8baCI2ywPXgX9bGDps938ZVCCnSXYXoJuHvoNdDQGgz/ZZoaMDGat3Jk4NmPFrZax4TjLmyj419PyG3FzJUTW+hvkgwuyXwZZsmYFa9rKZ4418xuZvyphsLHVaXfKFayC7gJseYt1MNhcC9FQkpSDMIRhrfvdTmJCi5WhhEHuo1RDgskZHkHfQYE56NCW9c6DhrMMg1gq9woTlNbzHhZzrac8cswiK33xrDBedtZPTfwitFQkqQFB5hfFKQtcmkXzgiFVQAybOoCJok5rQMZYYWJci10Nz10zBjYBU02E8NDquLy9Iini7l+BgM9Eco7F8LFFMeaUTYcQ45SZ3CIdL/FlpXk6nNBEL6qUzcvkIwYGSUVHVGDGIDao2Kpm9xkbVO4+eguuFwr2ncD3CShspHJKsBoVU+V6M0IPUQH/x6TiD9PBalq2FOCSKUogExA2LyFhUgOKRkqmSUXwin8p1AXAsav4nA9VtznDDWT34vitguPtdnenvX24u7X3ejGa694D75HX9Lrec2/Pe+cdeD2P1J7UDmsfa/36l/r3+o/6zzl1rbbwue9VVv3Xb+QmbVg=</latexit>

xi 4 xi 3 xi 2 xi 1
linear in n.
<latexit sha1_base64="nB8pK9PGEwV+LmMCOwKc8YP2Au8=">AAADenicfVJLb9NAEN4mPEp4tXDkYhpV4hklbSU4VsCBC6JIpKmUtaLxeuyssg9rdx0SWf4RXOGX8V84sHYMqpuKkbz+PA/PN99OlAlu3XD4a6fTvXHz1u3dO7279+4/eLi3/+jc6twwHDMttLmIwKLgCseOO4EXmUGQkcBJtHhfxSdLNJZr9dWtMwwlpIonnIHzrslqVvDXx+Vsrz8cDGsLtsGoAX3S2NlsvxPSWLNconJMgLXT0TBzYQHGcSaw7NHcYgZsASlOPVQg0YZFzbcMDr0nDhJt/KNcUHsvVxQgrV3LyGdKcHN7NVY5r4tNc5e8DQuustyhYptGSS4Cp4Nq+CDmBpkTaw+AGe65BmwOBpjzEvUOL7eZo1iiaw/CZFjYpO7eohRJ/21Q4TempQQVvyhoApKLdYwJ5MKVBbXJX3ydNK/iJc9so9JqI1OPCnRUG55yBUJg4mh1tN3+NXe0Pnv0A/q7MPjJE/ycoQGnjWcCJpWwKv3dpPQpreD/Mrn6l+lhe6yiJuCHqSTQGaqirCET2iKNUqPzrEV4q74m6n8AiVd8k4/tsk2GX8jR1fXbBudHg9Hx4OjLSf/0XbOau+QJOSDPyIi8IafkIzkjY8LIgnwnP8jPzu/uQfd59+UmtbPT1DwmLeue/AGugCyM</latexit>

<latexit sha1_base64="4Ef2fl86UdpH4dmLe8oJRQ2pHaQ=">AAADenicfVJLb9NAEN4mPEp4teXIxTSqxKtRUiKVYwUcuCCKRJpKWSsar8fOKvuwdtchkeUfwRV+Gf+FA2snoLqpGMnrz/PwfPPtRJng1vX7v3Za7Vu379zdvde5/+Dho8d7+wcXVueG4Yhpoc1lBBYFVzhy3Am8zAyCjASOo/n7Kj5eoLFcq69ulWEoIVU84Qycd42X04IfD8vpXrff69cWbIPBBnTJxs6n+62QxprlEpVjAqydDPqZCwswjjOBZYfmFjNgc0hx4qECiTYsar5lcOQ9cZBo4x/lgtp7taIAae1KRj5TgpvZ67HKeVNskrvkbVhwleUOFVs3SnIROB1UwwcxN8icWHkAzHDPNWAzMMCcl6hzdLXNDMUCXXMQJsPCJnX3BqVI+m+DCr8xLSWo+GVBE5BcrGJMIBeuLKhN/uKbpHkdL3hmNyot1zJ1qEBHteEpVyAEJo5WR9PtXzNH67NDP6C/C4OfPMHPGRpw2ngmYFIJy9LfTUqf0Qr+L5Orf5keNscqagJ+mEoCnaEqyhoyoS3SKDU6zxqEt+prov4HkHjF1/nYLFtn+IUcXF+/bXBx0hu86Z18GXbP3m1Wc5c8JYfkORmQU3JGPpJzMiKMzMl38oP8bP1uH7ZftF+tU1s7m5onpGHt4R+xkSyN</latexit> <latexit sha1_base64="0VBkd1Rw+AkOCuCd8jR6pjBnSJo=">AAADenicfVJLb9NAEN4mPEp4tXDkYhpV4tUoLpXgWAEHLogikaZS1orG67Gzyj6s3XVIZPlHcIVfxn/hwNoJqG4qRvL68zw833w7cS64dcPhr51O98bNW7d37/Tu3rv/4OHe/qNzqwvDcMS00OYiBouCKxw57gRe5AZBxgLH8fx9HR8v0Fiu1Ve3yjGSkCmecgbOu8bLacmPwmq61x8Oho0F2yDcgD7Z2Nl0vxPRRLNConJMgLWTcJi7qATjOBNY9WhhMQc2hwwnHiqQaKOy4VsFh96TBKk2/lEuaLyXK0qQ1q5k7DMluJm9Gqud18UmhUvfRiVXeeFQsXWjtBCB00E9fJBwg8yJlQfADPdcAzYDA8x5iXqHl9vMUCzQtQdhMipt2nRvUYql/zao8BvTUoJKXpQ0BcnFKsEUCuGqktr0L75OmlfJgud2o9JyLVOPCnRUG55xBUJg6mh9tN3+NXO0OXv0A/q7MPjJE/ycowGnjWcCJpOwrPzdZPQpreH/Mrn6l+lhe6yyIeCHqSXQOaqyaiAT2iKNM6OLvEV4q74h6n8AqVd8nY/tsnWGX8jw6vptg/PjQfh6cPzlpH/6brOau+QJOSDPSEjekFPykZyREWFkTr6TH+Rn53f3oPu8+3Kd2tnZ1DwmLeue/AGoXiyK</latexit>

<latexit sha1_base64="Lw5rWS0utPfv8sAZFj6BQfU2/QM=">AAADenicfVJLb9NAEN4mPEp4teXIxTSqxKtREiqVYwUcuCCKRJpKWSsar8fOKvuwdtchkeUfwRV+Gf+FA2snoLqpGMnrz/PwfPPtRJng1vX7v3Za7Vu379zdvde5/+Dho8d7+wcXVueG4Yhpoc1lBBYFVzhy3Am8zAyCjASOo/n7Kj5eoLFcq69ulWEoIVU84Qycd42X04IfD8vpXrff69cWbIPBBnTJxs6n+62QxprlEpVjAqydDPqZCwswjjOBZYfmFjNgc0hx4qECiTYsar5lcOQ9cZBo4x/lgtp7taIAae1KRj5TgpvZ67HKeVNskrvkbVhwleUOFVs3SnIROB1UwwcxN8icWHkAzHDPNWAzMMCcl6hzdLXNDMUCXXMQJsPCJnX3BqVI+m+DCr8xLSWo+GVBE5BcrGJMIBeuLKhN/uKbpHkdL3hmNyot1zJ1qEBHteEpVyAEJo5WR9PtXzNH67NDP6C/C4OfPMHPGRpw2ngmYFIJy9LfTUqf0Qr+L5Orf5keNscqagJ+mEoCnaEqyhoyoS3SKDU6zxqEt+prov4HkHjF1/nYLFtn+IUcXF+/bXAx7A3e9IZfTrpn7zaruUuekkPynAzIKTkjH8k5GRFG5uQ7+UF+tn63D9sv2q/Wqa2dTc0T0rD2yR+rbyyL</latexit>

breathless runners approached the

■ Each(x!z)
xi uses different rows wi 4 wi 3 wi 2 wi 1
of ⇥ . Weights not
<latexit sha1_base64="tqdvN1Co9lflZKGA9FYLbBE3SQc=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1OC+X1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4Dx3yZxA==</latexit>

<latexit sha1_base64="cCatZM/QvTiGjI99S0wQVTzzBoU=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1O6+P1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DxGqZww==</latexit>

<latexit sha1_base64="v4ajhT6LWKsgSBJtJY78BKo4WNs=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1OK+b1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4Dyo6ZxQ==</latexit> <latexit sha1_base64="CH1jSvOpKNyBxULr16kKG8gaFmQ=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1Oy+L1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DwViZwg==</latexit>

<latexit sha1_base64="bCdKuwGdUrZ6ar5XAT3GIHrKwSY=">AAAEOnicfVLLbtNAFHUaHsW8WliyMVSREh5R0lYCCSFVwIJNRSuatiJjovHkOhl1HtbMODgZ+c/Y8hFs2SG2fABjJ4G6KYzl8fGdc++c+4gSRrXpdL7V1upXrl67vn7Dv3nr9p27G5v3jrVMFYEekUyq0whrYFRAz1DD4DRRgHnE4CQ6e1Ocn0xAaSrFkZkmEHI8EjSmBBtnGmx8baCI2ywPXgX9bGDps938ZVCCnSXYXoJuHvoNdDQGgz/ZZoaMDGat3Jk4NmPFrZax4TjLmyj419PyG3FzJUTW+hvkgwuyXwZZsmYFa9rKZ4418xuZvyphsLHVaXfKFayC7gJseYt1MNhcC9FQkpSDMIRhrfvdTmJCi5WhhEHuo1RDgskZHkHfQYE56NCW9c6DhrMMg1gq9woTlNbzHhZzrac8cswiK33xrDBedtZPTfwitFQkqQFB5hfFKQtcmkXzgiFVQAybOoCJok5rQMZYYWJci10Nz10zBjYBU02E8NDquLy9Iini7l+BgM9Eco7F8LFFMeaUTYcQ45SZ3CIdL/FlpXk6nNBEL6qUzcvkIwYGSUVHVGDGIDao2Kpm9xkbVO4+eguuFwr2ncD3CShspHJKsBoVU+V6M0IPUQH/x6TiD9PBalq2FOCSKUogExA2LyFhUgOKRkqmSUXwin8p1AXAsav4nA9VtznDDWT34vitguPtdnenvX24u7X3ejGa694D75HX9Lrec2/Pe+cdeD2P1J7UDmsfa/36l/r3+o/6zzl1rbbwue9VVv3Xb+QmbVg=</latexit>

shared across words. 28

 A recurrent neural network language model
■ Solution: a recurrent neural network (RNN)
■ Maintain a context vector, h. At each timestep (wj), compose the context with the
current word xj to create a new context for the next timestep:
hj = RNN(xj , hj 1 )
wi+1
<latexit sha1_base64="80JppsAukr0HPO6TBJ+7U6Rxqkk=">AAAFEXicfVPdbtMwFM5CgRH+NrjkxjBFpDCmtpsEEkKagAtuNgbsT6pL5DhO685OItvZ0kZ5Ci55Eu4QtzwBb4OdtNCuBUdJvhx/5/ic75wEKaNStVq/VuwrjavXrq/ecG7eun3n7tr6vWOZZAKTI5ywRJwGSBJGY3KkqGLkNBUE8YCRk+Dsjdk/OSdC0iQ+VKOU9DjqxzSiGClt8tdXvrow4EVeglegm/sFfbZTvgQV2J6CzhS0y57jwsMBUehz4eVQJWDcLLWJIzUQvJBJpDjKSw+Cf11Nx428hRB582+QTzrIXhVkyhob1qhZjjVr7Lj50hwOvAuf6iKGAHIaapvGMBIIF5DkqQeV8fCHM+EgDhNlnAsoMw4Z5VRJvxg+hjQGx+WsmzaWyzwd96KWzHE/7u97ld0cPC0RDGrZwNOpgM1p8o47qEV29Hto5DcRcn+4aZyGFdVf22httaoFFkF7AjasyTrw1+0eDBOccRIrzJCU3XYrVb0CCUUxI6UDM0lShM9Qn3Q1jBEnsldUY1QCV1tCECVC37EClXXWo0BcyhEPNNP0Sl7eM8Zle91MRS96BY3TTJEY1wdFGQNaTTOTIKSCYMVGGiAsqM4V4AHSvVN6crVgM8cMCDsnar4QzHuFjKrT51IKuP4WJCYXOOEcxeGTAkaIUzYKSYQypkznoyleJs1meE5TOVEpr2VyICMKJoL2aYwYI5GC5jFv1q+BgtXTgW+J7oUgezrB9ykRSCVCZ4JE3/wrujd9+BAa+D8mjf8wNZwvq6gS0MUYCZKUxEVZQcwSSWDQF0mWziW84F8lqgOgSCte88m8W83QA9m+PH6L4Liz1d7e6nzY2dh9PRnNVeuB9cjyrLb13Nq13lkH1pGFbdv27LbdaXxpfGt8b/yoqfbKxOe+NbcaP38Dpi2zJg==</latexit>

■ Recurrent because we repeatedly apply the same softmax(

softmax(
<latexit sha1_base64="fXUrGibuPWDoPErC1D8zjEiH0LQ=">AAAFEXicfVNdb9MwFM1CgRG+NnjkxTBFpDCmppsEEkKagAdeNgbsS6pL5DhO685OItvZ0kb5FTzyS3hDvPIL+DfYaQvtWnDV9PT6nOvrc2/CjFGpWq1fK/aVxtVr11dvODdv3b5zd2393rFMc4HJEU5ZKk5DJAmjCTlSVDFymgmCeMjISXj2xuyfnBMhaZocqmFGuhz1EhpTjJQOBesrX10Y8rKowCvQKYKSPtupXoIabE9Bewr8quu48LBPFPpcegVUKRg1Kx3iSPUFL2UaK46KyoPgX5+m48beQoqi+TfJJ51kr04yZY0Ma9isRpo1ctxiaQ0H3kVA9SUGAHIa6ZjGMBYIl5AUmQeVUQSDmXQQR6ky4hLKnENGOVUyKAePIU3AcTUr08FqmdJxL8aWOe7H/X2vjpuDp1cE/bFt4OnUwOa0eMcon/paqTkDY7/JUASDTSMa1NRgbaO11aoXWAT+BGxYk3UQrNtdGKU45yRRmCEpO34rU90SCUUxI5UDc0kyhM9Qj3Q0TBAnslvWY1QBV0ciEKdCfxMF6uisokRcyiEPNdP0Sl7eM8Fle51cxS+6JU2yXJEEjw+Kcwa0m2YmQUQFwYoNNUBYUF0rwH2ke6f05GrDZo7pE3ZO1PxFMO+WMq5Pnysp5Pq/IAm5wCnnKImelDBGnLJhRGKUM2U6H0/xMms2o3OayYlLxdgmBzKiYCpojyaIMRIraB7zYf3TV7B+OvAt0b0QZE8X+D4jAqlU6EqQ6Jl3RfemBx9CA//HpMkfpobz1yrrAvRljAVpRpKyqiFmqSQw7Ik0z+YKXtDXheoEKNaOj/lkXjZm6IH0L4/fIjhub/nbW+0POxu7ryejuWo9sB5ZnuVbz61d6511YB1Z2LZtz/btduNL41vje+PHmGqvTDT3rbnV+Pkbys+zMQ==</latexit>

))
function RNN( · ) each time. <latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>

<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>

B
<latexit sha1_base64="zEYq/t/kb3jj4jjQXAD8GHfO+Hk=">AAAFEnicfVNdb9MwFM26AiN8bfDIi2GqaGFMbTcJJIQ0DR542RiwL2kukeM6rTs7iWxnSxvlX/DIL+EN8cof4N9wnbTQrgVXTU+vz7m+PvfGjwXXptn8tVRZrl67fmPlpnvr9p2791bX7h/rKFGUHdFIROrUJ5oJHrIjw41gp7FiRPqCnfjnb+z+yQVTmkfhoRnGrCNJL+QBp8RAyFtb+lrDvszSHL1GZ6mX8efb+StUgK0JaE9AK++4NXzYZ4Z8zuopNhEaNXIISWL6SmY6CowkaV7H6F+fhlsL6nMp0sbfJJ8gyV6RZMIaWdawkY+ANXJr6cIaDuqXHodLDBCWvAsxwDhQhGaYpXEdG6vwBlPpMO1GxoozrBOJBZfcaC8bPME8RMf5tAyC+SKlW7ssLXNrH/f360XcHjy5IuqXtqFnhYF5Y1J6KXzWAiFQBtZ9myD1BhtWMwBNw911vdX15mazWGgetMZg3RmvA2+t0sHdiCaShYYKovVZqxmbTkaU4VSw3MWJZjGh56THzgCGRDLdyYo5ylENIl0URAq+oUFFdFqREan1UPrAtM3SV/dscNHeWWKCl52Mh3FiWEjLg4JEILDTDiXqcsWoEUMAhCoOtSLaJ9A8A6MLnk0d02figpnZi1DZyXRQnD5Tki/hv2Ihu6SRlCTsPs1wQCQXwy4LSCKMbX0wwYus2ehe8FiPXUpLm1wsmMGR4j0eEiFYYLB9zIbhp29w8XTxWwa9UGwPCnwfM0VMpKASonr2ZYHe9PAjbOH/mDz8wwQ4e62sKAAuYy2IYhZmeQGpiDTDfk9FSTxT8Jy+KBQSkAAcL/lsVlYyYCBbV8dvHhy3N1tbm+0P2+s7u+PRXHEeOo+dutNyXjg7zjvnwDlyaGW50qi0K1vVL9Vv1e/VHyW1sjTWPHBmVvXnb7tRs04=</latexit>

h0 hi 3 hi 2 hi 1 hi
…
<latexit sha1_base64="UxJxKSE4Fv+hebaFtaEtzDBO/9k=">AAAFD3icfVNdb9MwFM1KgRG+NnjkxTAVWhij3SaBhJAm4IGXjQH7kuoSOY7TurOTyHa2tFF+BI/8Et4Qr/wE/g3XaQvtWnDV5PT6nOvrc2/9RHBtms1fS5VL1ctXri5fc6/fuHnr9srqnSMdp4qyQxqLWJ34RDPBI3ZouBHsJFGMSF+wY//0jd0/PmNK8zg6MIOEdSTpRjzklBgIeatLX2vYl3lWoFeonXk5f7pdvEQl2JqAzQloFR23hg96zJDPeT3DJkbDRgEhSUxPyVzHoZEkK+oY/evTcGthfS5F1vib5BMk2S2TTFhDyxo0iiGwhm4tW1jDfv3c43CJPsKSBxADjENFaI5ZltSxsQqvP5UO0yA2VpxjnUosuORGe3n/EeYROiqmZRAsFind2vnIMrf2cW+vXsbtwZMrot7INvRkYmBjUrxrt0AHr7413+ozr79uJf2S6K2sNTea5ULzoDUGa8547XurlQ4OYppKFhkqiNbtVjMxnZwow6lghYtTzRJCT0mXtQFGRDLdycshKlANIgEKYwXfyKAyOq3IidR6IH1g2k7pi3s2uGivnZrwRSfnUZIaFtHRQWEqEHhpJxIFXDFqxAAAoYpDrYj2CHTOwNyCXVPH9Jg4Y2b2IlR2ch2Wp8+U5Ev4rVjEzmksJYmCxzkOieRiELCQpMLYvocTvMia9eCMJ3rsUjayycWCGRwr3uUREYKFBtvHbBhePYPLp4vfMuiFYrtQ4PuEKWJiBZUQ1bX/FOhNF9/HFv6PyaM/TICz18rLAuAy1oI4YVFelJCKWDPsd1WcJjMFz+nLQiEBCcHxEZ/NykYMGMjWxfGbB0ebG62tjc0P22s7r8ejuezccx44daflPHd2nHfOvnPo0IpTeVh5VmlWv1S/Vb9Xf4yolaWx5q4zs6o/fwMmxrKy</latexit>

<latexit sha1_base64="0yCpXupPBKPlITaMzpUGAbo6k04=">AAAE9nicfVPdbtMwFE5LgREYbCDBBTeGqaLlZ2q3SSAhpAm44GZjwP6kulSOc9J6s+PIdra0USSehDvELa/DE/AaOEkL7VpwlOTL53OOz/nOiRdxpk2r9bNSvVS7fOXq0jX3+o3lm7dWVm8fahkrCgdUcqmOPaKBsxAODDMcjiMFRHgcjrzTN/n+0RkozWS4b4YRdAXphyxglBhL9VYrX+rYE2mSoVeok/RS9mwre4kKsDkBGxPQzrpuHe8PwJDPaSPBRqJRM7OUIGagRKplYARJsgZG/7qabj1ozIVImn+DfLJBdoogE6tRbjVsZiNrNXLrycIc9hrnPWaLOEFYMN9yFuNAEZpiSKIGNrlH72QqHKa+NLlzinUsMGeCGd1LTx5hFqLDbNrNktkiT7d+Xkrm1j/u7jYKPj94UiIalLKhJxMBm5Pk3UGpsdtbWWutt4qF5kF7DNac8drrrVa72Jc0FhAayonWnXYrMt2UKMMoh8zFsYaI0FPSh46FIRGgu2kxKxmqW8ZHgVT2Dg0q2GmPlAith8KzlnlD9MW9nFy014lN8KKbsjCKDYS0PCiIObKS5YOHfKaAGj60gFDFbK6IDohtkLHjaVWZOmYA/AzMbCFUdFMdFKfPpOQJ+60ghHMqhSCh/zjFARGMD30ISMxN3t5gghdJ89Q/Y5Eeq5SUMrmYg8FSsT4LCecQGJw/Zmn7GhhcPF38FmwvFOzYBN9HoIiRymZCVD//IWxv+vgBzuH/LFn4x9LC2bLSIgFbTC6BjCBMswJSLjVgr69kHM0kPOdfJGoDkMAqXtrDrFtpYQeyfXH85sHhxnp7c33jw9ba9uvxaC45952HTsNpO8+dbeeds+ccOLTyq7pcvVu9V0tqX2vfat9L02pl7HPHmVm1H78Boaeq7w==</latexit> <latexit sha1_base64="cDE5LnazxffjZV2goLymzw4XzaU=">AAAE9nicfVPdbtMwFE5LgREYbCDBBTeGqaLlZ2q3SSAhpAm44GZjwP6kulSOc9J6s+PIdra0USSehDvELa/DE/AaOEkL7VpwlOTL53OOz/nOiRdxpk2r9bNSvVS7fOXq0jX3+o3lm7dWVm8fahkrCgdUcqmOPaKBsxAODDMcjiMFRHgcjrzTN/n+0RkozWS4b4YRdAXphyxglBhL9VYrX+rYE2mSoVeok/RS9mwre4kKsDkBGxPQzrpuHe8PwJDPaSPBRqJRM7OUIGagRKplYARJsgZG/7qabj1ozIVImn+DfLJBdoogE6tRbjVsZiNrNXLrycIc9hrnPWaLOEFYMN9yFuNAEZpiSKIGNrlH72QqHKa+NLlzinUsMGeCGd1LTx5hFqLDbNrNktkiT7d+Xkrm1j/u7jYKPj94UiIalLKhJxMBm5Pk3UEprdtbWWutt4qF5kF7DNac8drrrVa72Jc0FhAayonWnXYrMt2UKMMoh8zFsYaI0FPSh46FIRGgu2kxKxmqW8ZHgVT2Dg0q2GmPlAith8KzlnlD9MW9nFy014lN8KKbsjCKDYS0PCiIObKS5YOHfKaAGj60gFDFbK6IDohtkLHjaVWZOmYA/AzMbCFUdFMdFKfPpOQJ+60ghHMqhSCh/zjFARGMD30ISMxN3t5gghdJ89Q/Y5Eeq5SUMrmYg8FSsT4LCecQGJw/Zmn7GhhcPF38FmwvFOzYBN9HoIiRymZCVD//IWxv+vgBzuH/LFn4x9LC2bLSIgFbTC6BjCBMswJSLjVgr69kHM0kPOdfJGoDkMAqXtrDrFtpYQeyfXH85sHhxnp7c33jw9ba9uvxaC45952HTsNpO8+dbeeds+ccOLTyq7pcvVu9V0tqX2vfat9L02pl7HPHmVm1H78BnpWq7g==</latexit>

<latexit sha1_base64="YwgeWZ8MmCpvBq9864+tOD6jYJc=">AAAE8nicfVNLb9NAEHZCgNa8+jhyWagiEh5V0lYCCSFVwIFLS4G+pGyw1utxsu2u19pdt06M/wg3xJU/xJk/wtpJIGkCa9n+/O3M7Mw3Yz/mTJtW62eleq12/cbNpWX31u07d++trK4da5koCkdUcqlOfaKBswiODDMcTmMFRPgcTvzzN8X+yQUozWR0aAYxdAXpRSxklBhLeauVL3XsiyzN0SvUSb2MPdvJX6ISbE/A1gS0865bx4d9MORz1kixkWjYzC0liOkrkWkZGkHSvIHRv66mWw8bcyHS5t8gn2yQvTLIxGpYWA2a+dBaDd16ujCHg8alx2wRZwgLFljOYhwqQjMMadzApvDwzqbCYRpIUzhnWCcCcyaY0V529gizCB3n026WzBd5uvXLkWRu/eP+fqPki4MnJaL+SDb0ZCJgc5K82/darrey0dpslQvNg/YYbDjjdeCtVrs4kDQREBnKidaddis23YwowyiH3MWJhpjQc9KDjoUREaC7WTknOapbJkChVPaODCrZaY+MCK0HwreWRTP01b2CXLTXSUz4opuxKE4MRHR0UJhwZOUqhg4FTAE1fGABoYrZXBHtE9scY0fTKjJ1TB/4BZjZQqjoZjosT59JyRf2W0EEl1QKQaLgcYZDIhgfBBCShJuiteEEL5LmaXDBYj1WKR3J5GIOBkvFeiwinENocPGYpe2rb3D5dPFbsL1QsGcTfB+DIkYqmwlRveJnsL3p4Qe4gP+zZNEfSwtny8rKBGwxhQQyhijLS0i51ID9npJJPJPwnH+ZqA1AQqv4yB5m3UYWdiDbV8dvHhxvbba3N7c+7Gzsvh6P5pJz33noNJy289zZdd45B86RQyu/qsvVtep6zdS+1r7Vvo9Mq5Wxz7ozs2o/fgM1Dak2</latexit>

<latexit sha1_base64="MS7PqebmWexBvDkOqk8USVMUetI=">AAAE9nicfVPdbtMwFE5LgREYbCDBBTeGqaLlZ2q3SSAhpAm44GZjwP6kulSOc9J6s+PIdra0USSehDvELa/DE/AaOGkD7VpwlOTL8XeOz/nOiRdxpk2r9bNSvVS7fOXq0jX3+o3lm7dWVm8fahkrCgdUcqmOPaKBsxAODDMcjiMFRHgcjrzTN/n+0RkozWS4b4YRdAXphyxglBhr6q1WvtSxJ9IkQ69QJ+ml7NlW9hIVYLMEGyVoZ123jvcHYMjntJFgI9GomVmTIGagRKplYARJsgZG/7qabj1ozIVImn+DfLJBdoogJWuUs4bNbGRZI7eeLMxhr3HeY7aIE4QF863NYhwoQlMMSdTAJvfonUyFw9SXJndOsY4F5kwwo3vpySPMQnSYTbtZY7bI062fjyVz6x93dxuFPT+4LBENxrKhJ6WAzTJ5d7Ll9lbWWuutYqF50J6ANWey9nqr1S72JY0FhIZyonWn3YpMNyXKMMohc3GsISL0lPShY2FIBOhuWsxKhurW4qNAKnuHBhXWaY+UCK2HwrPMvCH64l5uXLTXiU3wopuyMIoNhHR8UBBzZCXLBw/5TAE1fGgBoYrZXBEdENsgY8fTqjJ1zAD4GZjZQqjopjooTp9JyRP2W0EI51QKQUL/cYoDIhgf+hCQmJu8vUGJF0nz1D9jkZ6olIxlcjEHg6VifRYSziEwOH/Mmu1rYHDxdPFbsL1QsGMTfB+BIkYqmwlR/fyHsL3p4wc4h/9jsvAP08LZstIiAVtMLoGMIEyzAlIuNWCvr2QczSQ8518kagOQwCo+5sOs25hhB7J9cfzmweHGentzfePD1tr268loLjn3nYdOw2k7z51t552z5xw4tPKruly9W71XS2pfa99q38fUamXic8eZWbUfvwGbg6rt</latexit>

…
<latexit sha1_base64="nB8pK9PGEwV+LmMCOwKc8YP2Au8=">AAADenicfVJLb9NAEN4mPEp4tXDkYhpV4hklbSU4VsCBC6JIpKmUtaLxeuyssg9rdx0SWf4RXOGX8V84sHYMqpuKkbz+PA/PN99OlAlu3XD4a6fTvXHz1u3dO7279+4/eLi3/+jc6twwHDMttLmIwKLgCseOO4EXmUGQkcBJtHhfxSdLNJZr9dWtMwwlpIonnIHzrslqVvDXx+Vsrz8cDGsLtsGoAX3S2NlsvxPSWLNconJMgLXT0TBzYQHGcSaw7NHcYgZsASlOPVQg0YZFzbcMDr0nDhJt/KNcUHsvVxQgrV3LyGdKcHN7NVY5r4tNc5e8DQuustyhYptGSS4Cp4Nq+CDmBpkTaw+AGe65BmwOBpjzEvUOL7eZo1iiaw/CZFjYpO7eohRJ/21Q4TempQQVvyhoApKLdYwJ5MKVBbXJX3ydNK/iJc9so9JqI1OPCnRUG55yBUJg4mh1tN3+NXe0Pnv0A/q7MPjJE/ycoQGnjWcCJpWwKv3dpPQpreD/Mrn6l+lhe6yiJuCHqSTQGaqirCET2iKNUqPzrEV4q74m6n8AiVd8k4/tsk2GX8jR1fXbBudHg9Hx4OjLSf/0XbOau+QJOSDPyIi8IafkIzkjY8LIgnwnP8jPzu/uQfd59+UmtbPT1DwmLeue/AGugCyM</latexit>
xi 3 xi 2
<latexit sha1_base64="Lw5rWS0utPfv8sAZFj6BQfU2/QM=">AAADenicfVJLb9NAEN4mPEp4teXIxTSqxKtREiqVYwUcuCCKRJpKWSsar8fOKvuwdtchkeUfwRV+Gf+FA2snoLqpGMnrz/PwfPPtRJng1vX7v3Za7Vu379zdvde5/+Dho8d7+wcXVueG4Yhpoc1lBBYFVzhy3Am8zAyCjASOo/n7Kj5eoLFcq69ulWEoIVU84Qycd42X04IfD8vpXrff69cWbIPBBnTJxs6n+62QxprlEpVjAqydDPqZCwswjjOBZYfmFjNgc0hx4qECiTYsar5lcOQ9cZBo4x/lgtp7taIAae1KRj5TgpvZ67HKeVNskrvkbVhwleUOFVs3SnIROB1UwwcxN8icWHkAzHDPNWAzMMCcl6hzdLXNDMUCXXMQJsPCJnX3BqVI+m+DCr8xLSWo+GVBE5BcrGJMIBeuLKhN/uKbpHkdL3hmNyot1zJ1qEBHteEpVyAEJo5WR9PtXzNH67NDP6C/C4OfPMHPGRpw2ngmYFIJy9LfTUqf0Qr+L5Orf5keNscqagJ+mEoCnaEqyhoyoS3SKDU6zxqEt+prov4HkHjF1/nYLFtn+IUcXF+/bXAx7A3e9IZfTrpn7zaruUuekkPynAzIKTkjH8k5GRFG5uQ7+UF+tn63D9sv2q/Wqa2dTc0T0rD2yR+rbyyL</latexit>
<latexit sha1_base64="0VBkd1Rw+AkOCuCd8jR6pjBnSJo=">AAADenicfVJLb9NAEN4mPEp4tXDkYhpV4tUoLpXgWAEHLogikaZS1orG67Gzyj6s3XVIZPlHcIVfxn/hwNoJqG4qRvL68zw833w7cS64dcPhr51O98bNW7d37/Tu3rv/4OHe/qNzqwvDcMS00OYiBouCKxw57gRe5AZBxgLH8fx9HR8v0Fiu1Ve3yjGSkCmecgbOu8bLacmPwmq61x8Oho0F2yDcgD7Z2Nl0vxPRRLNConJMgLWTcJi7qATjOBNY9WhhMQc2hwwnHiqQaKOy4VsFh96TBKk2/lEuaLyXK0qQ1q5k7DMluJm9Gqud18UmhUvfRiVXeeFQsXWjtBCB00E9fJBwg8yJlQfADPdcAzYDA8x5iXqHl9vMUCzQtQdhMipt2nRvUYql/zao8BvTUoJKXpQ0BcnFKsEUCuGqktr0L75OmlfJgud2o9JyLVOPCnRUG55xBUJg6mh9tN3+NXO0OXv0A/q7MPjJE/ycowGnjWcCJpOwrPzdZPQpreH/Mrn6l+lhe6yyIeCHqSXQOaqyaiAT2iKNM6OLvEV4q74h6n8AqVd8nY/tsnWGX8jw6vptg/PjQfh6cPzlpH/6brOau+QJOSDPSEjekFPykZyREWFkTr6TH+Rn53f3oPu8+3Kd2tnZ1DwmLeue/AGoXiyK</latexit>
xi 1 <latexit sha1_base64="x4eMZtKrr1x8Ug2+LL0eW7kjARo=">AAAFD3icfVNdb9MwFM1KgRG+NnjkxTAVWhij3SaBhJAm4IGXjQH7kuoSOY7TurOTyHa2tFF+BI/8Et4Qr/wE/g3XaQvtWnDV9PT6nOvrc2/8RHBtms1fS5VL1ctXri5fc6/fuHnr9srqnSMdp4qyQxqLWJ34RDPBI3ZouBHsJFGMSF+wY//0jd0/PmNK8zg6MIOEdSTpRjzklBgIeatLX2vYl3lWoFeonXk5f7pdvEQl2JqAzQloFR23hg96zJDPeT3DJkbDRgEhSUxPyVzHoZEkK+oY/evTcGthfS5F1vib5BMk2S2TTFhDyxo0iiGwhm4tW1jDfv3c43CJPsKSBxADjENFaI5ZltSxsQqvP5UO0yA2VpxjnUosuORGe3n/EeYROiqmZRAsFind2vnIMrf2cW+vXsbtwZMrot7INvRkYmBjUrxrA6ADRt+ab/WZ11+3kn5J9FbWmhvNcqF50BqDNWe89r3VSgcHMU0liwwVROt2q5mYTk6U4VSwwsWpZgmhp6TL2gAjIpnu5OUQFagGkQCFsYJvZFAZnVbkRGo9kD4wbaf0xT0bXLTXTk34opPzKEkNi+jooDAVCLy0E4kCrhg1YgCAUMWhVkR7BDpnYG7BrqljekycMTN7ESo7uQ7L02dK8iX8Vyxi5zSWkkTB4xyHRHIxCFhIUmFs38MJXmTNenDGEz12KRvZ5GLBDI4V7/KICMFCg+1jNgw/PYPLp4vfMuiFYrtQ4PuEKWJiBZUQ1bVvCvSmi+9jC//H5NEfJsDZa+VlAXAZa0GcsCgvSkhFrBn2uypOk5mC5/RloZCAhOD4iM9mZSMGDGTr4vjNg6PNjdbWxuaH7bWd1+PRXHbuOQ+cutNynjs7zjtn3zl0aMWpPKw8qzSrX6rfqt+rP0bUytJYc9eZWdWfvwFZxrLC</latexit>
xi
… breathless runners approached the
wi <latexit sha1_base64="tqdvN1Co9lflZKGA9FYLbBE3SQc=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1OC+X1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4Dx3yZxA==</latexit>
3 wi <latexit sha1_base64="cCatZM/QvTiGjI99S0wQVTzzBoU=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1O6+P1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DxGqZww==</latexit>
2 wi
<latexit sha1_base64="CH1jSvOpKNyBxULr16kKG8gaFmQ=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1Oy+L1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DwViZwg==</latexit>
1 wi
<latexit sha1_base64="5EwDIiByx9xSWpoPXUBPQBRnzvo=">AAAFD3icfVNdb9MwFM1KgRG+NnjkxTAVWhij3SaBhJAm4IGXjQH7kuoSOY7TurOTyHa2tFF+BI/8Et4Qr/wE/g3XaQvtWnDV9PT6nOvrc2/8RHBtms1fS5VL1ctXri5fc6/fuHnr9srqnSMdp4qyQxqLWJ34RDPBI3ZouBHsJFGMSF+wY//0jd0/PmNK8zg6MIOEdSTpRjzklBgIeatLX2vYl3lWoFeonXk5f7pdvEQl2JqAzQloFR23hg96zJDPeT3DJkbDRgEhSUxPyVzHoZEkK+oY/evTcGthfS5F1vib5BMk2S2TTFhDyxo0iiGwhm4tW1jDfv3c43CJPsKSBxADjENFaI5ZltSxsQqvP5UO0yA2VpxjnUosuORGe3n/EeYROiqmZRAsFind2vnIMrf2cW+vXsbtwZMrot7INvRkYmBjUrxrlaADRt+ab/WZ11+3kn5J9FbWmhvNcqF50BqDNWe89r3VSgcHMU0liwwVROt2q5mYTk6U4VSwwsWpZgmhp6TL2gAjIpnu5OUQFagGkQCFsYJvZFAZnVbkRGo9kD4wbaf0xT0bXLTXTk34opPzKEkNi+jooDAVCLy0E4kCrhg1YgCAUMWhVkR7BDpnYG7BrqljekycMTN7ESo7uQ7L02dK8iX8Vyxi5zSWkkTB4xyHRHIxCFhIUmFs38MJXmTNenDGEz12KRvZ5GLBDI4V7/KICMFCg+1jNgw/PYPLp4vfMuiFYrtQ4PuEKWJiBZUQ1bVvCvSmi+9jC//H5NEfJsDZa+VlAXAZa0GcsCgvSkhFrBn2uypOk5mC5/RloZCAhOD4iM9mZSMGDGTr4vjNg6PNjdbWxuaH7bWd1+PRXHbuOQ+cutNynjs7zjtn3zl0aMWpPKw8qzSrX6rfqt+rP0bUytJYc9eZWdWfvwFWlrLB</latexit>
29
 A recurrent neural network language model
■ Elman network: RNN(·) = f (⇥hi
<latexit sha1_base64="3RpTdTICQfJPaAPGF1iRfSwW//4=">AAAFD3icfVNdb9MwFM1KgRG+NnjkxTAVWjZGu00CCSFNwAMvGwP2gVSXyHWc1p2dRLazpY3yI3jkl/CGeOUn8G+4Tlpo14KrpqfX51xfn3vTjQXXptn8tVS5VL185eryNff6jZu3bq+s3jnWUaIoO6KRiNSnLtFM8JAdGW4E+xQrRmRXsJPu6Wu7f3LGlOZReGiGMetI0gt5wCkxEPJWl77WcFdmaY5eonbqZfzJTv4CFWB7ArYmoJV33Bo+7DNDPmf1FJsIjRo5hCQxfSUzHQVGkjSvY/SvT8OtBfW5FGnjb5KPkGSvSDJhjSxr2MhHwBq5tXRhDQf1c4/DJQYIS+5DDDAOFKEZZmlcx8YqvMFUOkz9yFhxhnUiseCSG+1lg0eYh+g4n5ZBMF+kdGvnpWXuh/39ehG2505uiPqla2i98C9vTCovdestSACUgTXfJki9wYbVDEDTcL2VteZms1hoHrTGYM0ZrwNvtdLBfkQTyUJDBdG63WrGppMRZTgVLHdxollM6CnpsTbAkEimO1kxRDmqQcRHQaTgGxpURKcVGZFaD2UXmLZT+uKeDS7aaycmeN7JeBgnhoW0PChIBAIv7UQinytGjRgCIFRxqBXRPoHOGZhbcGzqmD4TZ8zMXoTKTqaD4vSZkroS/isWsnMaSUlC/3GGAyK5GPosIIkwtu/BBC+yZsM/47Eeu5SWNrlYMIMjxXs8JEKwwGD7mA3DT9/g4uniNwx6odgeFPguZoqYSEElRPXsmwK96eH72ML/MXn4hwlw9lpZUQBcxloQxSzM8gJSEWmGuz0VJfFMwXP6olBIQAJwvOSzWVnJgIFsXRy/eXC8tdna3tx6v7O2+2o8msvOPeeBU3dazjNn13nrHDhHDq04lYeVp5Vm9Uv1W/V79UdJrSyNNXedmVX9+RtcYrK/</latexit>
1 + xi )

■ If dword = dh, can tie input (X) and output (B) word embeddings.
tanh

wi+1
<latexit sha1_base64="fXUrGibuPWDoPErC1D8zjEiH0LQ=">AAAFEXicfVNdb9MwFM1CgRG+NnjkxTBFpDCmppsEEkKagAdeNgbsS6pL5DhO685OItvZ0kb5FTzyS3hDvPIL+DfYaQvtWnDV9PT6nOvrc2/CjFGpWq1fK/aVxtVr11dvODdv3b5zd2393rFMc4HJEU5ZKk5DJAmjCTlSVDFymgmCeMjISXj2xuyfnBMhaZocqmFGuhz1EhpTjJQOBesrX10Y8rKowCvQKYKSPtupXoIabE9Bewr8quu48LBPFPpcegVUKRg1Kx3iSPUFL2UaK46KyoPgX5+m48beQoqi+TfJJ51kr04yZY0Ma9isRpo1ctxiaQ0H3kVA9SUGAHIa6ZjGMBYIl5AUmQeVUQSDmXQQR6ky4hLKnENGOVUyKAePIU3AcTUr08FqmdJxL8aWOe7H/X2vjpuDp1cE/bFt4OnUwOa0eMcon/paqTkDY7/JUASDTSMa1NRgbaO11aoXWAT+BGxYk3UQrNtdGKU45yRRmCEpO34rU90SCUUxI5UDc0kyhM9Qj3Q0TBAnslvWY1QBV0ciEKdCfxMF6uisokRcyiEPNdP0Sl7eM8Fle51cxS+6JU2yXJEEjw+Kcwa0m2YmQUQFwYoNNUBYUF0rwH2ke6f05GrDZo7pE3ZO1PxFMO+WMq5Pnysp5Pq/IAm5wCnnKImelDBGnLJhRGKUM2U6H0/xMms2o3OayYlLxdgmBzKiYCpojyaIMRIraB7zYf3TV7B+OvAt0b0QZE8X+D4jAqlU6EqQ6Jl3RfemBx9CA//HpMkfpobz1yrrAvRljAVpRpKyqiFmqSQw7Ik0z+YKXtDXheoEKNaOj/lkXjZm6IH0L4/fIjhub/nbW+0POxu7ryejuWo9sB5ZnuVbz61d6511YB1Z2LZtz/btduNL41vje+PHmGqvTDT3rbnV+Pkbys+zMQ==</latexit>

softmax(
softmax(
<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>

<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>
)) dims:
[dh x dh] B
<latexit sha1_base64="zEYq/t/kb3jj4jjQXAD8GHfO+Hk=">AAAFEnicfVNdb9MwFM26AiN8bfDIi2GqaGFMbTcJJIQ0DR542RiwL2kukeM6rTs7iWxnSxvlX/DIL+EN8cof4N9wnbTQrgVXTU+vz7m+PvfGjwXXptn8tVRZrl67fmPlpnvr9p2791bX7h/rKFGUHdFIROrUJ5oJHrIjw41gp7FiRPqCnfjnb+z+yQVTmkfhoRnGrCNJL+QBp8RAyFtb+lrDvszSHL1GZ6mX8efb+StUgK0JaE9AK++4NXzYZ4Z8zuopNhEaNXIISWL6SmY6CowkaV7H6F+fhlsL6nMp0sbfJJ8gyV6RZMIaWdawkY+ANXJr6cIaDuqXHodLDBCWvAsxwDhQhGaYpXEdG6vwBlPpMO1GxoozrBOJBZfcaC8bPME8RMf5tAyC+SKlW7ssLXNrH/f360XcHjy5IuqXtqFnhYF5Y1J6KXzWAiFQBtZ9myD1BhtWMwBNw911vdX15mazWGgetMZg3RmvA2+t0sHdiCaShYYKovVZqxmbTkaU4VSw3MWJZjGh56THzgCGRDLdyYo5ylENIl0URAq+oUFFdFqREan1UPrAtM3SV/dscNHeWWKCl52Mh3FiWEjLg4JEILDTDiXqcsWoEUMAhCoOtSLaJ9A8A6MLnk0d02figpnZi1DZyXRQnD5Tki/hv2Ihu6SRlCTsPs1wQCQXwy4LSCKMbX0wwYus2ehe8FiPXUpLm1wsmMGR4j0eEiFYYLB9zIbhp29w8XTxWwa9UGwPCnwfM0VMpKASonr2ZYHe9PAjbOH/mDz8wwQ4e62sKAAuYy2IYhZmeQGpiDTDfk9FSTxT8Jy+KBQSkAAcL/lsVlYyYCBbV8dvHhy3N1tbm+0P2+s7u+PRXHEeOo+dutNyXjg7zjvnwDlyaGW50qi0K1vVL9Vv1e/VHyW1sjTWPHBmVvXnb7tRs04=</latexit>
[V x dh]
h0 hi hi hi
<latexit sha1_base64="YwgeWZ8MmCpvBq9864+tOD6jYJc=">AAAE8nicfVNLb9NAEHZCgNa8+jhyWagiEh5V0lYCCSFVwIFLS4G+pGyw1utxsu2u19pdt06M/wg3xJU/xJk/wtpJIGkCa9n+/O3M7Mw3Yz/mTJtW62eleq12/cbNpWX31u07d++trK4da5koCkdUcqlOfaKBswiODDMcTmMFRPgcTvzzN8X+yQUozWR0aAYxdAXpRSxklBhLeauVL3XsiyzN0SvUSb2MPdvJX6ISbE/A1gS0865bx4d9MORz1kixkWjYzC0liOkrkWkZGkHSvIHRv66mWw8bcyHS5t8gn2yQvTLIxGpYWA2a+dBaDd16ujCHg8alx2wRZwgLFljOYhwqQjMMadzApvDwzqbCYRpIUzhnWCcCcyaY0V529gizCB3n026WzBd5uvXLkWRu/eP+fqPki4MnJaL+SDb0ZCJgc5K82/darrey0dpslQvNg/YYbDjjdeCtVrs4kDQREBnKidaddis23YwowyiH3MWJhpjQc9KDjoUREaC7WTknOapbJkChVPaODCrZaY+MCK0HwreWRTP01b2CXLTXSUz4opuxKE4MRHR0UJhwZOUqhg4FTAE1fGABoYrZXBHtE9scY0fTKjJ1TB/4BZjZQqjoZjosT59JyRf2W0EEl1QKQaLgcYZDIhgfBBCShJuiteEEL5LmaXDBYj1WKR3J5GIOBkvFeiwinENocPGYpe2rb3D5dPFbsL1QsGcTfB+DIkYqmwlRveJnsL3p4Qe4gP+zZNEfSwtny8rKBGwxhQQyhijLS0i51ID9npJJPJPwnH+ZqA1AQqv4yB5m3UYWdiDbV8dvHhxvbba3N7c+7Gzsvh6P5pJz33noNJy289zZdd45B86RQyu/qsvVtep6zdS+1r7Vvo9Mq5Wxz7ozs2o/fgM1Dak2</latexit>

⇥ …
<latexit sha1_base64="0yCpXupPBKPlITaMzpUGAbo6k04=">AAAE9nicfVPdbtMwFE5LgREYbCDBBTeGqaLlZ2q3SSAhpAm44GZjwP6kulSOc9J6s+PIdra0USSehDvELa/DE/AaOEkL7VpwlOTL53OOz/nOiRdxpk2r9bNSvVS7fOXq0jX3+o3lm7dWVm8fahkrCgdUcqmOPaKBsxAODDMcjiMFRHgcjrzTN/n+0RkozWS4b4YRdAXphyxglBhL9VYrX+rYE2mSoVeok/RS9mwre4kKsDkBGxPQzrpuHe8PwJDPaSPBRqJRM7OUIGagRKplYARJsgZG/7qabj1ozIVImn+DfLJBdoogE6tRbjVsZiNrNXLrycIc9hrnPWaLOEFYMN9yFuNAEZpiSKIGNrlH72QqHKa+NLlzinUsMGeCGd1LTx5hFqLDbNrNktkiT7d+Xkrm1j/u7jYKPj94UiIalLKhJxMBm5Pk3UGpsdtbWWutt4qF5kF7DNac8drrrVa72Jc0FhAayonWnXYrMt2UKMMoh8zFsYaI0FPSh46FIRGgu2kxKxmqW8ZHgVT2Dg0q2GmPlAith8KzlnlD9MW9nFy014lN8KKbsjCKDYS0PCiIObKS5YOHfKaAGj60gFDFbK6IDohtkLHjaVWZOmYA/AzMbCFUdFMdFKfPpOQJ+60ghHMqhSCh/zjFARGMD30ISMxN3t5gghdJ89Q/Y5Eeq5SUMrmYg8FSsT4LCecQGJw/Zmn7GhhcPF38FmwvFOzYBN9HoIiRymZCVD//IWxv+vgBzuH/LFn4x9LC2bLSIgFbTC6BjCBMswJSLjVgr69kHM0kPOdfJGoDkMAqXtrDrFtpYQeyfXH85sHhxnp7c33jw9ba9uvxaC45952HTsNpO8+dbeeds+ccOLTyq7pcvVu9V0tqX2vfat9L02pl7HPHmVm1H78Boaeq7w==</latexit>
3
⇥ <latexit sha1_base64="cDE5LnazxffjZV2goLymzw4XzaU=">AAAE9nicfVPdbtMwFE5LgREYbCDBBTeGqaLlZ2q3SSAhpAm44GZjwP6kulSOc9J6s+PIdra0USSehDvELa/DE/AaOEkL7VpwlOTL53OOz/nOiRdxpk2r9bNSvVS7fOXq0jX3+o3lm7dWVm8fahkrCgdUcqmOPaKBsxAODDMcjiMFRHgcjrzTN/n+0RkozWS4b4YRdAXphyxglBhL9VYrX+rYE2mSoVeok/RS9mwre4kKsDkBGxPQzrpuHe8PwJDPaSPBRqJRM7OUIGagRKplYARJsgZG/7qabj1ozIVImn+DfLJBdoogE6tRbjVsZiNrNXLrycIc9hrnPWaLOEFYMN9yFuNAEZpiSKIGNrlH72QqHKa+NLlzinUsMGeCGd1LTx5hFqLDbNrNktkiT7d+Xkrm1j/u7jYKPj94UiIalLKhJxMBm5Pk3UEprdtbWWutt4qF5kF7DNac8drrrVa72Jc0FhAayonWnXYrMt2UKMMoh8zFsYaI0FPSh46FIRGgu2kxKxmqW8ZHgVT2Dg0q2GmPlAith8KzlnlD9MW9nFy014lN8KKbsjCKDYS0PCiIObKS5YOHfKaAGj60gFDFbK6IDohtkLHjaVWZOmYA/AzMbCFUdFMdFKfPpOQJ+60ghHMqhSCh/zjFARGMD30ISMxN3t5gghdJ89Q/Y5Eeq5SUMrmYg8FSsT4LCecQGJw/Zmn7GhhcPF38FmwvFOzYBN9HoIiRymZCVD//IWxv+vgBzuH/LFn4x9LC2bLSIgFbTC6BjCBMswJSLjVgr69kHM0kPOdfJGoDkMAqXtrDrFtpYQeyfXH85sHhxnp7c33jw9ba9uvxaC45952HTsNpO8+dbeeds+ccOLTyq7pcvVu9V0tqX2vfat9L02pl7HPHmVm1H78BnpWq7g==</latexit>
2 ⇥ hi
<latexit sha1_base64="MS7PqebmWexBvDkOqk8USVMUetI=">AAAE9nicfVPdbtMwFE5LgREYbCDBBTeGqaLlZ2q3SSAhpAm44GZjwP6kulSOc9J6s+PIdra0USSehDvELa/DE/AaOGkD7VpwlOTL8XeOz/nOiRdxpk2r9bNSvVS7fOXq0jX3+o3lm7dWVm8fahkrCgdUcqmOPaKBsxAODDMcjiMFRHgcjrzTN/n+0RkozWS4b4YRdAXphyxglBhr6q1WvtSxJ9IkQ69QJ+ml7NlW9hIVYLMEGyVoZ123jvcHYMjntJFgI9GomVmTIGagRKplYARJsgZG/7qabj1ozIVImn+DfLJBdoogJWuUs4bNbGRZI7eeLMxhr3HeY7aIE4QF863NYhwoQlMMSdTAJvfonUyFw9SXJndOsY4F5kwwo3vpySPMQnSYTbtZY7bI062fjyVz6x93dxuFPT+4LBENxrKhJ6WAzTJ5d7Ll9lbWWuutYqF50J6ANWey9nqr1S72JY0FhIZyonWn3YpMNyXKMMohc3GsISL0lPShY2FIBOhuWsxKhurW4qNAKnuHBhXWaY+UCK2HwrPMvCH64l5uXLTXiU3wopuyMIoNhHR8UBBzZCXLBw/5TAE1fGgBoYrZXBEdENsgY8fTqjJ1zAD4GZjZQqjopjooTp9JyRP2W0EI51QKQUL/cYoDIhgf+hCQmJu8vUGJF0nz1D9jkZ6olIxlcjEHg6VifRYSziEwOH/Mmu1rYHDxdPFbsL1QsGMTfB+BIkYqmwlR/fyHsL3p4wc4h/9jsvAP08LZstIiAVtMLoGMIEyzAlIuNWCvr2QczSQ8518kagOQwCo+5sOs25hhB7J9cfzmweHGentzfePD1tr268loLjn3nYdOw2k7z51t552z5xw4tPKruly9W71XS2pfa99q38fUamXic8eZWbUfvwGbg6rt</latexit>
1
⇥ <latexit sha1_base64="UxJxKSE4Fv+hebaFtaEtzDBO/9k=">AAAFD3icfVNdb9MwFM1KgRG+NnjkxTAVWhij3SaBhJAm4IGXjQH7kuoSOY7TurOTyHa2tFF+BI/8Et4Qr/wE/g3XaQvtWnDV5PT6nOvrc2/9RHBtms1fS5VL1ctXri5fc6/fuHnr9srqnSMdp4qyQxqLWJ34RDPBI3ZouBHsJFGMSF+wY//0jd0/PmNK8zg6MIOEdSTpRjzklBgIeatLX2vYl3lWoFeonXk5f7pdvEQl2JqAzQloFR23hg96zJDPeT3DJkbDRgEhSUxPyVzHoZEkK+oY/evTcGthfS5F1vib5BMk2S2TTFhDyxo0iiGwhm4tW1jDfv3c43CJPsKSBxADjENFaI5ZltSxsQqvP5UO0yA2VpxjnUosuORGe3n/EeYROiqmZRAsFind2vnIMrf2cW+vXsbtwZMrot7INvRkYmBjUrxrt0AHr7413+ozr79uJf2S6K2sNTea5ULzoDUGa8547XurlQ4OYppKFhkqiNbtVjMxnZwow6lghYtTzRJCT0mXtQFGRDLdycshKlANIgEKYwXfyKAyOq3IidR6IH1g2k7pi3s2uGivnZrwRSfnUZIaFtHRQWEqEHhpJxIFXDFqxAAAoYpDrYj2CHTOwNyCXVPH9Jg4Y2b2IlR2ch2Wp8+U5Ev4rVjEzmksJYmCxzkOieRiELCQpMLYvocTvMia9eCMJ3rsUjayycWCGRwr3uUREYKFBtvHbBhePYPLp4vfMuiFYrtQ4PuEKWJiBZUQ1bX/FOhNF9/HFv6PyaM/TICz18rLAuAy1oI4YVFelJCKWDPsd1WcJjMFz+nLQiEBCcHxEZ/NykYMGMjWxfGbB0ebG62tjc0P22s7r8ejuezccx44daflPHd2nHfOvnPo0IpTeVh5VmlWv1S/Vb9Xf4yolaWx5q4zs6o/fwMmxrKy</latexit>

[dh]
<latexit sha1_base64="PXEiyqto0QO3Q9mBhKP8IXxzbXU=">AAAE7XicfVNNb9NAEHVCgGJoaeHIZaGKSPiokrYSSAipAg5cWgr0S8qGaL0eJ9vueq3ddePE8s/ghrjym7jwW1jbCSRNYC3bz29nZmfejL2IM21arZ+V6rXa9Rs3V265t++srt1d37h3omWsKBxTyaU684gGzkI4NsxwOIsUEOFxOPUu3ub7p5egNJPhkRlF0BWkH7KAUWIs1duoDOvYE2mSodeok/RS9nw3e4UKsDMF21PQzrpuHR8NwJAvaSPBRqJxM7OUIGagRKplYARJsgZG/7qabj1oLIRImn+DfLZB9osgU6txbjVqZmNrNXbrydIcDhvDHrNFnCMsmG85i3GgCE0xJFEDm9yjdz4TDlNfmtw5xToWmDPBjO6l548xC9FJNutmyWyZp1sflpK59U8HB42Czw+elogGpWzo6VTAplvuuL31zdZWq1hoEbQnYNOZrMPeRrWLfUljAaGhnGjdabci002JMoxyyFwca4gIvSB96FgYEgG6mxYjkqG6ZXwUSGXv0KCCnfVIidB6JDxrmfdBX93LyWV7ndgEL7spC6PYQEjLg4KYI6tUPm/IZwqo4SMLCFXM5orogNi+GDuVtpMzxwyAX4KZL4SKbqqD4vS5lDxhvxWEMKRSCBL6T1IcEMH4yIeAxNzkXQ2meJk0z/xLFumJSkkpk4s5GCwV67OQcA6BwfljnravgcHF08XvwPZCwb5N8EMEihipbCZE9fP/wPamjx/iHP7PkoV/LC2cLystErDF5BLICMI0KyDlUgP2+krG0VzCC/5FojYACazipT3Mu5UWdiDbV8dvEZxsb7V3trY/7m7uvZmM5orzwHnkNJy288LZc947h86xQyu/qrXqanWtJmtfa99q30vTamXic9+ZW7UfvwFyGafe</latexit>

<latexit sha1_base64="PXEiyqto0QO3Q9mBhKP8IXxzbXU=">AAAE7XicfVNNb9NAEHVCgGJoaeHIZaGKSPiokrYSSAipAg5cWgr0S8qGaL0eJ9vueq3ddePE8s/ghrjym7jwW1jbCSRNYC3bz29nZmfejL2IM21arZ+V6rXa9Rs3V265t++srt1d37h3omWsKBxTyaU684gGzkI4NsxwOIsUEOFxOPUu3ub7p5egNJPhkRlF0BWkH7KAUWIs1duoDOvYE2mSodeok/RS9nw3e4UKsDMF21PQzrpuHR8NwJAvaSPBRqJxM7OUIGagRKplYARJsgZG/7qabj1oLIRImn+DfLZB9osgU6txbjVqZmNrNXbrydIcDhvDHrNFnCMsmG85i3GgCE0xJFEDm9yjdz4TDlNfmtw5xToWmDPBjO6l548xC9FJNutmyWyZp1sflpK59U8HB42Czw+elogGpWzo6VTAplvuuL31zdZWq1hoEbQnYNOZrMPeRrWLfUljAaGhnGjdabci002JMoxyyFwca4gIvSB96FgYEgG6mxYjkqG6ZXwUSGXv0KCCnfVIidB6JDxrmfdBX93LyWV7ndgEL7spC6PYQEjLg4KYI6tUPm/IZwqo4SMLCFXM5orogNi+GDuVtpMzxwyAX4KZL4SKbqqD4vS5lDxhvxWEMKRSCBL6T1IcEMH4yIeAxNzkXQ2meJk0z/xLFumJSkkpk4s5GCwV67OQcA6BwfljnravgcHF08XvwPZCwb5N8EMEihipbCZE9fP/wPamjx/iHP7PkoV/LC2cLystErDF5BLICMI0KyDlUgP2+krG0VzCC/5FojYACazipT3Mu5UWdiDbV8dvEZxsb7V3trY/7m7uvZmM5orzwHnkNJy288LZc947h86xQyu/qrXqanWtJmtfa99q30vTamXic9+ZW7UfvwFyGafe</latexit>

<latexit sha1_base64="PXEiyqto0QO3Q9mBhKP8IXxzbXU=">AAAE7XicfVNNb9NAEHVCgGJoaeHIZaGKSPiokrYSSAipAg5cWgr0S8qGaL0eJ9vueq3ddePE8s/ghrjym7jwW1jbCSRNYC3bz29nZmfejL2IM21arZ+V6rXa9Rs3V265t++srt1d37h3omWsKBxTyaU684gGzkI4NsxwOIsUEOFxOPUu3ub7p5egNJPhkRlF0BWkH7KAUWIs1duoDOvYE2mSodeok/RS9nw3e4UKsDMF21PQzrpuHR8NwJAvaSPBRqJxM7OUIGagRKplYARJsgZG/7qabj1oLIRImn+DfLZB9osgU6txbjVqZmNrNXbrydIcDhvDHrNFnCMsmG85i3GgCE0xJFEDm9yjdz4TDlNfmtw5xToWmDPBjO6l548xC9FJNutmyWyZp1sflpK59U8HB42Czw+elogGpWzo6VTAplvuuL31zdZWq1hoEbQnYNOZrMPeRrWLfUljAaGhnGjdabci002JMoxyyFwca4gIvSB96FgYEgG6mxYjkqG6ZXwUSGXv0KCCnfVIidB6JDxrmfdBX93LyWV7ndgEL7spC6PYQEjLg4KYI6tUPm/IZwqo4SMLCFXM5orogNi+GDuVtpMzxwyAX4KZL4SKbqqD4vS5lDxhvxWEMKRSCBL6T1IcEMH4yIeAxNzkXQ2meJk0z/xLFumJSkkpk4s5GCwV67OQcA6BwfljnravgcHF08XvwPZCwb5N8EMEihipbCZE9fP/wPamjx/iHP7PkoV/LC2cLystErDF5BLICMI0KyDlUgP2+krG0VzCC/5FojYACazipT3Mu5UWdiDbV8dvEZxsb7V3trY/7m7uvZmM5orzwHnkNJy288LZc947h86xQyu/qrXqanWtJmtfa99q30vTamXic9+ZW7UfvwFyGafe</latexit> <latexit sha1_base64="PXEiyqto0QO3Q9mBhKP8IXxzbXU=">AAAE7XicfVNNb9NAEHVCgGJoaeHIZaGKSPiokrYSSAipAg5cWgr0S8qGaL0eJ9vueq3ddePE8s/ghrjym7jwW1jbCSRNYC3bz29nZmfejL2IM21arZ+V6rXa9Rs3V265t++srt1d37h3omWsKBxTyaU684gGzkI4NsxwOIsUEOFxOPUu3ub7p5egNJPhkRlF0BWkH7KAUWIs1duoDOvYE2mSodeok/RS9nw3e4UKsDMF21PQzrpuHR8NwJAvaSPBRqJxM7OUIGagRKplYARJsgZG/7qabj1oLIRImn+DfLZB9osgU6txbjVqZmNrNXbrydIcDhvDHrNFnCMsmG85i3GgCE0xJFEDm9yjdz4TDlNfmtw5xToWmDPBjO6l548xC9FJNutmyWyZp1sflpK59U8HB42Czw+elogGpWzo6VTAplvuuL31zdZWq1hoEbQnYNOZrMPeRrWLfUljAaGhnGjdabci002JMoxyyFwca4gIvSB96FgYEgG6mxYjkqG6ZXwUSGXv0KCCnfVIidB6JDxrmfdBX93LyWV7ndgEL7spC6PYQEjLg4KYI6tUPm/IZwqo4SMLCFXM5orogNi+GDuVtpMzxwyAX4KZL4SKbqqD4vS5lDxhvxWEMKRSCBL6T1IcEMH4yIeAxNzkXQ2meJk0z/xLFumJSkkpk4s5GCwV67OQcA6BwfljnravgcHF08XvwPZCwb5N8EMEihipbCZE9fP/wPamjx/iHP7PkoV/LC2cLystErDF5BLICMI0KyDlUgP2+krG0VzCC/5FojYACazipT3Mu5UWdiDbV8dvEZxsb7V3trY/7m7uvZmM5orzwHnkNJy288LZc947h86xQyu/qrXqanWtJmtfa99q30vTamXic9+ZW7UfvwFyGafe</latexit>

+ + + + +
… [dword]
<latexit sha1_base64="nB8pK9PGEwV+LmMCOwKc8YP2Au8=">AAADenicfVJLb9NAEN4mPEp4tXDkYhpV4hklbSU4VsCBC6JIpKmUtaLxeuyssg9rdx0SWf4RXOGX8V84sHYMqpuKkbz+PA/PN99OlAlu3XD4a6fTvXHz1u3dO7279+4/eLi3/+jc6twwHDMttLmIwKLgCseOO4EXmUGQkcBJtHhfxSdLNJZr9dWtMwwlpIonnIHzrslqVvDXx+Vsrz8cDGsLtsGoAX3S2NlsvxPSWLNconJMgLXT0TBzYQHGcSaw7NHcYgZsASlOPVQg0YZFzbcMDr0nDhJt/KNcUHsvVxQgrV3LyGdKcHN7NVY5r4tNc5e8DQuustyhYptGSS4Cp4Nq+CDmBpkTaw+AGe65BmwOBpjzEvUOL7eZo1iiaw/CZFjYpO7eohRJ/21Q4TempQQVvyhoApKLdYwJ5MKVBbXJX3ydNK/iJc9so9JqI1OPCnRUG55yBUJg4mh1tN3+NXe0Pnv0A/q7MPjJE/ycoQGnjWcCJpWwKv3dpPQpreD/Mrn6l+lhe6yiJuCHqSTQGaqirCET2iKNUqPzrEV4q74m6n8AiVd8k4/tsk2GX8jR1fXbBudHg9Hx4OjLSf/0XbOau+QJOSDPyIi8IafkIzkjY8LIgnwnP8jPzu/uQfd59+UmtbPT1DwmLeue/AGugCyM</latexit>
xi 3 xi 2
<latexit sha1_base64="Lw5rWS0utPfv8sAZFj6BQfU2/QM=">AAADenicfVJLb9NAEN4mPEp4teXIxTSqxKtREiqVYwUcuCCKRJpKWSsar8fOKvuwdtchkeUfwRV+Gf+FA2snoLqpGMnrz/PwfPPtRJng1vX7v3Za7Vu379zdvde5/+Dho8d7+wcXVueG4Yhpoc1lBBYFVzhy3Am8zAyCjASOo/n7Kj5eoLFcq69ulWEoIVU84Qycd42X04IfD8vpXrff69cWbIPBBnTJxs6n+62QxprlEpVjAqydDPqZCwswjjOBZYfmFjNgc0hx4qECiTYsar5lcOQ9cZBo4x/lgtp7taIAae1KRj5TgpvZ67HKeVNskrvkbVhwleUOFVs3SnIROB1UwwcxN8icWHkAzHDPNWAzMMCcl6hzdLXNDMUCXXMQJsPCJnX3BqVI+m+DCr8xLSWo+GVBE5BcrGJMIBeuLKhN/uKbpHkdL3hmNyot1zJ1qEBHteEpVyAEJo5WR9PtXzNH67NDP6C/C4OfPMHPGRpw2ngmYFIJy9LfTUqf0Qr+L5Orf5keNscqagJ+mEoCnaEqyhoyoS3SKDU6zxqEt+prov4HkHjF1/nYLFtn+IUcXF+/bXAx7A3e9IZfTrpn7zaruUuekkPynAzIKTkjH8k5GRFG5uQ7+UF+tn63D9sv2q/Wqa2dTc0T0rD2yR+rbyyL</latexit>
<latexit sha1_base64="0VBkd1Rw+AkOCuCd8jR6pjBnSJo=">AAADenicfVJLb9NAEN4mPEp4tXDkYhpV4tUoLpXgWAEHLogikaZS1orG67Gzyj6s3XVIZPlHcIVfxn/hwNoJqG4qRvL68zw833w7cS64dcPhr51O98bNW7d37/Tu3rv/4OHe/qNzqwvDcMS00OYiBouCKxw57gRe5AZBxgLH8fx9HR8v0Fiu1Ve3yjGSkCmecgbOu8bLacmPwmq61x8Oho0F2yDcgD7Z2Nl0vxPRRLNConJMgLWTcJi7qATjOBNY9WhhMQc2hwwnHiqQaKOy4VsFh96TBKk2/lEuaLyXK0qQ1q5k7DMluJm9Gqud18UmhUvfRiVXeeFQsXWjtBCB00E9fJBwg8yJlQfADPdcAzYDA8x5iXqHl9vMUCzQtQdhMipt2nRvUYql/zao8BvTUoJKXpQ0BcnFKsEUCuGqktr0L75OmlfJgud2o9JyLVOPCnRUG55xBUJg6mh9tN3+NXO0OXv0A/q7MPjJE/ycowGnjWcCJpOwrPzdZPQpreH/Mrn6l+lhe6yyIeCHqSXQOaqyaiAT2iKNM6OLvEV4q74h6n8AqVd8nY/tsnWGX8jw6vptg/PjQfh6cPzlpH/6brOau+QJOSDPSEjekFPykZyREWFkTr6TH+Rn53f3oPu8+3Kd2tnZ1DwmLeue/AGoXiyK</latexit>
xi 1 <latexit sha1_base64="x4eMZtKrr1x8Ug2+LL0eW7kjARo=">AAAFD3icfVNdb9MwFM1KgRG+NnjkxTAVWhij3SaBhJAm4IGXjQH7kuoSOY7TurOTyHa2tFF+BI/8Et4Qr/wE/g3XaQvtWnDV9PT6nOvrc2/8RHBtms1fS5VL1ctXri5fc6/fuHnr9srqnSMdp4qyQxqLWJ34RDPBI3ZouBHsJFGMSF+wY//0jd0/PmNK8zg6MIOEdSTpRjzklBgIeatLX2vYl3lWoFeonXk5f7pdvEQl2JqAzQloFR23hg96zJDPeT3DJkbDRgEhSUxPyVzHoZEkK+oY/evTcGthfS5F1vib5BMk2S2TTFhDyxo0iiGwhm4tW1jDfv3c43CJPsKSBxADjENFaI5ZltSxsQqvP5UO0yA2VpxjnUosuORGe3n/EeYROiqmZRAsFind2vnIMrf2cW+vXsbtwZMrot7INvRkYmBjUrxrA6ADRt+ab/WZ11+3kn5J9FbWmhvNcqF50BqDNWe89r3VSgcHMU0liwwVROt2q5mYTk6U4VSwwsWpZgmhp6TL2gAjIpnu5OUQFagGkQCFsYJvZFAZnVbkRGo9kD4wbaf0xT0bXLTXTk34opPzKEkNi+jooDAVCLy0E4kCrhg1YgCAUMWhVkR7BDpnYG7BrqljekycMTN7ESo7uQ7L02dK8iX8Vyxi5zSWkkTB4xyHRHIxCFhIUmFs38MJXmTNenDGEz12KRvZ5GLBDI4V7/KICMFCg+1jNgw/PYPLp4vfMuiFYrtQ4PuEKWJiBZUQ1bVvCvSmi+9jC//H5NEfJsDZa+VlAXAZa0GcsCgvSkhFrBn2uypOk5mC5/RloZCAhOD4iM9mZSMGDGTr4vjNg6PNjdbWxuaH7bWd1+PRXHbuOQ+cutNynjs7zjtn3zl0aMWpPKw8qzSrX6rfqt+rP0bUytJYc9eZWdWfvwFZxrLC</latexit>
xi
… breathless runners approached the dword = dh
wi <latexit sha1_base64="tqdvN1Co9lflZKGA9FYLbBE3SQc=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1OC+X1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4Dx3yZxA==</latexit>
3 wi <latexit sha1_base64="cCatZM/QvTiGjI99S0wQVTzzBoU=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1O6+P1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DxGqZww==</latexit>
2 wi
<latexit sha1_base64="CH1jSvOpKNyBxULr16kKG8gaFmQ=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1Oy+L1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DwViZwg==</latexit>
1 wi
<latexit sha1_base64="5EwDIiByx9xSWpoPXUBPQBRnzvo=">AAAFD3icfVNdb9MwFM1KgRG+NnjkxTAVWhij3SaBhJAm4IGXjQH7kuoSOY7TurOTyHa2tFF+BI/8Et4Qr/wE/g3XaQvtWnDV9PT6nOvrc2/8RHBtms1fS5VL1ctXri5fc6/fuHnr9srqnSMdp4qyQxqLWJ34RDPBI3ZouBHsJFGMSF+wY//0jd0/PmNK8zg6MIOEdSTpRjzklBgIeatLX2vYl3lWoFeonXk5f7pdvEQl2JqAzQloFR23hg96zJDPeT3DJkbDRgEhSUxPyVzHoZEkK+oY/evTcGthfS5F1vib5BMk2S2TTFhDyxo0iiGwhm4tW1jDfv3c43CJPsKSBxADjENFaI5ZltSxsQqvP5UO0yA2VpxjnUosuORGe3n/EeYROiqmZRAsFind2vnIMrf2cW+vXsbtwZMrot7INvRkYmBjUrxrlaADRt+ab/WZ11+3kn5J9FbWmhvNcqF50BqDNWe89r3VSgcHMU0liwwVROt2q5mYTk6U4VSwwsWpZgmhp6TL2gAjIpnu5OUQFagGkQCFsYJvZFAZnVbkRGo9kD4wbaf0xT0bXLTXTk34opPzKEkNi+jooDAVCLy0E4kCrhg1YgCAUMWhVkR7BDpnYG7BrqljekycMTN7ESo7uQ7L02dK8iX8Vyxi5zSWkkTB4xyHRHIxCFhIUmFs38MJXmTNenDGEz12KRvZ5GLBDI4V7/KICMFCg+1jNgw/PYPLp4vfMuiFYrtQ4PuEKWJiBZUQ1bVvCvSmi+9jC//H5NEfJsDZa+VlAXAZa0GcsCgvSkhFrBn2uypOk5mC5/RloZCAhOD4iM9mZSMGDGTr4vjNg6PNjdbWxuaH7bWd1+PRXHbuOQ+cutNynjs7zjtn3zl0aMWpPKw8qzSrX6rfqt+rP0bUytJYc9eZWdWfvwFWlrLB</latexit>
30
 A recurrent neural network language model
■ If we don’t want to tie X with B, can also add another projection matrix ⇥ <latexit sha1_base64="buQ0kPDjWBeJH6grSAxRTv7QNIM=">AAAFInicfVNdb9MwFM1KgRG+NvbIi2GqaGFM7TYJEEKaNh542RiwL2kukeM4rTs7iWxnSxvl3/DIL+EN8YTEj+EmaaFZC66anl6fc33v8Y0bCa5Nu/1zoXatfv3GzcVb9u07d+/dX1p+cKzDWFF2REMRqlOXaCZ4wI4MN4KdRooR6Qp24p7v5vsnF0xpHgaHZhixriS9gPucEgMhZ3nhawO7Mk0y9AadJU7Kn29lr1EBNidgYwI6Wddu4MM+M+Rz2kywCdGolUFIEtNXMtWhbyRJsiZG//q07IbfnEmRtP4m+QRJ9ookE9YoZw1b2QhYI7uRzK3hoHnpcGhigLDkHsQAY18RmmKWRE1scoUzmEqHqReaXJxiHUssuORGO+ngCeYBOs6mZRDM5intxmVpmd34uL/fLOL5wZMWUb+0DT0rDMxak9JL4bMOCIEyyN3PEyTOYC3XDEDTsqtd9uE4Z2m1vd4uFpoFnTFYtcbrwFmudbEX0liywFBBtD7rtCPTTYkynAqW2TjWLCL0nPTYGcCASKa7aTFXGWpAxEN+qOAbGFREpxUpkVoPpQvM/PL01b08OG/vLDb+y27Kgyg2LKDlQX4sELSZDynyuGLUiCEAQhWHWhHtE7hMA6MMHk4d02figplqI1R2U+0Xp1dKciX8VyxglzSUkgTe0xT7RHIx9JhPYmHyUfAneJ41a94Fj/TYpaS0ycaCGRwq3uMBEYL5BuePahh++gYXTxu/ZXAXiu1Bge8jpogJFVRCVC9/eeBuevgRzuH/mDz4wwRYbSstCoBmcgvCiAVpVkAqQs2w21NhHFUKntEXhUIC4oPjJZ9VZSUDBrJzdfxmwfHGemdzfePD1ur2zng0F62H1mOraXWsF9a29c46sI4sWlupvart1HbrX+rf6t/rP0pqbWGsWbEqq/7rN0QJueQ=</latexit>
(x!h)

wi+1
<latexit sha1_base64="fXUrGibuPWDoPErC1D8zjEiH0LQ=">AAAFEXicfVNdb9MwFM1CgRG+NnjkxTBFpDCmppsEEkKagAdeNgbsS6pL5DhO685OItvZ0kb5FTzyS3hDvPIL+DfYaQvtWnDV9PT6nOvrc2/CjFGpWq1fK/aVxtVr11dvODdv3b5zd2393rFMc4HJEU5ZKk5DJAmjCTlSVDFymgmCeMjISXj2xuyfnBMhaZocqmFGuhz1EhpTjJQOBesrX10Y8rKowCvQKYKSPtupXoIabE9Bewr8quu48LBPFPpcegVUKRg1Kx3iSPUFL2UaK46KyoPgX5+m48beQoqi+TfJJ51kr04yZY0Ma9isRpo1ctxiaQ0H3kVA9SUGAHIa6ZjGMBYIl5AUmQeVUQSDmXQQR6ky4hLKnENGOVUyKAePIU3AcTUr08FqmdJxL8aWOe7H/X2vjpuDp1cE/bFt4OnUwOa0eMcon/paqTkDY7/JUASDTSMa1NRgbaO11aoXWAT+BGxYk3UQrNtdGKU45yRRmCEpO34rU90SCUUxI5UDc0kyhM9Qj3Q0TBAnslvWY1QBV0ciEKdCfxMF6uisokRcyiEPNdP0Sl7eM8Fle51cxS+6JU2yXJEEjw+Kcwa0m2YmQUQFwYoNNUBYUF0rwH2ke6f05GrDZo7pE3ZO1PxFMO+WMq5Pnysp5Pq/IAm5wCnnKImelDBGnLJhRGKUM2U6H0/xMms2o3OayYlLxdgmBzKiYCpojyaIMRIraB7zYf3TV7B+OvAt0b0QZE8X+D4jAqlU6EqQ6Jl3RfemBx9CA//HpMkfpobz1yrrAvRljAVpRpKyqiFmqSQw7Ik0z+YKXtDXheoEKNaOj/lkXjZm6IH0L4/fIjhub/nbW+0POxu7ryejuWo9sB5ZnuVbz61d6511YB1Z2LZtz/btduNL41vje+PHmGqvTDT3rbnV+Pkbys+zMQ==</latexit>

softmax(
softmax(
<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>

<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>
))
[dh x dh] B
<latexit sha1_base64="zEYq/t/kb3jj4jjQXAD8GHfO+Hk=">AAAFEnicfVNdb9MwFM26AiN8bfDIi2GqaGFMbTcJJIQ0DR542RiwL2kukeM6rTs7iWxnSxvlX/DIL+EN8cof4N9wnbTQrgVXTU+vz7m+PvfGjwXXptn8tVRZrl67fmPlpnvr9p2791bX7h/rKFGUHdFIROrUJ5oJHrIjw41gp7FiRPqCnfjnb+z+yQVTmkfhoRnGrCNJL+QBp8RAyFtb+lrDvszSHL1GZ6mX8efb+StUgK0JaE9AK++4NXzYZ4Z8zuopNhEaNXIISWL6SmY6CowkaV7H6F+fhlsL6nMp0sbfJJ8gyV6RZMIaWdawkY+ANXJr6cIaDuqXHodLDBCWvAsxwDhQhGaYpXEdG6vwBlPpMO1GxoozrBOJBZfcaC8bPME8RMf5tAyC+SKlW7ssLXNrH/f360XcHjy5IuqXtqFnhYF5Y1J6KXzWAiFQBtZ9myD1BhtWMwBNw911vdX15mazWGgetMZg3RmvA2+t0sHdiCaShYYKovVZqxmbTkaU4VSw3MWJZjGh56THzgCGRDLdyYo5ylENIl0URAq+oUFFdFqREan1UPrAtM3SV/dscNHeWWKCl52Mh3FiWEjLg4JEILDTDiXqcsWoEUMAhCoOtSLaJ9A8A6MLnk0d02figpnZi1DZyXRQnD5Tki/hv2Ihu6SRlCTsPs1wQCQXwy4LSCKMbX0wwYus2ehe8FiPXUpLm1wsmMGR4j0eEiFYYLB9zIbhp29w8XTxWwa9UGwPCnwfM0VMpKASonr2ZYHe9PAjbOH/mDz8wwQ4e62sKAAuYy2IYhZmeQGpiDTDfk9FSTxT8Jy+KBQSkAAcL/lsVlYyYCBbV8dvHhy3N1tbm+0P2+s7u+PRXHEeOo+dutNyXjg7zjvnwDlyaGW50qi0K1vVL9Vv1e/VHyW1sjTWPHBmVvXnb7tRs04=</latexit>

h0 hi hi hi dims:
<latexit sha1_base64="YwgeWZ8MmCpvBq9864+tOD6jYJc=">AAAE8nicfVNLb9NAEHZCgNa8+jhyWagiEh5V0lYCCSFVwIFLS4G+pGyw1utxsu2u19pdt06M/wg3xJU/xJk/wtpJIGkCa9n+/O3M7Mw3Yz/mTJtW62eleq12/cbNpWX31u07d++trK4da5koCkdUcqlOfaKBswiODDMcTmMFRPgcTvzzN8X+yQUozWR0aAYxdAXpRSxklBhLeauVL3XsiyzN0SvUSb2MPdvJX6ISbE/A1gS0865bx4d9MORz1kixkWjYzC0liOkrkWkZGkHSvIHRv66mWw8bcyHS5t8gn2yQvTLIxGpYWA2a+dBaDd16ujCHg8alx2wRZwgLFljOYhwqQjMMadzApvDwzqbCYRpIUzhnWCcCcyaY0V529gizCB3n026WzBd5uvXLkWRu/eP+fqPki4MnJaL+SDb0ZCJgc5K82/darrey0dpslQvNg/YYbDjjdeCtVrs4kDQREBnKidaddis23YwowyiH3MWJhpjQc9KDjoUREaC7WTknOapbJkChVPaODCrZaY+MCK0HwreWRTP01b2CXLTXSUz4opuxKE4MRHR0UJhwZOUqhg4FTAE1fGABoYrZXBHtE9scY0fTKjJ1TB/4BZjZQqjoZjosT59JyRf2W0EEl1QKQaLgcYZDIhgfBBCShJuiteEEL5LmaXDBYj1WKR3J5GIOBkvFeiwinENocPGYpe2rb3D5dPFbsL1QsGcTfB+DIkYqmwlRveJnsL3p4Qe4gP+zZNEfSwtny8rKBGwxhQQyhijLS0i51ID9npJJPJPwnH+ZqA1AQqv4yB5m3UYWdiDbV8dvHhxvbba3N7c+7Gzsvh6P5pJz33noNJy289zZdd45B86RQyu/qsvVtep6zdS+1r7Vvo9Mq5Wxz7ozs2o/fgM1Dak2</latexit>

⇥<latexit sha1_base64="PXEiyqto0QO3Q9mBhKP8IXxzbXU=">AAAE7XicfVNNb9NAEHVCgGJoaeHIZaGKSPiokrYSSAipAg5cWgr0S8qGaL0eJ9vueq3ddePE8s/ghrjym7jwW1jbCSRNYC3bz29nZmfejL2IM21arZ+V6rXa9Rs3V265t++srt1d37h3omWsKBxTyaU684gGzkI4NsxwOIsUEOFxOPUu3ub7p5egNJPhkRlF0BWkH7KAUWIs1duoDOvYE2mSodeok/RS9nw3e4UKsDMF21PQzrpuHR8NwJAvaSPBRqJxM7OUIGagRKplYARJsgZG/7qabj1oLIRImn+DfLZB9osgU6txbjVqZmNrNXbrydIcDhvDHrNFnCMsmG85i3GgCE0xJFEDm9yjdz4TDlNfmtw5xToWmDPBjO6l548xC9FJNutmyWyZp1sflpK59U8HB42Czw+elogGpWzo6VTAplvuuL31zdZWq1hoEbQnYNOZrMPeRrWLfUljAaGhnGjdabci002JMoxyyFwca4gIvSB96FgYEgG6mxYjkqG6ZXwUSGXv0KCCnfVIidB6JDxrmfdBX93LyWV7ndgEL7spC6PYQEjLg4KYI6tUPm/IZwqo4SMLCFXM5orogNi+GDuVtpMzxwyAX4KZL4SKbqqD4vS5lDxhvxWEMKRSCBL6T1IcEMH4yIeAxNzkXQ2meJk0z/xLFumJSkkpk4s5GCwV67OQcA6BwfljnravgcHF08XvwPZCwb5N8EMEihipbCZE9fP/wPamjx/iHP7PkoV/LC2cLystErDF5BLICMI0KyDlUgP2+krG0VzCC/5FojYACazipT3Mu5UWdiDbV8dvEZxsb7V3trY/7m7uvZmM5orzwHnkNJy288LZc947h86xQyu/qrXqanWtJmtfa99q30vTamXic9+ZW7UfvwFyGafe</latexit>

…
<latexit sha1_base64="0yCpXupPBKPlITaMzpUGAbo6k04=">AAAE9nicfVPdbtMwFE5LgREYbCDBBTeGqaLlZ2q3SSAhpAm44GZjwP6kulSOc9J6s+PIdra0USSehDvELa/DE/AaOEkL7VpwlOTL53OOz/nOiRdxpk2r9bNSvVS7fOXq0jX3+o3lm7dWVm8fahkrCgdUcqmOPaKBsxAODDMcjiMFRHgcjrzTN/n+0RkozWS4b4YRdAXphyxglBhL9VYrX+rYE2mSoVeok/RS9mwre4kKsDkBGxPQzrpuHe8PwJDPaSPBRqJRM7OUIGagRKplYARJsgZG/7qabj1ozIVImn+DfLJBdoogE6tRbjVsZiNrNXLrycIc9hrnPWaLOEFYMN9yFuNAEZpiSKIGNrlH72QqHKa+NLlzinUsMGeCGd1LTx5hFqLDbNrNktkiT7d+Xkrm1j/u7jYKPj94UiIalLKhJxMBm5Pk3UGpsdtbWWutt4qF5kF7DNac8drrrVa72Jc0FhAayonWnXYrMt2UKMMoh8zFsYaI0FPSh46FIRGgu2kxKxmqW8ZHgVT2Dg0q2GmPlAith8KzlnlD9MW9nFy014lN8KKbsjCKDYS0PCiIObKS5YOHfKaAGj60gFDFbK6IDohtkLHjaVWZOmYA/AzMbCFUdFMdFKfPpOQJ+60ghHMqhSCh/zjFARGMD30ISMxN3t5gghdJ89Q/Y5Eeq5SUMrmYg8FSsT4LCecQGJw/Zmn7GhhcPF38FmwvFOzYBN9HoIiRymZCVD//IWxv+vgBzuH/LFn4x9LC2bLSIgFbTC6BjCBMswJSLjVgr69kHM0kPOdfJGoDkMAqXtrDrFtpYQeyfXH85sHhxnp7c33jw9ba9uvxaC45952HTsNpO8+dbeeds+ccOLTyq7pcvVu9V0tqX2vfat9L02pl7HPHmVm1H78Boaeq7w==</latexit>
3
⇥
<latexit sha1_base64="PXEiyqto0QO3Q9mBhKP8IXxzbXU=">AAAE7XicfVNNb9NAEHVCgGJoaeHIZaGKSPiokrYSSAipAg5cWgr0S8qGaL0eJ9vueq3ddePE8s/ghrjym7jwW1jbCSRNYC3bz29nZmfejL2IM21arZ+V6rXa9Rs3V265t++srt1d37h3omWsKBxTyaU684gGzkI4NsxwOIsUEOFxOPUu3ub7p5egNJPhkRlF0BWkH7KAUWIs1duoDOvYE2mSodeok/RS9nw3e4UKsDMF21PQzrpuHR8NwJAvaSPBRqJxM7OUIGagRKplYARJsgZG/7qabj1oLIRImn+DfLZB9osgU6txbjVqZmNrNXbrydIcDhvDHrNFnCMsmG85i3GgCE0xJFEDm9yjdz4TDlNfmtw5xToWmDPBjO6l548xC9FJNutmyWyZp1sflpK59U8HB42Czw+elogGpWzo6VTAplvuuL31zdZWq1hoEbQnYNOZrMPeRrWLfUljAaGhnGjdabci002JMoxyyFwca4gIvSB96FgYEgG6mxYjkqG6ZXwUSGXv0KCCnfVIidB6JDxrmfdBX93LyWV7ndgEL7spC6PYQEjLg4KYI6tUPm/IZwqo4SMLCFXM5orogNi+GDuVtpMzxwyAX4KZL4SKbqqD4vS5lDxhvxWEMKRSCBL6T1IcEMH4yIeAxNzkXQ2meJk0z/xLFumJSkkpk4s5GCwV67OQcA6BwfljnravgcHF08XvwPZCwb5N8EMEihipbCZE9fP/wPamjx/iHP7PkoV/LC2cLystErDF5BLICMI0KyDlUgP2+krG0VzCC/5FojYACazipT3Mu5UWdiDbV8dvEZxsb7V3trY/7m7uvZmM5orzwHnkNJy288LZc947h86xQyu/qrXqanWtJmtfa99q30vTamXic9+ZW7UfvwFyGafe</latexit>
<latexit sha1_base64="cDE5LnazxffjZV2goLymzw4XzaU=">AAAE9nicfVPdbtMwFE5LgREYbCDBBTeGqaLlZ2q3SSAhpAm44GZjwP6kulSOc9J6s+PIdra0USSehDvELa/DE/AaOEkL7VpwlOTL53OOz/nOiRdxpk2r9bNSvVS7fOXq0jX3+o3lm7dWVm8fahkrCgdUcqmOPaKBsxAODDMcjiMFRHgcjrzTN/n+0RkozWS4b4YRdAXphyxglBhL9VYrX+rYE2mSoVeok/RS9mwre4kKsDkBGxPQzrpuHe8PwJDPaSPBRqJRM7OUIGagRKplYARJsgZG/7qabj1ozIVImn+DfLJBdoogE6tRbjVsZiNrNXLrycIc9hrnPWaLOEFYMN9yFuNAEZpiSKIGNrlH72QqHKa+NLlzinUsMGeCGd1LTx5hFqLDbNrNktkiT7d+Xkrm1j/u7jYKPj94UiIalLKhJxMBm5Pk3UEprdtbWWutt4qF5kF7DNac8drrrVa72Jc0FhAayonWnXYrMt2UKMMoh8zFsYaI0FPSh46FIRGgu2kxKxmqW8ZHgVT2Dg0q2GmPlAith8KzlnlD9MW9nFy014lN8KKbsjCKDYS0PCiIObKS5YOHfKaAGj60gFDFbK6IDohtkLHjaVWZOmYA/AzMbCFUdFMdFKfPpOQJ+60ghHMqhSCh/zjFARGMD30ISMxN3t5gghdJ89Q/Y5Eeq5SUMrmYg8FSsT4LCecQGJw/Zmn7GhhcPF38FmwvFOzYBN9HoIiRymZCVD//IWxv+vgBzuH/LFn4x9LC2bLSIgFbTC6BjCBMswJSLjVgr69kHM0kPOdfJGoDkMAqXtrDrFtpYQeyfXH85sHhxnp7c33jw9ba9uvxaC45952HTsNpO8+dbeeds+ccOLTyq7pcvVu9V0tqX2vfat9L02pl7HPHmVm1H78BnpWq7g==</latexit>
2 ⇥
<latexit sha1_base64="PXEiyqto0QO3Q9mBhKP8IXxzbXU=">AAAE7XicfVNNb9NAEHVCgGJoaeHIZaGKSPiokrYSSAipAg5cWgr0S8qGaL0eJ9vueq3ddePE8s/ghrjym7jwW1jbCSRNYC3bz29nZmfejL2IM21arZ+V6rXa9Rs3V265t++srt1d37h3omWsKBxTyaU684gGzkI4NsxwOIsUEOFxOPUu3ub7p5egNJPhkRlF0BWkH7KAUWIs1duoDOvYE2mSodeok/RS9nw3e4UKsDMF21PQzrpuHR8NwJAvaSPBRqJxM7OUIGagRKplYARJsgZG/7qabj1oLIRImn+DfLZB9osgU6txbjVqZmNrNXbrydIcDhvDHrNFnCMsmG85i3GgCE0xJFEDm9yjdz4TDlNfmtw5xToWmDPBjO6l548xC9FJNutmyWyZp1sflpK59U8HB42Czw+elogGpWzo6VTAplvuuL31zdZWq1hoEbQnYNOZrMPeRrWLfUljAaGhnGjdabci002JMoxyyFwca4gIvSB96FgYEgG6mxYjkqG6ZXwUSGXv0KCCnfVIidB6JDxrmfdBX93LyWV7ndgEL7spC6PYQEjLg4KYI6tUPm/IZwqo4SMLCFXM5orogNi+GDuVtpMzxwyAX4KZL4SKbqqD4vS5lDxhvxWEMKRSCBL6T1IcEMH4yIeAxNzkXQ2meJk0z/xLFumJSkkpk4s5GCwV67OQcA6BwfljnravgcHF08XvwPZCwb5N8EMEihipbCZE9fP/wPamjx/iHP7PkoV/LC2cLystErDF5BLICMI0KyDlUgP2+krG0VzCC/5FojYACazipT3Mu5UWdiDbV8dvEZxsb7V3trY/7m7uvZmM5orzwHnkNJy288LZc947h86xQyu/qrXqanWtJmtfa99q30vTamXic9+ZW7UfvwFyGafe</latexit>
hi <latexit sha1_base64="MS7PqebmWexBvDkOqk8USVMUetI=">AAAE9nicfVPdbtMwFE5LgREYbCDBBTeGqaLlZ2q3SSAhpAm44GZjwP6kulSOc9J6s+PIdra0USSehDvELa/DE/AaOGkD7VpwlOTL8XeOz/nOiRdxpk2r9bNSvVS7fOXq0jX3+o3lm7dWVm8fahkrCgdUcqmOPaKBsxAODDMcjiMFRHgcjrzTN/n+0RkozWS4b4YRdAXphyxglBhr6q1WvtSxJ9IkQ69QJ+ml7NlW9hIVYLMEGyVoZ123jvcHYMjntJFgI9GomVmTIGagRKplYARJsgZG/7qabj1ozIVImn+DfLJBdoogJWuUs4bNbGRZI7eeLMxhr3HeY7aIE4QF863NYhwoQlMMSdTAJvfonUyFw9SXJndOsY4F5kwwo3vpySPMQnSYTbtZY7bI062fjyVz6x93dxuFPT+4LBENxrKhJ6WAzTJ5d7Ll9lbWWuutYqF50J6ANWey9nqr1S72JY0FhIZyonWn3YpMNyXKMMohc3GsISL0lPShY2FIBOhuWsxKhurW4qNAKnuHBhXWaY+UCK2HwrPMvCH64l5uXLTXiU3wopuyMIoNhHR8UBBzZCXLBw/5TAE1fGgBoYrZXBEdENsgY8fTqjJ1zAD4GZjZQqjopjooTp9JyRP2W0EI51QKQUL/cYoDIhgf+hCQmJu8vUGJF0nz1D9jkZ6olIxlcjEHg6VifRYSziEwOH/Mmu1rYHDxdPFbsL1QsGMTfB+BIkYqmwlR/fyHsL3p4wc4h/9jsvAP08LZstIiAVtMLoGMIEyzAlIuNWCvr2QczSQ8518kagOQwCo+5sOs25hhB7J9cfzmweHGentzfePD1tr268loLjn3nYdOw2k7z51t552z5xw4tPKruly9W71XS2pfa99q38fUamXic8eZWbUfvwGbg6rt</latexit>
1
⇥
<latexit sha1_base64="PXEiyqto0QO3Q9mBhKP8IXxzbXU=">AAAE7XicfVNNb9NAEHVCgGJoaeHIZaGKSPiokrYSSAipAg5cWgr0S8qGaL0eJ9vueq3ddePE8s/ghrjym7jwW1jbCSRNYC3bz29nZmfejL2IM21arZ+V6rXa9Rs3V265t++srt1d37h3omWsKBxTyaU684gGzkI4NsxwOIsUEOFxOPUu3ub7p5egNJPhkRlF0BWkH7KAUWIs1duoDOvYE2mSodeok/RS9nw3e4UKsDMF21PQzrpuHR8NwJAvaSPBRqJxM7OUIGagRKplYARJsgZG/7qabj1oLIRImn+DfLZB9osgU6txbjVqZmNrNXbrydIcDhvDHrNFnCMsmG85i3GgCE0xJFEDm9yjdz4TDlNfmtw5xToWmDPBjO6l548xC9FJNutmyWyZp1sflpK59U8HB42Czw+elogGpWzo6VTAplvuuL31zdZWq1hoEbQnYNOZrMPeRrWLfUljAaGhnGjdabci002JMoxyyFwca4gIvSB96FgYEgG6mxYjkqG6ZXwUSGXv0KCCnfVIidB6JDxrmfdBX93LyWV7ndgEL7spC6PYQEjLg4KYI6tUPm/IZwqo4SMLCFXM5orogNi+GDuVtpMzxwyAX4KZL4SKbqqD4vS5lDxhvxWEMKRSCBL6T1IcEMH4yIeAxNzkXQ2meJk0z/xLFumJSkkpk4s5GCwV67OQcA6BwfljnravgcHF08XvwPZCwb5N8EMEihipbCZE9fP/wPamjx/iHP7PkoV/LC2cLystErDF5BLICMI0KyDlUgP2+krG0VzCC/5FojYACazipT3Mu5UWdiDbV8dvEZxsb7V3trY/7m7uvZmM5orzwHnkNJy288LZc947h86xQyu/qrXqanWtJmtfa99q30vTamXic9+ZW7UfvwFyGafe</latexit>
<latexit sha1_base64="UxJxKSE4Fv+hebaFtaEtzDBO/9k=">AAAFD3icfVNdb9MwFM1KgRG+NnjkxTAVWhij3SaBhJAm4IGXjQH7kuoSOY7TurOTyHa2tFF+BI/8Et4Qr/wE/g3XaQvtWnDV5PT6nOvrc2/9RHBtms1fS5VL1ctXri5fc6/fuHnr9srqnSMdp4qyQxqLWJ34RDPBI3ZouBHsJFGMSF+wY//0jd0/PmNK8zg6MIOEdSTpRjzklBgIeatLX2vYl3lWoFeonXk5f7pdvEQl2JqAzQloFR23hg96zJDPeT3DJkbDRgEhSUxPyVzHoZEkK+oY/evTcGthfS5F1vib5BMk2S2TTFhDyxo0iiGwhm4tW1jDfv3c43CJPsKSBxADjENFaI5ZltSxsQqvP5UO0yA2VpxjnUosuORGe3n/EeYROiqmZRAsFind2vnIMrf2cW+vXsbtwZMrot7INvRkYmBjUrxrt0AHr7413+ozr79uJf2S6K2sNTea5ULzoDUGa8547XurlQ4OYppKFhkqiNbtVjMxnZwow6lghYtTzRJCT0mXtQFGRDLdycshKlANIgEKYwXfyKAyOq3IidR6IH1g2k7pi3s2uGivnZrwRSfnUZIaFtHRQWEqEHhpJxIFXDFqxAAAoYpDrYj2CHTOwNyCXVPH9Jg4Y2b2IlR2ch2Wp8+U5Ev4rVjEzmksJYmCxzkOieRiELCQpMLYvocTvMia9eCMJ3rsUjayycWCGRwr3uUREYKFBtvHbBhePYPLp4vfMuiFYrtQ4PuEKWJiBZUQ1bX/FOhNF9/HFv6PyaM/TICz18rLAuAy1oI4YVFelJCKWDPsd1WcJjMFz+nLQiEBCcHxEZ/NykYMGMjWxfGbB0ebG62tjc0P22s7r8ejuezccx44daflPHd2nHfOvnPo0IpTeVh5VmlWv1S/Vb9Xf4yolaWx5q4zs6o/fwMmxrKy</latexit>

[V x dh]
+ + + + +
[dh]
⇥
<latexit sha1_base64="buQ0kPDjWBeJH6grSAxRTv7QNIM=">AAAFInicfVNdb9MwFM1KgRG+NvbIi2GqaGFM7TYJEEKaNh542RiwL2kukeM4rTs7iWxnSxvl3/DIL+EN8YTEj+EmaaFZC66anl6fc33v8Y0bCa5Nu/1zoXatfv3GzcVb9u07d+/dX1p+cKzDWFF2REMRqlOXaCZ4wI4MN4KdRooR6Qp24p7v5vsnF0xpHgaHZhixriS9gPucEgMhZ3nhawO7Mk0y9AadJU7Kn29lr1EBNidgYwI6Wddu4MM+M+Rz2kywCdGolUFIEtNXMtWhbyRJsiZG//q07IbfnEmRtP4m+QRJ9ookE9YoZw1b2QhYI7uRzK3hoHnpcGhigLDkHsQAY18RmmKWRE1scoUzmEqHqReaXJxiHUssuORGO+ngCeYBOs6mZRDM5intxmVpmd34uL/fLOL5wZMWUb+0DT0rDMxak9JL4bMOCIEyyN3PEyTOYC3XDEDTsqtd9uE4Z2m1vd4uFpoFnTFYtcbrwFmudbEX0liywFBBtD7rtCPTTYkynAqW2TjWLCL0nPTYGcCASKa7aTFXGWpAxEN+qOAbGFREpxUpkVoPpQvM/PL01b08OG/vLDb+y27Kgyg2LKDlQX4sELSZDynyuGLUiCEAQhWHWhHtE7hMA6MMHk4d02figplqI1R2U+0Xp1dKciX8VyxglzSUkgTe0xT7RHIx9JhPYmHyUfAneJ41a94Fj/TYpaS0ycaCGRwq3uMBEYL5BuePahh++gYXTxu/ZXAXiu1Bge8jpogJFVRCVC9/eeBuevgRzuH/mDz4wwRYbSstCoBmcgvCiAVpVkAqQs2w21NhHFUKntEXhUIC4oPjJZ9VZSUDBrJzdfxmwfHGemdzfePD1ur2zng0F62H1mOraXWsF9a29c46sI4sWlupvart1HbrX+rf6t/rP0pqbWGsWbEqq/7rN0QJueQ=</latexit>
(x!h)
⇥
<latexit sha1_base64="buQ0kPDjWBeJH6grSAxRTv7QNIM=">AAAFInicfVNdb9MwFM1KgRG+NvbIi2GqaGFM7TYJEEKaNh542RiwL2kukeM4rTs7iWxnSxvl3/DIL+EN8YTEj+EmaaFZC66anl6fc33v8Y0bCa5Nu/1zoXatfv3GzcVb9u07d+/dX1p+cKzDWFF2REMRqlOXaCZ4wI4MN4KdRooR6Qp24p7v5vsnF0xpHgaHZhixriS9gPucEgMhZ3nhawO7Mk0y9AadJU7Kn29lr1EBNidgYwI6Wddu4MM+M+Rz2kywCdGolUFIEtNXMtWhbyRJsiZG//q07IbfnEmRtP4m+QRJ9ookE9YoZw1b2QhYI7uRzK3hoHnpcGhigLDkHsQAY18RmmKWRE1scoUzmEqHqReaXJxiHUssuORGO+ngCeYBOs6mZRDM5intxmVpmd34uL/fLOL5wZMWUb+0DT0rDMxak9JL4bMOCIEyyN3PEyTOYC3XDEDTsqtd9uE4Z2m1vd4uFpoFnTFYtcbrwFmudbEX0liywFBBtD7rtCPTTYkynAqW2TjWLCL0nPTYGcCASKa7aTFXGWpAxEN+qOAbGFREpxUpkVoPpQvM/PL01b08OG/vLDb+y27Kgyg2LKDlQX4sELSZDynyuGLUiCEAQhWHWhHtE7hMA6MMHk4d02figplqI1R2U+0Xp1dKciX8VyxglzSUkgTe0xT7RHIx9JhPYmHyUfAneJ41a94Fj/TYpaS0ycaCGRwq3uMBEYL5BuePahh++gYXTxu/ZXAXiu1Bge8jpogJFVRCVC9/eeBuevgRzuH/mDz4wwRYbSstCoBmcgvCiAVpVkAqQs2w21NhHFUKntEXhUIC4oPjJZ9VZSUDBrJzdfxmwfHGemdzfePD1ur2zng0F62H1mOraXWsF9a29c46sI4sWlupvart1HbrX+rf6t/rP0pqbWGsWbEqq/7rN0QJueQ=</latexit>
(x!h)
⇥
<latexit sha1_base64="buQ0kPDjWBeJH6grSAxRTv7QNIM=">AAAFInicfVNdb9MwFM1KgRG+NvbIi2GqaGFM7TYJEEKaNh542RiwL2kukeM4rTs7iWxnSxvl3/DIL+EN8YTEj+EmaaFZC66anl6fc33v8Y0bCa5Nu/1zoXatfv3GzcVb9u07d+/dX1p+cKzDWFF2REMRqlOXaCZ4wI4MN4KdRooR6Qp24p7v5vsnF0xpHgaHZhixriS9gPucEgMhZ3nhawO7Mk0y9AadJU7Kn29lr1EBNidgYwI6Wddu4MM+M+Rz2kywCdGolUFIEtNXMtWhbyRJsiZG//q07IbfnEmRtP4m+QRJ9ookE9YoZw1b2QhYI7uRzK3hoHnpcGhigLDkHsQAY18RmmKWRE1scoUzmEqHqReaXJxiHUssuORGO+ngCeYBOs6mZRDM5intxmVpmd34uL/fLOL5wZMWUb+0DT0rDMxak9JL4bMOCIEyyN3PEyTOYC3XDEDTsqtd9uE4Z2m1vd4uFpoFnTFYtcbrwFmudbEX0liywFBBtD7rtCPTTYkynAqW2TjWLCL0nPTYGcCASKa7aTFXGWpAxEN+qOAbGFREpxUpkVoPpQvM/PL01b08OG/vLDb+y27Kgyg2LKDlQX4sELSZDynyuGLUiCEAQhWHWhHtE7hMA6MMHk4d02figplqI1R2U+0Xp1dKciX8VyxglzSUkgTe0xT7RHIx9JhPYmHyUfAneJ41a94Fj/TYpaS0ycaCGRwq3uMBEYL5BuePahh++gYXTxu/ZXAXiu1Bge8jpogJFVRCVC9/eeBuevgRzuH/mDz4wwRYbSstCoBmcgvCiAVpVkAqQs2w21NhHFUKntEXhUIC4oPjJZ9VZSUDBrJzdfxmwfHGemdzfePD1ur2zng0F62H1mOraXWsF9a29c46sI4sWlupvart1HbrX+rf6t/rP0pqbWGsWbEqq/7rN0QJueQ=</latexit>
(x!h)
⇥
<latexit sha1_base64="buQ0kPDjWBeJH6grSAxRTv7QNIM=">AAAFInicfVNdb9MwFM1KgRG+NvbIi2GqaGFM7TYJEEKaNh542RiwL2kukeM4rTs7iWxnSxvl3/DIL+EN8YTEj+EmaaFZC66anl6fc33v8Y0bCa5Nu/1zoXatfv3GzcVb9u07d+/dX1p+cKzDWFF2REMRqlOXaCZ4wI4MN4KdRooR6Qp24p7v5vsnF0xpHgaHZhixriS9gPucEgMhZ3nhawO7Mk0y9AadJU7Kn29lr1EBNidgYwI6Wddu4MM+M+Rz2kywCdGolUFIEtNXMtWhbyRJsiZG//q07IbfnEmRtP4m+QRJ9ookE9YoZw1b2QhYI7uRzK3hoHnpcGhigLDkHsQAY18RmmKWRE1scoUzmEqHqReaXJxiHUssuORGO+ngCeYBOs6mZRDM5intxmVpmd34uL/fLOL5wZMWUb+0DT0rDMxak9JL4bMOCIEyyN3PEyTOYC3XDEDTsqtd9uE4Z2m1vd4uFpoFnTFYtcbrwFmudbEX0liywFBBtD7rtCPTTYkynAqW2TjWLCL0nPTYGcCASKa7aTFXGWpAxEN+qOAbGFREpxUpkVoPpQvM/PL01b08OG/vLDb+y27Kgyg2LKDlQX4sELSZDynyuGLUiCEAQhWHWhHtE7hMA6MMHk4d02figplqI1R2U+0Xp1dKciX8VyxglzSUkgTe0xT7RHIx9JhPYmHyUfAneJ41a94Fj/TYpaS0ycaCGRwq3uMBEYL5BuePahh++gYXTxu/ZXAXiu1Bge8jpogJFVRCVC9/eeBuevgRzuH/mDz4wwRYbSstCoBmcgvCiAVpVkAqQs2w21NhHFUKntEXhUIC4oPjJZ9VZSUDBrJzdfxmwfHGemdzfePD1ur2zng0F62H1mOraXWsF9a29c46sI4sWlupvart1HbrX+rf6t/rP0pqbWGsWbEqq/7rN0QJueQ=</latexit>
(x!h)
⇥
<latexit sha1_base64="buQ0kPDjWBeJH6grSAxRTv7QNIM=">AAAFInicfVNdb9MwFM1KgRG+NvbIi2GqaGFM7TYJEEKaNh542RiwL2kukeM4rTs7iWxnSxvl3/DIL+EN8YTEj+EmaaFZC66anl6fc33v8Y0bCa5Nu/1zoXatfv3GzcVb9u07d+/dX1p+cKzDWFF2REMRqlOXaCZ4wI4MN4KdRooR6Qp24p7v5vsnF0xpHgaHZhixriS9gPucEgMhZ3nhawO7Mk0y9AadJU7Kn29lr1EBNidgYwI6Wddu4MM+M+Rz2kywCdGolUFIEtNXMtWhbyRJsiZG//q07IbfnEmRtP4m+QRJ9ookE9YoZw1b2QhYI7uRzK3hoHnpcGhigLDkHsQAY18RmmKWRE1scoUzmEqHqReaXJxiHUssuORGO+ngCeYBOs6mZRDM5intxmVpmd34uL/fLOL5wZMWUb+0DT0rDMxak9JL4bMOCIEyyN3PEyTOYC3XDEDTsqtd9uE4Z2m1vd4uFpoFnTFYtcbrwFmudbEX0liywFBBtD7rtCPTTYkynAqW2TjWLCL0nPTYGcCASKa7aTFXGWpAxEN+qOAbGFREpxUpkVoPpQvM/PL01b08OG/vLDb+y27Kgyg2LKDlQX4sELSZDynyuGLUiCEAQhWHWhHtE7hMA6MMHk4d02figplqI1R2U+0Xp1dKciX8VyxglzSUkgTe0xT7RHIx9JhPYmHyUfAneJ41a94Fj/TYpaS0ycaCGRwq3uMBEYL5BuePahh++gYXTxu/ZXAXiu1Bge8jpogJFVRCVC9/eeBuevgRzuH/mDz4wwRYbSstCoBmcgvCiAVpVkAqQs2w21NhHFUKntEXhUIC4oPjJZ9VZSUDBrJzdfxmwfHGemdzfePD1ur2zng0F62H1mOraXWsF9a29c46sI4sWlupvart1HbrX+rf6t/rP0pqbWGsWbEqq/7rN0QJueQ=</latexit>
(x!h) [dword x dh]
… [dword]
<latexit sha1_base64="nB8pK9PGEwV+LmMCOwKc8YP2Au8=">AAADenicfVJLb9NAEN4mPEp4tXDkYhpV4hklbSU4VsCBC6JIpKmUtaLxeuyssg9rdx0SWf4RXOGX8V84sHYMqpuKkbz+PA/PN99OlAlu3XD4a6fTvXHz1u3dO7279+4/eLi3/+jc6twwHDMttLmIwKLgCseOO4EXmUGQkcBJtHhfxSdLNJZr9dWtMwwlpIonnIHzrslqVvDXx+Vsrz8cDGsLtsGoAX3S2NlsvxPSWLNconJMgLXT0TBzYQHGcSaw7NHcYgZsASlOPVQg0YZFzbcMDr0nDhJt/KNcUHsvVxQgrV3LyGdKcHN7NVY5r4tNc5e8DQuustyhYptGSS4Cp4Nq+CDmBpkTaw+AGe65BmwOBpjzEvUOL7eZo1iiaw/CZFjYpO7eohRJ/21Q4TempQQVvyhoApKLdYwJ5MKVBbXJX3ydNK/iJc9so9JqI1OPCnRUG55yBUJg4mh1tN3+NXe0Pnv0A/q7MPjJE/ycoQGnjWcCJpWwKv3dpPQpreD/Mrn6l+lhe6yiJuCHqSTQGaqirCET2iKNUqPzrEV4q74m6n8AiVd8k4/tsk2GX8jR1fXbBudHg9Hx4OjLSf/0XbOau+QJOSDPyIi8IafkIzkjY8LIgnwnP8jPzu/uQfd59+UmtbPT1DwmLeue/AGugCyM</latexit>
xi 3 xi 2
<latexit sha1_base64="Lw5rWS0utPfv8sAZFj6BQfU2/QM=">AAADenicfVJLb9NAEN4mPEp4teXIxTSqxKtREiqVYwUcuCCKRJpKWSsar8fOKvuwdtchkeUfwRV+Gf+FA2snoLqpGMnrz/PwfPPtRJng1vX7v3Za7Vu379zdvde5/+Dho8d7+wcXVueG4Yhpoc1lBBYFVzhy3Am8zAyCjASOo/n7Kj5eoLFcq69ulWEoIVU84Qycd42X04IfD8vpXrff69cWbIPBBnTJxs6n+62QxprlEpVjAqydDPqZCwswjjOBZYfmFjNgc0hx4qECiTYsar5lcOQ9cZBo4x/lgtp7taIAae1KRj5TgpvZ67HKeVNskrvkbVhwleUOFVs3SnIROB1UwwcxN8icWHkAzHDPNWAzMMCcl6hzdLXNDMUCXXMQJsPCJnX3BqVI+m+DCr8xLSWo+GVBE5BcrGJMIBeuLKhN/uKbpHkdL3hmNyot1zJ1qEBHteEpVyAEJo5WR9PtXzNH67NDP6C/C4OfPMHPGRpw2ngmYFIJy9LfTUqf0Qr+L5Orf5keNscqagJ+mEoCnaEqyhoyoS3SKDU6zxqEt+prov4HkHjF1/nYLFtn+IUcXF+/bXAx7A3e9IZfTrpn7zaruUuekkPynAzIKTkjH8k5GRFG5uQ7+UF+tn63D9sv2q/Wqa2dTc0T0rD2yR+rbyyL</latexit>
<latexit sha1_base64="0VBkd1Rw+AkOCuCd8jR6pjBnSJo=">AAADenicfVJLb9NAEN4mPEp4tXDkYhpV4tUoLpXgWAEHLogikaZS1orG67Gzyj6s3XVIZPlHcIVfxn/hwNoJqG4qRvL68zw833w7cS64dcPhr51O98bNW7d37/Tu3rv/4OHe/qNzqwvDcMS00OYiBouCKxw57gRe5AZBxgLH8fx9HR8v0Fiu1Ve3yjGSkCmecgbOu8bLacmPwmq61x8Oho0F2yDcgD7Z2Nl0vxPRRLNConJMgLWTcJi7qATjOBNY9WhhMQc2hwwnHiqQaKOy4VsFh96TBKk2/lEuaLyXK0qQ1q5k7DMluJm9Gqud18UmhUvfRiVXeeFQsXWjtBCB00E9fJBwg8yJlQfADPdcAzYDA8x5iXqHl9vMUCzQtQdhMipt2nRvUYql/zao8BvTUoJKXpQ0BcnFKsEUCuGqktr0L75OmlfJgud2o9JyLVOPCnRUG55xBUJg6mh9tN3+NXO0OXv0A/q7MPjJE/ycowGnjWcCJpOwrPzdZPQpreH/Mrn6l+lhe6yyIeCHqSXQOaqyaiAT2iKNM6OLvEV4q74h6n8AqVd8nY/tsnWGX8jw6vptg/PjQfh6cPzlpH/6brOau+QJOSDPSEjekFPykZyREWFkTr6TH+Rn53f3oPu8+3Kd2tnZ1DwmLeue/AGoXiyK</latexit>
xi 1 <latexit sha1_base64="x4eMZtKrr1x8Ug2+LL0eW7kjARo=">AAAFD3icfVNdb9MwFM1KgRG+NnjkxTAVWhij3SaBhJAm4IGXjQH7kuoSOY7TurOTyHa2tFF+BI/8Et4Qr/wE/g3XaQvtWnDV9PT6nOvrc2/8RHBtms1fS5VL1ctXri5fc6/fuHnr9srqnSMdp4qyQxqLWJ34RDPBI3ZouBHsJFGMSF+wY//0jd0/PmNK8zg6MIOEdSTpRjzklBgIeatLX2vYl3lWoFeonXk5f7pdvEQl2JqAzQloFR23hg96zJDPeT3DJkbDRgEhSUxPyVzHoZEkK+oY/evTcGthfS5F1vib5BMk2S2TTFhDyxo0iiGwhm4tW1jDfv3c43CJPsKSBxADjENFaI5ZltSxsQqvP5UO0yA2VpxjnUosuORGe3n/EeYROiqmZRAsFind2vnIMrf2cW+vXsbtwZMrot7INvRkYmBjUrxrA6ADRt+ab/WZ11+3kn5J9FbWmhvNcqF50BqDNWe89r3VSgcHMU0liwwVROt2q5mYTk6U4VSwwsWpZgmhp6TL2gAjIpnu5OUQFagGkQCFsYJvZFAZnVbkRGo9kD4wbaf0xT0bXLTXTk34opPzKEkNi+jooDAVCLy0E4kCrhg1YgCAUMWhVkR7BDpnYG7BrqljekycMTN7ESo7uQ7L02dK8iX8Vyxi5zSWkkTB4xyHRHIxCFhIUmFs38MJXmTNenDGEz12KRvZ5GLBDI4V7/KICMFCg+1jNgw/PYPLp4vfMuiFYrtQ4PuEKWJiBZUQ1bVvCvSmi+9jC//H5NEfJsDZa+VlAXAZa0GcsCgvSkhFrBn2uypOk5mC5/RloZCAhOD4iM9mZSMGDGTr4vjNg6PNjdbWxuaH7bWd1+PRXHbuOQ+cutNynjs7zjtn3zl0aMWpPKw8qzSrX6rfqt+rP0bUytJYc9eZWdWfvwFZxrLC</latexit>
xi
… breathless runners approached the
wi <latexit sha1_base64="tqdvN1Co9lflZKGA9FYLbBE3SQc=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1OC+X1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4Dx3yZxA==</latexit>
3 wi <latexit sha1_base64="cCatZM/QvTiGjI99S0wQVTzzBoU=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1O6+P1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DxGqZww==</latexit>
2 wi
<latexit sha1_base64="CH1jSvOpKNyBxULr16kKG8gaFmQ=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1Oy+L1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DwViZwg==</latexit>
1 wi
<latexit sha1_base64="5EwDIiByx9xSWpoPXUBPQBRnzvo=">AAAFD3icfVNdb9MwFM1KgRG+NnjkxTAVWhij3SaBhJAm4IGXjQH7kuoSOY7TurOTyHa2tFF+BI/8Et4Qr/wE/g3XaQvtWnDV9PT6nOvrc2/8RHBtms1fS5VL1ctXri5fc6/fuHnr9srqnSMdp4qyQxqLWJ34RDPBI3ZouBHsJFGMSF+wY//0jd0/PmNK8zg6MIOEdSTpRjzklBgIeatLX2vYl3lWoFeonXk5f7pdvEQl2JqAzQloFR23hg96zJDPeT3DJkbDRgEhSUxPyVzHoZEkK+oY/evTcGthfS5F1vib5BMk2S2TTFhDyxo0iiGwhm4tW1jDfv3c43CJPsKSBxADjENFaI5ZltSxsQqvP5UO0yA2VpxjnUosuORGe3n/EeYROiqmZRAsFind2vnIMrf2cW+vXsbtwZMrot7INvRkYmBjUrxrlaADRt+ab/WZ11+3kn5J9FbWmhvNcqF50BqDNWe89r3VSgcHMU0liwwVROt2q5mYTk6U4VSwwsWpZgmhp6TL2gAjIpnu5OUQFagGkQCFsYJvZFAZnVbkRGo9kD4wbaf0xT0bXLTXTk34opPzKEkNi+jooDAVCLy0E4kCrhg1YgCAUMWhVkR7BDpnYG7BrqljekycMTN7ESo7uQ7L02dK8iX8Vyxi5zSWkkTB4xyHRHIxCFhIUmFs38MJXmTNenDGEz12KRvZ5GLBDI4V7/KICMFCg+1jNgw/PYPLp4vfMuiFYrtQ4PuEKWJiBZUQ1bVvCvSmi+9jC//H5NEfJsDZa+VlAXAZa0GcsCgvSkhFrBn2uypOk5mC5/RloZCAhOD4iM9mZSMGDGTr4vjNg6PNjdbWxuaH7bWd1+PRXHbuOQ+cutNynjs7zjtn3zl0aMWpPKw8qzSrX6rfqt+rP0bUytJYc9eZWdWfvwFWlrLB</latexit>
31
 Learning RNNs
■ Like feed-forward neural networks, RNNs can be trained using backpropagation
and stochastic gradient descent.

■ Backpropagation through time: In addition to each layer, now gradients must

be computed with respect to each timestep. “Unroll” the RNN and treat the
parameters at each timestep as if it were another layer; apply the chain rule.
`1 `i 3 `i 2 `i 1 `i
<latexit sha1_base64="yy2TrLono1eHdV/T0RW0h/Rk3ro=">AAAFbXicfVNdTxNBFF0qVVy/QOOTxqyShlaQtECiiTEh6IM+gKiAJEzdTGfvtlNmdjczs7DtZH+or/4B/4J3ty1SqE7T7emde+4992M7ieDaNJs/5yo35qs3by3cdu/cvXf/weLSwyMdp4rBIYtFrI47VIPgERwabgQcJwqo7Aj43jl9X9x/PwOleRwdmEECbUm7EQ85owZN/tLcrxrpSJvl3jvvJPMtf7WVv/VKsDkBGxPQyttujRz0wNAftp4RE3vDRo4mSU1PSavj0Eia5XXi/evTcGth/VqIrPE3yDcMslsGmXgNC69BIx+i19CtZTM17NfPfY5F9D0ieYA2xCRUlFkCWVInpmD4/UvhCAtiU5At0akkgktutG/7K4RH3lF+mYbGfBbTrZ2PWubWvu7t1Ut7kXhSotcbtc1bLRuYNybSR8TVFhLRpV90vwiQ+f21gtNHTuNqmT3MR8qJ2z16NtgRKeTjCgMQmIyAEL5t5fnE0Cv/uf7icnO9WR7vOmiNwbIzPvv+UkWRIGaphMgwQbU+aTUT07ZUGc4EoIxUQ0LZKe3CCcKIStBtW2rLvRpaAi+MFX4j45XWywxLpdYD2UHPYuT66l1hnHV3kprwTdvyKEkNRGyUKEyFh70pVtsLuAJmxAABZYqjVo/1KDbI4AuA3byUpgfiDMx0IUy2rQ7L7FOSOjKfJmejQl2iIIJzFktJo+ClJSGVXAwCCGkqTLFV4QTP6tdacMYTPW7dRUgBhsSKd3lEhYDQkOIxbcafniHl0yUfAAekYBdVf05AURMrVEJVt3gPcWBd8pwU8H+ePLrwRDhdli0FYDFFX+IEIpuXkIlYA+l0VZwmU4Kv8UuhGICGOIaRP0zTRh64pa2rO3kdHG2stzbXN75sLW/vjPd1wXnivHDqTst57Ww7H51959BhlU+VuJJVBvO/q4+rT6vPRq6VuTHnkTN1qit/AFGi1YI=</latexit>
h1 <latexit sha1_base64="9OKHVvLZw8dWd+0FJc7OO6aUzKU=">AAAFcXicfVPbbtNAEHVDA8XcWnhCvCytIhJaqqStBBJCqgoPvLQU6AWpG6LNepxsumtbu+vWqfGH8gP8AD/A2E5K0gQ2inMyO2fmzMXdSApjm82fC5Vbi9Xbd5buuvfuP3j4aHnl8YkJY83hmIcy1N+6zIAUARxbYSV8izQw1ZVw2j1/n9+fXoA2IgyO7DCCtmK9QPiCM4umzsrCrxrtqjTJyDtylnRS8Wone0sKsD0GW2PQytpujR71wbLvaT2hNiRXjQxNitm+VqkJfatYktUp+den4db8+kyIpPE3yFcMsl8EGXtd5V7DRnaFXlduLZmr4bB+2RFYxIBQJTy0Iaa+ZjylkER1anNGZzARjnIvtDk5pSZWVAolrOmkgxdUBOQkm6ShMZvHdGuXZcvc2peDg3phzxOPSyT9sm1kvWhg1hhLL4nrLSSiyyDvfh4g6Qw2cs4AOY2bZfYxHy0mnh6wi+GejCEbVeiBxGQUpCwHl41N/dF/t7O81txsFofMgtYIrDmjc9hZqWjqhTxWEFgumTFnrWZk2ynTVnAJKCU2EDF+znpwhjBgCkw7LfRlpIYWj/ihxm9gSWGdZKRMGTNUXfTMx25u3uXGeXdnsfXftFMRRLGFgJeJ/FgS7E++3sQTGriVQwSMa4FaCe8zbJLFlwA7OpGmD/IC7HQhXLVT4xfZpyR1VTZNTspCXaohgEseKsUC72VKfaaEHHrgs1jafLP8MZ7Xrw3vQkRm1LrrkBIsDbXoiYBJCb6l+WPajD99S4unSz8ADkjDPqr+FIFmNtSohOle/i7iwHr0Oc3h/zxFcO2JcLqstBCAxeR9CSMI0qyAXIYGaLenwziaEjzDL4RiAObjGEp/mKaVHrilrZs7OQtOtjZb25tbn3fWdvdG+7rkPHNWnbrTcl47u85H59A5dnhlv2IqPyrZ4u/q0yqprpaulYUR54kzdarrfwA/Cdba</latexit>
hi 3 <latexit sha1_base64="oDli+/BzSU+NcK+rduUzIkkQ6eg=">AAAFcXicfVPbbtNAEHVDA8XcWnhCvCytIhJaqqStBBJCqgoPvLQU6AWpG6LNepxsumtbu+vWqfGH8gP8AD/A2E5K0gQ2inMyO2fmzMXdSApjm82fC5Vbi9Xbd5buuvfuP3j4aHnl8YkJY83hmIcy1N+6zIAUARxbYSV8izQw1ZVw2j1/n9+fXoA2IgyO7DCCtmK9QPiCM4umzsrCrxrtqjTJyDtylnRS8Wone0sKsD0GW2PQytpujR71wbLvaT2hNiRXjQxNitm+VqkJfatYktUp+den4db8+kyIpPE3yFcMsl8EGXtd5V7DRnaFXlduLZmr4bB+2RFYxIBQJTy0Iaa+ZjylkER1anNGZzARjnIvtDk5pSZWVAolrOmkgxdUBOQkm6ShMZvHdGuXZcvc2peDg3phzxOPSyT9sm1kvWhg1hhLL4nrLSSiyyDvfh4g6Qw2cs4AOY2bZfYxHy0mnh6wi+GejCEbVeiBxGQUpCznlY1N/dF/t7O81txsFofMgtYIrDmjc9hZqWjqhTxWEFgumTFnrWZk2ynTVnAJKCU2EDF+znpwhjBgCkw7LfRlpIYWj/ihxm9gSWGdZKRMGTNUXfTMx25u3uXGeXdnsfXftFMRRLGFgJeJ/FgS7E++3sQTGriVQwSMa4FaCe8zbJLFlwA7OpGmD/IC7HQhXLVT4xfZpyR1VTZNTspCXaohgEseKsUC72VKfaaEHHrgs1jafLP8MZ7Xrw3vQkRm1LrrkBIsDbXoiYBJCb6l+WPajD99S4unSz8ADkjDPqr+FIFmNtSohOle/i7iwHr0Oc3h/zxFcO2JcLqstBCAxeR9CSMI0qyAXIYGaLenwziaEjzDL4RiAObjGEp/mKaVHrilrZs7OQtOtjZb25tbn3fWdvdG+7rkPHNWnbrTcl47u85H59A5dnhlv2IqPyrZ4u/q0yqprpaulYUR54kzdarrfwA4qdbY</latexit>
hi 2 <latexit sha1_base64="e7zliX89eY4wMgDLtm0kcMLg5qU=">AAAFcXicfVPbbtNAEHVDA8XcWnhCvCytIhJaqqStBBJCqgoPvLQU6AWpG6LNepxsumtbu+vWqfGH8gP8AD/A2E5K0gQ2in00O2fmzMXdSApjm82fC5Vbi9Xbd5buuvfuP3j4aHnl8YkJY83hmIcy1N+6zIAUARxbYSV8izQw1ZVw2j1/n9+fXoA2IgyO7DCCtmK9QPiCM4umzsrCrxrtqjTJyDtylnRS8Wone0sKsD0GW2PQytpujR71wbLvaT2hNiRXjQxNitm+VqkJfatYktUp+dev4db8+kyIpPE3yFcMsl8EGXtd5V7DRnaFXlduLZmr4bB+2RFYxIBQJTy0Iaa+ZjylkER1anNGZzARjnIvtDk5pSZWVAolrOmkgxdUBOQkm6ShMZvHdGuXZcvc2peDg3phzxOPSyT9sm1kvWhg1hhLL4nrLSSiyyDvfh4g6Qw2cs4AOY2bZfYxHy0mnh6wi+GejCEbVeiBxGQUpCzzZWPTKH/mdpbXmpvN4pBZ0BqBNWd0DjsrFU29kMcKAsslM+as1YxsO2XaCi4BpcQGIsbPWQ/OEAZMgWmnhb6M1NDiET/U+A8sKayTjJQpY4aqi5752M3Nu9w47+4stv6bdiqCKLYQ8DKRH0uC/cnXm3hCA7dyiIBxLVAr4X2GTbL4EWBHJ9L0QV6AnS6Eq3Zq/CL7lKSuyqbJSVmoSzUEcMlDpVjgvUypz5SQQw98Fkubb5Y/xvP6teFdiMiMWncdUoKloRY9ETApwbc0f0yb8dW3tHi69APggDTso+pPEWhmQ41KmO7l3yIOrEef0xz+z1ME154Ip8tKCwFYTN6XMIIgzQrIZWiAdns6jKMpwTP8QigGYD6OofSHaVrpgVvaurmTs+Bka7O1vbn1eWdtd2+0r0vOM2fVqTst57Wz63x0Dp1jh1f2K6byo5It/q4+rZLqaulaWRhxnjhTp7r+BzJJ1tY=</latexit>
hi 1 <latexit sha1_base64="4WQNsvpdGNPfYJ3MHBFQSRY3bFU=">AAAFa3icfVPbbtNAEHVDA8XcWvoGPBiqiISWKmkrgYSQqoIELy0FepOywdqsx8mmu7a1u06dWv5Q3vgB/oFZJylNW9gozsnsnJkzF3cTwbVpNn/OVW7NV2/fWbjr3rv/4OGjxaXHRzpOFYNDFotYnXSpBsEjODTcCDhJFFDZFXDcPf1g74+HoDSPowMzSqAjaS/iIWfUoMlfmvtVI12ZZ4X33mtnfs5fbxXvvBJsTsHGFLSKjlsjB30w9Edez4iJvfNGgSZJTV/JXMehkTQr6sT716fh1sL6tRBZ42+Q7xhktwwy9Tq3XqNGcY5e524tu1HDfv3M51jEwCOSB2hDTEJFWU4gS+rEWIY/uBSOsCA2lpwTnUoiuORG+/ngJeGRd1RcpqGxuInp1s7GLXNr3/b26qXdJp6W6PXHbfNWywYWjan0MXG1hUR0Gdju2wCZP1iznAFyGlfL7GM+Uk4836PD0Y5IoZhUGIDAZASE8Hkx/WtzF4XrL64015vl8a6D1gSsOJOz7y9VFAlilkqIDBNU63armZhOTpXhTACKSDUklJ3SHrQRRlSC7uSlssKroSXwwljhNzJeab3MyKnUeiS76GkHrq/eWeNNd+3UhG87OY+S1EDExonCVHjYGbvYXsAVMCNGCChTHLV6rE+xPQbXH3t5KU0fxBDMbCFMdnIdltlnJHVlMUvOxoW6REEEZyyWkkbBq5yEVHIxCiCkqTB2p8Ipvqlfa8GQJ3rSuouQAgyJFe/xiAoBoSH2MWvGn74h5dMlHwEHpGAXVX9JQFETK1RCVc++hTiwHnlOLPyfJ48uPBHOlpWXArAY25c4gSgvSshErIF0eypOkxnB1/ilUAxAQxzD2B9maWMP3NLW1Z28Do421lub6xtft1a2dyb7uuA8dV44daflvHG2nc/OvnPosMqniqwMK2fzv6vL1SfVZ2PXytyEs+zMnGrtDw6+1OY=</latexit>
hi

h1 hi 2 hi 1 hi
h0 h0 hi 3 hi hi 2 hi 2 hi 1 hi 1 hi
…
<latexit sha1_base64="YwgeWZ8MmCpvBq9864+tOD6jYJc=">AAAE8nicfVNLb9NAEHZCgNa8+jhyWagiEh5V0lYCCSFVwIFLS4G+pGyw1utxsu2u19pdt06M/wg3xJU/xJk/wtpJIGkCa9n+/O3M7Mw3Yz/mTJtW62eleq12/cbNpWX31u07d++trK4da5koCkdUcqlOfaKBswiODDMcTmMFRPgcTvzzN8X+yQUozWR0aAYxdAXpRSxklBhLeauVL3XsiyzN0SvUSb2MPdvJX6ISbE/A1gS0865bx4d9MORz1kixkWjYzC0liOkrkWkZGkHSvIHRv66mWw8bcyHS5t8gn2yQvTLIxGpYWA2a+dBaDd16ujCHg8alx2wRZwgLFljOYhwqQjMMadzApvDwzqbCYRpIUzhnWCcCcyaY0V529gizCB3n026WzBd5uvXLkWRu/eP+fqPki4MnJaL+SDb0ZCJgc5K82/darrey0dpslQvNg/YYbDjjdeCtVrs4kDQREBnKidaddis23YwowyiH3MWJhpjQc9KDjoUREaC7WTknOapbJkChVPaODCrZaY+MCK0HwreWRTP01b2CXLTXSUz4opuxKE4MRHR0UJhwZOUqhg4FTAE1fGABoYrZXBHtE9scY0fTKjJ1TB/4BZjZQqjoZjosT59JyRf2W0EEl1QKQaLgcYZDIhgfBBCShJuiteEEL5LmaXDBYj1WKR3J5GIOBkvFeiwinENocPGYpe2rb3D5dPFbsL1QsGcTfB+DIkYqmwlRveJnsL3p4Qe4gP+zZNEfSwtny8rKBGwxhQQyhijLS0i51ID9npJJPJPwnH+ZqA1AQqv4yB5m3UYWdiDbV8dvHhxvbba3N7c+7Gzsvh6P5pJz33noNJy289zZdd45B86RQyu/qsvVtep6zdS+1r7Vvo9Mq5Wxz7ozs2o/fgM1Dak2</latexit>
<latexit sha1_base64="GHe3MuhiRgDSc+8P82XaFNzJO24=">AAAFanicfVNbb9MwFM7KCiPctvGE9hIYFS0bU7NNAgkhTQMhXjYG7IJUl8p1nNadnUS206WN/EN54g/wIzhJ2tGsBVdNvx6f75zvXNKNOFO62fy5VLm1XL19Z+Wufe/+g4ePVtfWz1UYS0LPSMhD+b2LFeUsoGeaaU6/R5Ji0eX0onv5Pru/GFKpWBic6lFE2wL3AuYzgjWYOmtLv2qoK9LEOO+cVtJJ2at989bJwd4U7E6Ba9p2DZ32qcY/0nqCdOiMGwZMAuu+FKkKfS1wYurI+denYdf8+lyIpPE3yDcIcpQHmXqNM69Rw4zBa2zXkoUaTupXHQZFDBwkmAc2wMiXmKSIJlEd6YzRGcyEQ8QLdUZOkYoF4kwwrTrp4AVigXNuZmlgNIuYdu2qaJld+3p8XM/tWeJpiU6/aJuzlTfQNKbSC+KWC0RwGWTdzwIkncF2xhkAp3GzzD7kQ/nE02M8HB3ymJpJhR7lRTLXmJl/TWPszupmc6eZH2ceuBOwaU3OSWetIpEXkljQQBOOlWq5zUi3Uyw1I5yChljRCJNL3KMtgAEWVLXTXJhxamDxHD+U8A20k1tnGSkWSo1EFzyzeaubd5lx0V0r1v6bdsqCKNY0IEUiP+YONCbba8djkhLNRwAwkQy0OqSPoTsath9aOZOmT/mQ6nIhRLRT5efZS5K6wpTJSVGojSQN6BUJhcCB9zJFPhaMjzzq45jrbKX8KV7Ur21vyCI1ad11SE41CiXrsQBzTn2NskfZDD99jfKnjT5QGJCkR6D6c0Ql1qEEJVj2spcQBtZDT1EG/+fJgmtPgOWy0lwAFJP1JYxokJocEh4qiro9GcZRSfAcPxcKAbAPYyj8aZlWeMCWujd3ch6c7+64ezu7X/Y3Dw4n+7pibVjPrLrlWq+tA+uTdWKdWaTyscIrcWW4/Lu6Xn1S3ShcK0sTzmOrdKrP/wArANQy</latexit>

<latexit sha1_base64="0yCpXupPBKPlITaMzpUGAbo6k04=">AAAE9nicfVPdbtMwFE5LgREYbCDBBTeGqaLlZ2q3SSAhpAm44GZjwP6kulSOc9J6s+PIdra0USSehDvELa/DE/AaOEkL7VpwlOTL53OOz/nOiRdxpk2r9bNSvVS7fOXq0jX3+o3lm7dWVm8fahkrCgdUcqmOPaKBsxAODDMcjiMFRHgcjrzTN/n+0RkozWS4b4YRdAXphyxglBhL9VYrX+rYE2mSoVeok/RS9mwre4kKsDkBGxPQzrpuHe8PwJDPaSPBRqJRM7OUIGagRKplYARJsgZG/7qabj1ozIVImn+DfLJBdoogE6tRbjVsZiNrNXLrycIc9hrnPWaLOEFYMN9yFuNAEZpiSKIGNrlH72QqHKa+NLlzinUsMGeCGd1LTx5hFqLDbNrNktkiT7d+Xkrm1j/u7jYKPj94UiIalLKhJxMBm5Pk3UGpsdtbWWutt4qF5kF7DNac8drrrVa72Jc0FhAayonWnXYrMt2UKMMoh8zFsYaI0FPSh46FIRGgu2kxKxmqW8ZHgVT2Dg0q2GmPlAith8KzlnlD9MW9nFy014lN8KKbsjCKDYS0PCiIObKS5YOHfKaAGj60gFDFbK6IDohtkLHjaVWZOmYA/AzMbCFUdFMdFKfPpOQJ+60ghHMqhSCh/zjFARGMD30ISMxN3t5gghdJ89Q/Y5Eeq5SUMrmYg8FSsT4LCecQGJw/Zmn7GhhcPF38FmwvFOzYBN9HoIiRymZCVD//IWxv+vgBzuH/LFn4x9LC2bLSIgFbTC6BjCBMswJSLjVgr69kHM0kPOdfJGoDkMAqXtrDrFtpYQeyfXH85sHhxnp7c33jw9ba9uvxaC45952HTsNpO8+dbeeds+ccOLTyq7pcvVu9V0tqX2vfat9L02pl7HPHmVm1H78Boaeq7w==</latexit>

<latexit sha1_base64="G9Ubti2R4WGSeAwX+o/GFEJo26U=">AAAFbnicfVPtbtMwFM3KCiN8bSDxByECU0XLxtR2k0BCSNPgB0LaGLAvqS6V4zitOzuJbKdLG/lB+c0L8AjcJO1otoKrpqfX99x77kfciDOlm82fS5Uby9Wbt1Zu23fu3rv/YHXt4YkKY0noMQl5KM9crChnAT3WTHN6FkmKhcvpqXv+Ibs/HVGpWBgc6XFEuwL3A+YzgjWYemtLv2rIFWlinPdOJ+ml7PWOeefkYHsG2jPQMl27ho4GVOMfaT1BOnQmDQMmgfVAilSFvhY4MXXk/OvTsGt+/VqIpPE3yHcIsp8HmXlNMq9xw0zAa2LXkoUaDusXPQZFDB0kmAc2wMiXmKSIJlEd6YzRG86FQ8QLdUZOkYoF4kwwrXrp8CVigXNi5mlgNIuYdu2iaJld+3ZwUM/tWeJZic6gaJuzkTfQNGbSC+JGC4jgMsy6nwVIesPNjDMETuNqmQPIh/KJpwd4NN7jMTXTCj3KZ8naxpT+bxtj91bXm1vN/DjXQWsK1q3pOeytVSTyQhILGmjCsVKdVjPS3RRLzQinoCNWNMLkHPdpB2CABVXdNBdnnBpYPMcPJXwD7eTWeUaKhVJj4YJnNnN19S4zLrrrxNp/201ZEMWaBqRI5MfcgeZku+14TFKi+RgAJpKBVocMMHRIwxsA7ZxLM6B8RHW5ECK6qfLz7CVJrjBlclIUaiNJA3pBQiFw4L1KkY8F42OP+jjmOlsrf4YX9WvTG7FITVt3GZJTjULJ+izAnFNfo+xRNsPPQKP8aaOPFAYk6T6o/hJRiXUoQQmW/exFhIH10XOUwf95suDSE2C5rDQXAMVkfQkjGqQmh4SHiiK3L8M4Kgm+xs+FQgDswxgKf1qmFR6wpa2rO3kdnLS3Wttb7a8767t7031dsZ5YL6y61bLeWLvWJ+vQOrZI5XMlqowrk+Xf1cfVp9VnhWtlacp5ZJVOtf4HE3vVig==</latexit>
3 <latexit sha1_base64="cDE5LnazxffjZV2goLymzw4XzaU=">AAAE9nicfVPdbtMwFE5LgREYbCDBBTeGqaLlZ2q3SSAhpAm44GZjwP6kulSOc9J6s+PIdra0USSehDvELa/DE/AaOEkL7VpwlOTL53OOz/nOiRdxpk2r9bNSvVS7fOXq0jX3+o3lm7dWVm8fahkrCgdUcqmOPaKBsxAODDMcjiMFRHgcjrzTN/n+0RkozWS4b4YRdAXphyxglBhL9VYrX+rYE2mSoVeok/RS9mwre4kKsDkBGxPQzrpuHe8PwJDPaSPBRqJRM7OUIGagRKplYARJsgZG/7qabj1ozIVImn+DfLJBdoogE6tRbjVsZiNrNXLrycIc9hrnPWaLOEFYMN9yFuNAEZpiSKIGNrlH72QqHKa+NLlzinUsMGeCGd1LTx5hFqLDbNrNktkiT7d+Xkrm1j/u7jYKPj94UiIalLKhJxMBm5Pk3UEprdtbWWutt4qF5kF7DNac8drrrVa72Jc0FhAayonWnXYrMt2UKMMoh8zFsYaI0FPSh46FIRGgu2kxKxmqW8ZHgVT2Dg0q2GmPlAith8KzlnlD9MW9nFy014lN8KKbsjCKDYS0PCiIObKS5YOHfKaAGj60gFDFbK6IDohtkLHjaVWZOmYA/AzMbCFUdFMdFKfPpOQJ+60ghHMqhSCh/zjFARGMD30ISMxN3t5gghdJ89Q/Y5Eeq5SUMrmYg8FSsT4LCecQGJw/Zmn7GhhcPF38FmwvFOzYBN9HoIiRymZCVD//IWxv+vgBzuH/LFn4x9LC2bLSIgFbTC6BjCBMswJSLjVgr69kHM0kPOdfJGoDkMAqXtrDrFtpYQeyfXH85sHhxnp7c33jw9ba9uvxaC45952HTsNpO8+dbeeds+ccOLTyq7pcvVu9V0tqX2vfat9L02pl7HPHmVm1H78BnpWq7g==</latexit>

<latexit sha1_base64="ERzsA9UkIsQB3oYz4ipdKKG5eIE=">AAAFbnicfVNdb9MwFM3KCiN8bSDxghCBqSJlY2q7SSAhpGnwgJA2BmxjUl0ix3Fad3YS2U6XNsoP5Zk/wE/gJmm3Ziu4anp6fc+9537EjThTutX6tVS7sVy/eWvltnnn7r37D1bXHp6oMJaEHpOQh/LUxYpyFtBjzTSnp5GkWLic/nDPPuT3P0ZUKhYGR3oc0Z7A/YD5jGANJmdt6XcDuSJNMuu91U2clL3eyd5ZBdiegc4MtLOe2UBHA6rxz9ROkA6tSTMDk8B6IEWqQl8LnGQ2sv71aZoN374WImleBvkOQfaLIDOvSe41bmYT8JqYjWShhkP73GFQxNBCgnlgA4x8iUmKaBLZSOcMZzgXDhEv1Dk5RSoWiDPBtHLS4UvEAuskm6eBMVvENBvnZcvMxreDA7uw54lnJVqDsm3WRtHArDmTXhI32kAEl2He/TxA4gw3c84QOM2rZQ4gHyomnh7g0XiPxzSbVuhRfpksq/zvZJnprK63tlrFsa6D9hSsG9Nz6KzVJPJCEgsaaMKxUt12K9K9FEvNCKegI1Y0wuQM92kXYIAFVb20EJdZDbB4lh9K+AbaKqzzjBQLpcbCBc985urqXW5cdNeNtf+2l7IgijUNSJnIj7kFzcl32/KYpETzMQBMJAOtFhlg6JCGNwDaOZdmQPmI6mohRPRS5RfZK5JckVXJSVmoiSQN6DkJhcCB9ypFPhaMjz3q45jrfK38GV7Ur01vxCI1bd1FSE41CiXrswBzTn2N8kfVDD8DjYqniT5SGJCk+6D6S0Ql1qEEJVj28xcRBtZHz1EO/+fJggtPgNWy0kIAFJP3JYxokGYFJDxUFLl9GcZRRfA1fiEUAmAfxlD60yqt9IAtbV/dyevgpLPV3t7qfN1Z392b7uuK8cR4YdhG23hj7BqfjEPj2CC1z7WoNq5Nlv/UH9ef1p+VrrWlKeeRUTl1+y8NG9WI</latexit>
<latexit sha1_base64="MS7PqebmWexBvDkOqk8USVMUetI=">AAAE9nicfVPdbtMwFE5LgREYbCDBBTeGqaLlZ2q3SSAhpAm44GZjwP6kulSOc9J6s+PIdra0USSehDvELa/DE/AaOGkD7VpwlOTL8XeOz/nOiRdxpk2r9bNSvVS7fOXq0jX3+o3lm7dWVm8fahkrCgdUcqmOPaKBsxAODDMcjiMFRHgcjrzTN/n+0RkozWS4b4YRdAXphyxglBhr6q1WvtSxJ9IkQ69QJ+ml7NlW9hIVYLMEGyVoZ123jvcHYMjntJFgI9GomVmTIGagRKplYARJsgZG/7qabj1ozIVImn+DfLJBdoogJWuUs4bNbGRZI7eeLMxhr3HeY7aIE4QF863NYhwoQlMMSdTAJvfonUyFw9SXJndOsY4F5kwwo3vpySPMQnSYTbtZY7bI062fjyVz6x93dxuFPT+4LBENxrKhJ6WAzTJ5d7Ll9lbWWuutYqF50J6ANWey9nqr1S72JY0FhIZyonWn3YpMNyXKMMohc3GsISL0lPShY2FIBOhuWsxKhurW4qNAKnuHBhXWaY+UCK2HwrPMvCH64l5uXLTXiU3wopuyMIoNhHR8UBBzZCXLBw/5TAE1fGgBoYrZXBEdENsgY8fTqjJ1zAD4GZjZQqjopjooTp9JyRP2W0EI51QKQUL/cYoDIhgf+hCQmJu8vUGJF0nz1D9jkZ6olIxlcjEHg6VifRYSziEwOH/Mmu1rYHDxdPFbsL1QsGMTfB+BIkYqmwlR/fyHsL3p4wc4h/9jsvAP08LZstIiAVtMLoGMIEyzAlIuNWCvr2QczSQ8518kagOQwCo+5sOs25hhB7J9cfzmweHGentzfePD1tr268loLjn3nYdOw2k7z51t552z5xw4tPKruly9W71XS2pfa99q38fUamXic8eZWbUfvwGbg6rt</latexit>

<latexit sha1_base64="uJ/AoZNwEmlZRI8BCP1go/0RtMo=">AAAFanicfVNbb9MwFM7KCiPctvGE9mIYFS0bU7tNAgkhTQMhXjYG7CbVJXIcp3VnJ5HtdGmj/FCe+AP8CE6Sdmu3gqOkn47Pd853LnUjwbVpNn8tVO4sVu/eW7pvP3j46PGT5ZXVUx3GirITGopQnbtEM8EDdmK4Eew8UoxIV7Az9+Jjfn82YErzMDg2w4h1JOkG3OeUGDA5Kwu/a9iVaZKhD6idOCl/s5u9RwXYmYDtCWhlHbuGj3vMkJ9pPcEmRKNGBiZJTE/JVIe+kSTJ6hj962nYNb9+K0TSuA7yA4IcFEEmXqPca9jIRuA1smvJXA1H9UuHQxF9hCX3wAYY+4rQFLMkqmOTM5z+VDhMvdDk5BTrWGLBJTfaSfuvMA/QaTZNA2M2j2nXLsuW2bXvh4f1wp4nnpSIemXb0EbRwKwxkV4SN1pABJd+3v08QOL0N3NOHziNm2X2IB8uJp4eksFwX8QsG1foMVEk49k1LhJntrO83txqFgfdBq0xWLfG58hZqSjshTSWLDBUEK3brWZkOilRhlPBQEOsWUToBemyNsCASKY7aSEsQzWweMgPFbyBQYV1mpESqfVQuuCZz1vfvMuN8+7asfHfdVIeRLFhAS0T+bFA0Jh8r5HHFaNGDAEQqjhoRbRHoDsGth9aOZWmx8SAmdlCqOyk2i+yz0hyZTZLTspCbaxYwC5pKCUJvNcp9onkYugxn8TC5CvlT/C8fm16Ax7pceuuQgpmcKh4lwdECOYbnH9mzfDTM7j42vgTgwEpdgCqv0ZMERMqUEJUN/8TwsC6+DnO4f88eXDlCXC2rLQQAMXkfQkjFqRZAakINcNuV4VxNCP4Fr8QCgGID2Mo/dksrfSALW3d3Mnb4HR7q7Wztf1td31vf7yvS9aa9cKqWy3rrbVnfbGOrBOLVj5XRCWuDBb/VFerz6prpWtlYcx5as2c6su/qfPUCQ==</latexit>
<latexit sha1_base64="UxJxKSE4Fv+hebaFtaEtzDBO/9k=">AAAFD3icfVNdb9MwFM1KgRG+NnjkxTAVWhij3SaBhJAm4IGXjQH7kuoSOY7TurOTyHa2tFF+BI/8Et4Qr/wE/g3XaQvtWnDV5PT6nOvrc2/9RHBtms1fS5VL1ctXri5fc6/fuHnr9srqnSMdp4qyQxqLWJ34RDPBI3ZouBHsJFGMSF+wY//0jd0/PmNK8zg6MIOEdSTpRjzklBgIeatLX2vYl3lWoFeonXk5f7pdvEQl2JqAzQloFR23hg96zJDPeT3DJkbDRgEhSUxPyVzHoZEkK+oY/evTcGthfS5F1vib5BMk2S2TTFhDyxo0iiGwhm4tW1jDfv3c43CJPsKSBxADjENFaI5ZltSxsQqvP5UO0yA2VpxjnUosuORGe3n/EeYROiqmZRAsFind2vnIMrf2cW+vXsbtwZMrot7INvRkYmBjUrxrt0AHr7413+ozr79uJf2S6K2sNTea5ULzoDUGa8547XurlQ4OYppKFhkqiNbtVjMxnZwow6lghYtTzRJCT0mXtQFGRDLdycshKlANIgEKYwXfyKAyOq3IidR6IH1g2k7pi3s2uGivnZrwRSfnUZIaFtHRQWEqEHhpJxIFXDFqxAAAoYpDrYj2CHTOwNyCXVPH9Jg4Y2b2IlR2ch2Wp8+U5Ev4rVjEzmksJYmCxzkOieRiELCQpMLYvocTvMia9eCMJ3rsUjayycWCGRwr3uUREYKFBtvHbBhePYPLp4vfMuiFYrtQ4PuEKWJiBZUQ1bX/FOhNF9/HFv6PyaM/TICz18rLAuAy1oI4YVFelJCKWDPsd1WcJjMFz+nLQiEBCcHxEZ/NykYMGMjWxfGbB0ebG62tjc0P22s7r8ejuezccx44daflPHd2nHfOvnPo0IpTeVh5VmlWv1S/Vb9Xf4yolaWx5q4zs6o/fwMmxrKy</latexit>

h1 hi 3 hi 2 hi 1 hi
x1 xi 3 xi 2 xi 1 xi
…
<latexit sha1_base64="PI7IHyJKA9IQbZQSEk3sbbjAszQ=">AAAFbnicfVPbbtNAEHVDA8XcWpB4QQhDFeHQUiVpJZAQUlV4QEgtBXqTssFar9fJpru2tbtOnVj+UJ75AT6BsROXuA1sFOdkds7MmYvdiDOlW62fS7Uby/Wbt1Zum3fu3rv/YHXt4YkKY0noMQl5KM9crChnAT3WTHN6FkmKhcvpqXv+Ib8/HVGpWBgc6XFEewL3A+YzgjWYnLWlXw3kijTJrPdWN3FS9none2cVYLsEnRK0s57ZQEcDqvGP1E6QDq1JMwOTwHogRapCXwucZDay/vVpmg3fvhYiaf4N8h2C7BdBSq9J7jVuZhPwmpiNZKGGQ/vCYVDE0EKCeWADjHyJSYpoEtlI5wxnOBcOES/UOTlFKhaIM8G0ctLhS8QC6ySbp4ExW8Q0GxfTlpmNbwcHdmHPE5clWoNp26yNooFZs5Q+JW60gQguw7z7eYDEGW7mnCFwmlfLHEA+VEw8PcCj8R6PaTar0KO8TNbJsvL/bHiZ6ayut7ZaxbGug/YMrBuzc+is1STyQhILGmjCsVLddivSvRRLzQinoCNWNMLkHPdpF2CABVW9tBCXWQ2weJYfSvgG2iqs84wUC6XGwgXPfObq6l1uXHTXjbX/tpeyIIo1Dcg0kR9zC5qT77blMUmJ5mMAmEgGWi0ywNAhDW8AtHMuzYDyEdXVQojopcovslckuSKrkpNpoSaSNKAXJBQCB96rFPlYMD72qI9jrvO18ku8qF+b3ohFata6y5CcahRK1mcB5pz6GuWPqhl+BhoVTxN9pDAgSfdB9ZeISqxDCUqw7OcvIgysj56jHP7PkwWXngCrZaWFACgm70sY0SDNCkh4qChy+zKMo4rga/xCKATAPoxh6k+rtKkHbGn76k5eByedrfb2Vufrzvru3mxfV4wnxgvDNtrGG2PX+GQcGscGqX2uRbVxbbL8u/64/rT+bOpaW5pxHhmVU7f/AEMj1Zk=</latexit>

<latexit sha1_base64="RCp13GpHpKU2YLvEzr1hJ/OgKcs=">AAAFanicfVNbb9MwFM7KCiPctvGE9hIYFS0bU7tNAgkhTQMhXjYG7IJUl8hxnNadnUS206W18kN54g/wIzhJ29GsBVdNvx6f75zvXOLFnCndbP5cqtxart6+s3LXvnf/wcNHq2vr5ypKJKFnJOKR/O5hRTkL6ZlmmtPvsaRYeJxeeJfv8/uLAZWKReGpHsa0I3A3ZAEjWIPJXVv6VUOeMGnmvHPaqWvYq/3srVOAvSnYnYJW1rFr6LRHNf5h6inSkTNqZGASWPekMCoKtMBpVkfOvz4NuxbU50Kkjb9BvkGQoyLI1GuUew0b2Qi8RnYtXajhpH7lMiii7yDBfLABRoHExCCaxnWkc4bbnwmHiB/pnGyQSgTiTDCtXNN/gVjonGezNDBmi5h27WrcMrv29fi4XtjzxNMSnd64bc5W0cCsMZU+Jm61gAgu/bz7eYDU7W/nnD5wGjfL7EE+VEzcHOPB8JAnNJtU6FM+SZZl039Fwsx2VzebO83iOPOgNQGb1uScuGsVifyIJIKGmnCsVLvVjHXHYKkZ4RQ0JIrGmFziLm0DDLGgqmMKYZlTA4vvBJGEb6idwjrLMFgoNRQeeObzVjfvcuOiu3aigzcdw8I40TQk40RBwh1oTL7Xjs8kJZoPAWAiGWh1SA9DdzRsP7RyJk2P8gHV5UKI6BgVFNlLkjyRlcnpuFAbSRrSKxIJgUP/pUEBFowPfRrghOt8pYIpXtSvbX/AYjVp3XVITjWKJOuyEHNOA43yR9kMPz2NiqeNPlAYkKRHoPpzTCXWkQQlWHbzlxAG1kVPUQ7/58nCa0+A5bJMIQCKyfsSxTQ0WQEJjxRFXldGSVwSPMcvhEIAHMAYxv60TBt7wJa2bu7kPDjf3Wnt7ex+2d88OJzs64q1YT2z6lbLem0dWJ+sE+vMIpWPFV5JKoPl39X16pPqxti1sjThPLZKp/r8D8V31LM=</latexit>

<latexit sha1_base64="c8e89NgcQwqPwCrKyr0X0eRpKAs=">AAAFbnicfVPbbtNAEHVDA8XcWpB4QQhDFeHQUiVtJZAQUlV4QEgtBXqTssFar9fJpru2tbtOnVj+UJ75AT6BsROXuAlsFOdkds7MmYvdiDOlW62fS7Uby/Wbt1Zum3fu3rv/YHXt4akKY0noCQl5KM9drChnAT3RTHN6HkmKhcvpmXvxIb8/G1KpWBgc61FEuwL3AuYzgjWYnLWlXw3kijTJrPdWJ3FS9no3e2cVYKcE2yVoZ12zgY77VOMfqZ0gHVrjZgYmgXVfilSFvhY4yWxk/evTNBu+PRciaf4N8h2CHBRBSq9x7jVqZmPwGpuNZKGGI/vSYVDEwEKCeWADjHyJSYpoEtlI5wxnMBMOES/UOTlFKhaIM8G0ctLBS8QC6zSbpYExW8Q0G5eTlpmNb4eHdmHPE5clWv1J26yNooFZs5Q+IW60gQgug7z7eYDEGWzmnAFwmtfL7EM+VEw8PcTD0T6PaTat0KO8TLaTZeX/6RQz01ldb221imPNg/YUrBvTc+Ss1STyQhILGmjCsVKddivS3RRLzQinoCNWNMLkAvdoB2CABVXdtBCXWQ2weJYfSvgG2iqss4wUC6VGwgXPfObq+l1uXHTXibX/tpuyIIo1DcgkkR9zC5qT77blMUmJ5iMAmEgGWi3Sx9AhDW8AtHMmTZ/yIdXVQojopsovslckuSKrkpNJoSaSNKCXJBQCB96rFPlYMD7yqI9jrvO18ku8qF+b3pBFatq6q5CcahRK1mMB5pz6GuWPqhl++hoVTxN9pDAgSQ9A9ZeISqxDCUqw7OUvIgysh56jHP7PkwVXngCrZaWFACgm70sY0SDNCkh4qChyezKMo4rgOX4hFAJgH8Yw8adV2sQDtrR9fSfnwen2Vntna/vr7vre/nRfV4wnxgvDNtrGG2PP+GQcGScGqX2uRbVRbbz8u/64/rT+bOJaW5pyHhmVU7f/AEmD1Zs=</latexit>

<latexit sha1_base64="X5uezfEz9hEtVM2JeBg68eGu4hg=">AAAFbnicfVNdb9MwFM3KCiN8bSDxghCBqSJlY2q3SSAhpGnwgJA2BmxjUl0ix3Fad3YS2U6XNsoP5Zk/wE/gJmm3Zi24anp6fc+9537EjThTutX6tVS7sVy/eWvltnnn7r37D1bXHp6qMJaEnpCQh/LMxYpyFtATzTSnZ5GkWLic/nDPP+T3P4ZUKhYGx3oU0a7AvYD5jGANJmdt6XcDuSJNMuu91UmclL3ezd5ZBdiZgu0paGdds4GO+1Tjn6mdIB1a42YGJoF1X4pUhb4WOMlsZP3r0zQbvj0XImleBfkOQQ6KIFOvce41amZj8BqbjWShhiP7wmFQxMBCgnlgA4x8iUmKaBLZSOcMZzATDhEv1Dk5RSoWiDPBtHLSwUvEAus0m6WBMVvENBsXZcvMxrfDQ7uw54mnJVr9sm3WRtHArDmVXhI32kAEl0He/TxA4gw2c84AOM3rZfYhHyomnh7i4WifxzSbVOhRfpUsm/6fzCwzndX11larONY8aE/AujE5R85aTSIvJLGggSYcK9VptyLdTbHUjHAKOmJFI0zOcY92AAZYUNVNC3GZ1QCLZ/mhhG+grcI6y0ixUGokXPDMZ66u3+XGRXedWPtvuykLoljTgJSJ/Jhb0Jx8ty2PSUo0HwHARDLQapE+hg5peAOgnTNp+pQPqa4WQkQ3VX6RvSLJFVmVnJSFmkjSgF6QUAgceK9S5GPB+MijPo65ztfKn+JF/dr0hixSk9ZdhuRUo1CyHgsw59TXKH9UzfDT16h4mugjhQFJegCqv0RUYh1KUIJlL38RYWA99Bzl8H+eLLj0BFgtKy0EQDF5X8KIBmlWQMJDRZHbk2EcVQTP8QuhEAD7MIbSn1ZppQdsafv6Ts6D0+2t9s7W9tfd9b39yb6uGE+MF4ZttI03xp7xyTgyTgxS+1yLaqPaePlP/XH9af1Z6VpbmnAeGZVTt/8CPMPVlw==</latexit>

<latexit sha1_base64="WiKubLZ9MTQF7juk9+RjhtI8mSo=">AAAFanicfVPbbtNAEHVDA8Xc2vKE+mIoEQktVdxWAgkhVQUhXloK9IKUDdZ6vU423bWt3XXqxPKH8sQP8BGM7aTETWCjOMezc2bOXOJGnCndbv9cqt1art++s3LXvHf/wcNHq2vr5yqMJaFnJOSh/O5iRTkL6JlmmtPvkaRYuJxeuJfv8/uLIZWKhcGpHkW0K3AvYD4jWIPJWVv61UCuSJPMemd1Eidlr/azt1YB9qZgdwrsrGs20GmfavwjbSZIh9a4lYFJYN2XIlWhrwVOsiay/vVpmQ2/ORciaf0N8g2CHBVBpl7j3GvUysbgNTYbyUINJ80rh0ERAwsJ5oENMPIlJimiSdREOmc4g5lwiHihzskpUrFAnAmmlZMOXiAWWOfZLA2M2SKm2bgqW2Y2vh4fNwt7nnhaotUv22ZtFQ3MWlPpJXHLBiK4DPLu5wESZ7CdcwbAad0ssw/5UDHx9BgPR4c8ptmkQo/yMpmdZdO3pHgzndXN9k67ONY8sCdg05icE2etJpEXkljQQBOOlerY7Uh3Uyw1I5yChljRCJNL3KMdgAEWVHXTQlhmNcDiWX4o4Rtoq7DOMlIslBoJFzzzeaubd7lx0V0n1v6bbsqCKNY0IGUiP+YWNCbfa8tjkhLNRwAwkQy0WqSPoTsath9aOZOmT/mQ6mohRHRT5RfZK5JckVXJSVmoiSQN6BUJhcCB9zJFPhaMjzzq45jrfKX8KV7Ur21vyCI1ad11SE41CiXrsQBzTn2N8kfVDD99jYqniT5QGJCkR6D6c0Ql1qEEJVj28j8hDKyHnqIc/s+TBdeeAKtlpYUAKCbvSxjRIM0KSHioKHJ7MoyjiuA5fiEUAmAfxlD60yqt9IAttW/u5Dw4392x93Z2v+xvHhxO9nXF2DCeGU3DNl4bB8Yn48Q4M0jtY43X4tpw+Xd9vf6kvlG61pYmnMdG5dSf/wFg2NRD</latexit>

<latexit sha1_base64="/ywPgIAaaM4YYopjrBIDrmGT/kg=">AAAFb3icfVNbb9MwFM7KCiPcNnjgAQkZpoqWjanZJoGEkKbBAxLaGLCbVJfIcZzWnZ0E2+nSWvmhvPMH+Ac4STuateAoyZfj853LdxwvZlSqdvvnUu3Gcv3mrZXb9p279+4/WF17eCqjRGBygiMWiXMPScJoSE4UVYycx4Ig7jFy5l28z/fPhkRIGoXHahSTLke9kAYUI2VM7trSrwb0uE4z8A50UlfTV7vZW1CAnSnYngIn69oNeNwnCn3XzRSqCIxbmTFxpPqCaxkFiqM0a0Lwr6tlN4LmXIi09TfINxPkoAgy9RrnXqNWNjZeY7uRLqzhqHnpUtPEAEBOfWMzGAYCYQ1JGjehyhnuYCYcxH6kcrKGMuGQUU6VdPXgBaQhOM1macaYLWLajctSMrvx9fCwWdjzxNMWQb+UDWwUAmataeklccMxROMyyNXPA6TuYDPnDAyndb3NfiF1MXJ9iIajfZaQbNKiT1iZzcmy6VdafNmp69ju6np7q10sMA+cCVi3JuvIXasJ6Ec44SRUmCEpO047Vl2NhKKYkcyGiSQxwheoRzoGhogT2dVFcRloGIsPgkiYO1SgsM4yNOJSjrhnPPOhy+t7uXHRXidRwZuupmGcKBLiMlGQMGDUyQ838KkgWLGRAQgLamoFuI+MQsr8Aka8mTR9woZEVRvBvKtlUGSvlOTxrEpOy0ZtKEhILnHEOQr9lxoGiFM28kmAEqbycxVM8SK9Nv0hjeVEuquQjCgYCdqjIWKMBArmj6rZvPoKFk8bfiBmQIIcmKo/x0QgFQlTCRK9/E80A+vBZzCH//Ok4ZWngdW2dFGAaSbXJYpJqLMCYhZJAr2eiJK4UvAcvyjUBECBGUPpT6q00sOcUuf6mZwHp9tbzs7W9pfd9b39yXldsZ5Yz62m5VivrT3ro3VknVi49qn2ozau6eXf9cf1p3VQutaWJpxHVmXVW38AIj/VrA==</latexit>
x1 <latexit sha1_base64="nB8pK9PGEwV+LmMCOwKc8YP2Au8=">AAADenicfVJLb9NAEN4mPEp4tXDkYhpV4hklbSU4VsCBC6JIpKmUtaLxeuyssg9rdx0SWf4RXOGX8V84sHYMqpuKkbz+PA/PN99OlAlu3XD4a6fTvXHz1u3dO7279+4/eLi3/+jc6twwHDMttLmIwKLgCseOO4EXmUGQkcBJtHhfxSdLNJZr9dWtMwwlpIonnIHzrslqVvDXx+Vsrz8cDGsLtsGoAX3S2NlsvxPSWLNconJMgLXT0TBzYQHGcSaw7NHcYgZsASlOPVQg0YZFzbcMDr0nDhJt/KNcUHsvVxQgrV3LyGdKcHN7NVY5r4tNc5e8DQuustyhYptGSS4Cp4Nq+CDmBpkTaw+AGe65BmwOBpjzEvUOL7eZo1iiaw/CZFjYpO7eohRJ/21Q4TempQQVvyhoApKLdYwJ5MKVBbXJX3ydNK/iJc9so9JqI1OPCnRUG55yBUJg4mh1tN3+NXe0Pnv0A/q7MPjJE/ycoQGnjWcCJpWwKv3dpPQpreD/Mrn6l+lhe6yiJuCHqSTQGaqirCET2iKNUqPzrEV4q74m6n8AiVd8k4/tsk2GX8jR1fXbBudHg9Hx4OjLSf/0XbOau+QJOSDPyIi8IafkIzkjY8LIgnwnP8jPzu/uQfd59+UmtbPT1DwmLeue/AGugCyM</latexit>
xi 3 <latexit sha1_base64="Lw5rWS0utPfv8sAZFj6BQfU2/QM=">AAADenicfVJLb9NAEN4mPEp4teXIxTSqxKtREiqVYwUcuCCKRJpKWSsar8fOKvuwdtchkeUfwRV+Gf+FA2snoLqpGMnrz/PwfPPtRJng1vX7v3Za7Vu379zdvde5/+Dho8d7+wcXVueG4Yhpoc1lBBYFVzhy3Am8zAyCjASOo/n7Kj5eoLFcq69ulWEoIVU84Qycd42X04IfD8vpXrff69cWbIPBBnTJxs6n+62QxprlEpVjAqydDPqZCwswjjOBZYfmFjNgc0hx4qECiTYsar5lcOQ9cZBo4x/lgtp7taIAae1KRj5TgpvZ67HKeVNskrvkbVhwleUOFVs3SnIROB1UwwcxN8icWHkAzHDPNWAzMMCcl6hzdLXNDMUCXXMQJsPCJnX3BqVI+m+DCr8xLSWo+GVBE5BcrGJMIBeuLKhN/uKbpHkdL3hmNyot1zJ1qEBHteEpVyAEJo5WR9PtXzNH67NDP6C/C4OfPMHPGRpw2ngmYFIJy9LfTUqf0Qr+L5Orf5keNscqagJ+mEoCnaEqyhoyoS3SKDU6zxqEt+prov4HkHjF1/nYLFtn+IUcXF+/bXAx7A3e9IZfTrpn7zaruUuekkPynAzIKTkjH8k5GRFG5uQ7+UF+tn63D9sv2q/Wqa2dTc0T0rD2yR+rbyyL</latexit>
xi 2 <latexit sha1_base64="0VBkd1Rw+AkOCuCd8jR6pjBnSJo=">AAADenicfVJLb9NAEN4mPEp4tXDkYhpV4tUoLpXgWAEHLogikaZS1orG67Gzyj6s3XVIZPlHcIVfxn/hwNoJqG4qRvL68zw833w7cS64dcPhr51O98bNW7d37/Tu3rv/4OHe/qNzqwvDcMS00OYiBouCKxw57gRe5AZBxgLH8fx9HR8v0Fiu1Ve3yjGSkCmecgbOu8bLacmPwmq61x8Oho0F2yDcgD7Z2Nl0vxPRRLNConJMgLWTcJi7qATjOBNY9WhhMQc2hwwnHiqQaKOy4VsFh96TBKk2/lEuaLyXK0qQ1q5k7DMluJm9Gqud18UmhUvfRiVXeeFQsXWjtBCB00E9fJBwg8yJlQfADPdcAzYDA8x5iXqHl9vMUCzQtQdhMipt2nRvUYql/zao8BvTUoJKXpQ0BcnFKsEUCuGqktr0L75OmlfJgud2o9JyLVOPCnRUG55xBUJg6mh9tN3+NXO0OXv0A/q7MPjJE/ycowGnjWcCJpOwrPzdZPQpreH/Mrn6l+lhe6yyIeCHqSXQOaqyaiAT2iKNM6OLvEV4q74h6n8AqVd8nY/tsnWGX8jw6vptg/PjQfh6cPzlpH/6brOau+QJOSDPSEjekFPykZyREWFkTr6TH+Rn53f3oPu8+3Kd2tnZ1DwmLeue/AGoXiyK</latexit>
xi 1 <latexit sha1_base64="x4eMZtKrr1x8Ug2+LL0eW7kjARo=">AAAFD3icfVNdb9MwFM1KgRG+NnjkxTAVWhij3SaBhJAm4IGXjQH7kuoSOY7TurOTyHa2tFF+BI/8Et4Qr/wE/g3XaQvtWnDV9PT6nOvrc2/8RHBtms1fS5VL1ctXri5fc6/fuHnr9srqnSMdp4qyQxqLWJ34RDPBI3ZouBHsJFGMSF+wY//0jd0/PmNK8zg6MIOEdSTpRjzklBgIeatLX2vYl3lWoFeonXk5f7pdvEQl2JqAzQloFR23hg96zJDPeT3DJkbDRgEhSUxPyVzHoZEkK+oY/evTcGthfS5F1vib5BMk2S2TTFhDyxo0iiGwhm4tW1jDfv3c43CJPsKSBxADjENFaI5ZltSxsQqvP5UO0yA2VpxjnUosuORGe3n/EeYROiqmZRAsFind2vnIMrf2cW+vXsbtwZMrot7INvRkYmBjUrxrA6ADRt+ab/WZ11+3kn5J9FbWmhvNcqF50BqDNWe89r3VSgcHMU0liwwVROt2q5mYTk6U4VSwwsWpZgmhp6TL2gAjIpnu5OUQFagGkQCFsYJvZFAZnVbkRGo9kD4wbaf0xT0bXLTXTk34opPzKEkNi+jooDAVCLy0E4kCrhg1YgCAUMWhVkR7BDpnYG7BrqljekycMTN7ESo7uQ7L02dK8iX8Vyxi5zSWkkTB4xyHRHIxCFhIUmFs38MJXmTNenDGEz12KRvZ5GLBDI4V7/KICMFCg+1jNgw/PYPLp4vfMuiFYrtQ4PuEKWJiBZUQ1bVvCvSmi+9jC//H5NEfJsDZa+VlAXAZa0GcsCgvSkhFrBn2uypOk5mC5/RloZCAhOD4iM9mZSMGDGTr4vjNg6PNjdbWxuaH7bWd1+PRXHbuOQ+cutNynjs7zjtn3zl0aMWpPKw8qzSrX6rfqt+rP0bUytJYc9eZWdWfvwFZxrLC</latexit>
xi 32
 Problems learning RNNs
■ In theory, should be able to propagate information over arbitrarily long contexts.
■ In practice, RNNs suffer from vanishing gradients that decay to 0, or
exploding gradients that increase towards in nity.

■ Exploding gradients mostly resolved by gradient clipping: thresholding

gradient values, or rescaling them. Threshold/scale is a hyperparameter.

■ Vanishing gradients mostly resolved by adding gating to the RNN composition

function.

Figure: Goodfellow, Bengio and Courville. Deep Learning. MIT Press, 2016. 33
fi
 Long short term memory: LSTMs
■ Adds a memory cell cm and three gates: input i, output o, and forget f
(h!f ) (x!f )
fm+1 = (⇥
<latexit sha1_base64="T4Vawq6KjhA/qwwu+BR5v9i4W3o=">AAAF3nicfVNNj9Q2GA5Dp0D6xceRi9vVqDNduposSCAhJAQceoDttrAL0noIjuNMPNhJZDu7mbV85Vb12t/CP+mNK/0VfZ3MwGZ3W48mefL6fd6P57WTSnBtptO/LwwufjH88tLlK+FXX3/z7XdXr13f12WtKNujpSjVq4RoJnjB9gw3gr2qFCMyEexl8vax3395yJTmZfHCLCs2k2Re8IxTYsAUXxs8H+FE2sahB+igiS3/+Y67j1pwew221yBys3CEX+TMkNd23GBTouOJA5MkJlfS6jIzkjRujNF//SbhKBufCdFMPgd5DkGetUHWXsfeazlxx+B1HI6ac2vYHR/FHJpYICx5CjbAOFOEWsyaaoyNZ8SLE+EwTUvjyRbrWmLBJTc6tosfMS/QvjtJA6M7jxmOjjrJwtHvOzvj1u4Tr1tEeScb2mwFdJN16R1xMwIiuCy8+j5AEy9uec4COJPTbeat1O3I7Q45XD4SNXOrFlMmumyRc+uvpv0CveIIeEyIeBvecyIlpPctQ54HkXv9VMcLyB+FWWwllOSF03wuyecJ5D5/NnF5LKGVfllgblbETZTE2SQM46sb061pu9BZEK3ARrBau3AMFU5LWktWGCqI1gfRtDIzS5ThVDAX4lqzitC3ZM4OABZEMj2zrRYOjcCSoqxU8C8Maq0nGZZIrZcyAU9/xvTpPW88b++gNtm9meVFVRtW0C5RVgsEXfu7hFKuGDViCYBQxaFWRHMCAzFw40DjE2lyJg6Z6TdC5czqrM3eKymRrk9uukZDrFjBjmgJEyzSnyzOiORimbKM1ML4Y5yt8Xl63UoPeaVX0n0KKZjBpeJzXhAhWGawf/TN8MoNbp8hfsJgQIo9g6p/rZgiplRQCVFzf/FhYHP8Pfbw/zx58ckTYL8t2xYAzXhdyooV1rWQilIznMxVWVe9gs/w20IhAMlgDJ0/69M6Dzil0ekzeRbsb29Ft7e2f7uz8fDR6rxeDm4GPwTjIAruBg+DX4LdYC+gg/eDD4OPg3+Gb4bvhn8M/+xcBxdWnBtBbw3/+hdGN/ut</latexit>
hm + ⇥ xm+1 + bf )
■ Gates are functions of the input and im+1 = (⇥ (h!i)
hm + ⇥ (x!i)
xm+1 + bi )
previous hidden state, in range [0, 1], <latexit sha1_base64="5Hqv0EEyN8MdyCgKQrUwzWbjGlE=">AAAGJ3icfVTdjtQ2FA47TKHpD1AuuQmsRp10YTWZRSoSQkKUi17AdtuyC9J6iBzHmXiwk8h2djNr+U14AZ6Gu6q97Jv0OJmBze4Ujyb5cny+c75zjpOk4kzpyeSfK1uDq8Ovrl3/2v/m2+++v3Hz1g9HqqwloYek5KV8k2BFOSvooWaa0zeVpFgknL5O3v3i9l+fUKlYWbzSy4rOBJ4XLGMEazDFt7bej1AiTGODJ8FxExv24KF9HLRgbw2maxDZmT9Cr3Kq8VszbpAug7PQgklgnUthVJlpgRs7RsH//UJ/lI0vhWjCz0H+hCAv2yBrrzPntQztGXid+aNmo4aD8WnMoIhFgARLwQYYZRITg2hTjZF2jHhxLhwiaakd2SBVC8SZYFrFZvEjYkVwZM/TwGg3Mf3Radcyf/TH/v64tbvE6xKDvGtbsNM20IZr6R1xJwIiuCxc912AJl7cd5wFcMKLZeZtq9uRm318snzGa2pXJaaUd9kia9dPTfsE/Yoj4FHO4ync51gISO9KhjxPIvv2hYoXkB+cstgI0OQ6p9hc4M8jyJ2ALLR5LKCWvi4wNyviTpDEWeizL8dhm+OwfhwW+n58c3uyO2lXcBlEK7DtrdYBnGeJ0pLUghaacKzUcTSp9MxgqRnh1PqoVrTC5B2e02OABRZUzUzbVBuMwJIGWSnhX+igtZ5nGCyUWooEPN1hVRf3nHHT3nGts0czw4qq1rQgXaKs5gFU7V7KIGWSEs2XADCRDLQGJMcwWQ2vLgzrXJqc8hOq+4UQMTMqa7P3JCXC9slNV6iPJC3oKSnhKBTpTwZlWDC+TGmGa67d+5Ct8aZ+3U9PWKVWrfsUklONSsnmrMCc00wjd+mb4ZZr1F599JzCgCR9Cap/q6jEupSgBMu5+4LAwOboLnLwS56s+OQJsF+WaQVAMa4vZUULY1tIeKkoSuayrKue4Ev8VigEwBmMofOnfVrnAac0ungmL4Oj6W60tzv9/eH202er83rdu+Pd88Ze5P3sPfV+9Q68Q48MBoNwMB3sDT8MPw7/Gv7duW5dWXFue701/Pc/hNISuw==</latexit>

applied element-wise w/ Hadamard <latexit sha1_base64="6AZWwg4B2+1h+vg6z2Jr3N4+m/4=">AAAHInicfVRdb9s2FFWTzGu1jzbrY1/UZcbspQ2stEAHDAWKbg97WLNsa9ICoSvQFGXRJUVBpBI5BH/H/kB/zd6GPQ3Ybxl2KcmJZXuRYenq6p5zz7mkNMk5U3o0+vvW1vbOR72Pb9/xP/n0s8/v3tv94lTJsiD0hEgui7cTrChnGT3RTHP6Ni8oFhNO30zef++evzmnhWIye63nOR0LPM1YwgjWkIp2t3/vo4kwlQ2eB2dVZNjjp/a7oA6eLILDRRDasd9Hr1Oq8TszqJCWweXQQkpgnRbCKJlogSs7QMH//YZ+PxmsUVTDa5LfgORVTbKounRV86G9hKpLv19t1HA8uIgYmJgFSLAYchCjpMDEIFrlA6QdIpot0SESS+3ABqlSIM4E0yoys68Ry4JTuwyDpN2E9PsXzcj8/q9HR4M67xovLAZpM7Zgvx6gHS6kN8D9EIBQMnPTdwRVNHvkMDPADFdtpvWo6yU3R/h8/pKX1LYWY8qbbqG1i7uqvoN5RSHgKOfRIVynWAho7yxDn+ehffeTimbQH4qSyAjQ5Can2FTg6yVInYBkaNNIgJeuLkhXLXA/mEQJCGc3E7HNRKxLxNwENOMxNcReE2qcpSt8ZI3vok23fMBErhhal0i6RSQ1ji2lVlr6fXmzGbnZjOyakcN6pVuilrLZR40jslBai/D96N7e6GBUH8F6ELbBntcex9HuVoFiSUpBM004VuosHOV6bHChGeHU+qhUNMfkPZ7SMwgzLKgam3o/2aAPmThIZAH/DMbisssIg4VSczGBSveeqtVnLrnp2Vmpk2/HhmV5qWlGmkZJyQMYkfseBTErKNF8DgEmBQOtAUkxbGoNXy3YAEttUsrPqe4aIWJsVFJ370iaCNsFV41RHxU0oxdEwluQxd8YlGDB+DymCS65dp+CZBFvmtej+Jzlqh3dFSWnGsmCTVmGOaeJRu7UTcMl1ag+++gHCgtU0Feg+uecFljLApTgYuo+nrBgU/QQufCmSpZdVULYtWVqAWDGzUXmNDO2DgmXiqLJtJBl3hG8hq+FAgFOYBmaetqFNRWwS8PVPbkenB4ehE8ODn95uvfiZbtfb3sPvC+9gRd6z7wX3o/esXfike1/d77aebxz0PvQ+6P3Z++vpnTrVou573WO3j//AdtZaE4=</latexit>

om+1 = (⇥ <latexit sha1_base64="O+S80G7mzF8HKqPEaMRgbt5tbaM=">AAAG9HicfVTPb9s2FFbjzuu0X8163EVbYMxe2sBKC6zAUKDodthhzbKtSQuErkBRlEWXFAWRcuQQ/E92G3bd37Lr/pFdt0dZTizbiwxLT4/v+9773qMYF5wpPR7/fWevd/e9/vv3PvA//OjjTz69v//ZuZJVSegZkVyWb2KsKGc5PdNMc/qmKCkWMaev43ffufXXc1oqJvNXelHQicDTnKWMYA2uaL/HBigWprbBs+Cijgx79MR+GzTG45VxvDJCO/EH6FVGNX5rhjXSMrgaWXAJrLNSGCVTLXBthyj4v9/IH6TDLYp6dEPyK5C8bEhWUVcuajGyVxB15Q/qnTWcDi8jBiJmARIsAR/YKC0xMYjWxRBph4hma3SIJFI7sEGqEogzwbSKzOwrxPLg3K7DwGl3If3B5bJl/uCXk5Nh43eJVxKDbNm24LBpoB2tSl8CD0MAQsjMdd8R1NHsocPMADPalJk1rW5Gbk7wfPGCV9S2EhPKl9lCa1dvdfMG/YpCwFHOo2N4TrEQkN5JhjzPQvv2RxXNID8EpZERUJPrnGJTgW9GkLkC0pHNIgFaunWBu26Bh0EcpVA4u52I7SZiXSLmOqAZT6gh9oZQ4zzb4CNbfJetu+UDJnLN0KpE0g2RNDi25tpI6cvbtcjdWmRXixz5fnT/YHw0bq5g2whb48Brr9Nof69EiSSVoLkmHCt1EY4LPTG41Ixwan1UKVpg8g5P6QWYORZUTUyzQ2wwAE8SpLKEfw5CnXcdYbBQaiFiiHRfntpcc85daxeVTp9ODMuLStOcLBOlFQ9AtTthgoSVlGi+AAOTkkGtAckwbFMN5xCMdC1NRvmc6q4QIiZGpU32TkmxsF1wvRTqo5Lm9JJI2Nd58rVBKRaMLxKa4opr93GnK3tXvx4mc1aotnXXlJxqJEs2ZTnmnKYauVvXDY9Mo+buo+8pDKikL6HqnwpaYi1LqASXU3ccwsCm6AvkzNsiWX4dCWZXlmkKADGuL7KgubGNSbhUFMXTUlZFp+AtfFMoEOAUxrCMp13YMgJ2abi5J7eN8+Oj8PHR8c9PDp6/aPfrPe9z70tv6IXeN95z7wfv1DvzSO+v3j+9f+96/Xn/t/7v/T+WoXt3WswDr3P1//wPZ3JZew==</latexit>
(h!o)
hm + ⇥ (x!o)
xm+1 + bo )

■ Memory cell is updated as a gated sum

of its previous value, and an update c̃ , cm+1 = fm+1 cm + im+1 c̃m+1
which is computed from input and
<latexit sha1_base64="6gf1eKL/XMA04gWlX9FfFUmNUIE=">AAAGq3icfVTfb9s2EFbjzeu0X836uBdtgTF7aQMrLbACQ4Gi28Me1ixbm7RY6AoURVl0SUkgqUQOwef9NXvd/pb9NztKcmLZXmVYOh3v++67O4pxyZnS0+m/d/YGH3w4/Ojux/4nn372+Rf39r88V0UlCT0jBS/kmxgryllOzzTTnL4pJcUi5vR1/O5Ht/76kkrFivyVXpZ0JvA8ZykjWIMr2h8EIxQLU9vgaXBRR4Y9fGx/CBrj0co4Xhmhnfkj9CqjGr814xrpIrieWHAJrDMpjCpSLXBtxyj4v9/EH6XjLYp6ckvyEkheNCSrqGsXtZzYa4i69kf1Tg2n46uIQRGLAAmWgA9slEpMDKJ1OUbaIaLFGh0iSaEd2CBVCcSZYFpFZvEtYnlwbtdh4LS7kP7oqm2ZP/r95GTc+F3iVYlB1rYtOGwaaCcr6S3wMAQghCxc9x1BHS0eOMwCMJPNMrOm1c3IzQm+XD7nFbVdiQnlbbbQ2tVb3bxBv6IQcJTz6BiecywEpHclQ56noX37i4oWkB+C0sgI0OQ6p9hc4NsRZE5AOrFZJKCWvi5w1x3wMIijFISz9xOx3USsT8RcBzTjCTXE3hJqnGcbfGSL76pzd3wTn9wQdEWiws2QNDC25trI6PvRvYPp0bS5gm0j7IwDr7tOo/09iZKCVILmmnCs1EU4LfXMYKkZ4dT6qFK0xOQdntMLMHMsqJqZZrA2GIEnCdJCwj8Hgc67jjBYKLUUMUS6D0ZtrjnnrrWLSqdPZoblZaVpTtpEacUDaJQ7GIKESUo0X4KBiWSgNSAZht2l4fiASaylySi/pLpfCBEzo9Ime09SLGwfXLeF+kjSnF6RArZjnnxnUIoF48uEprji2n2T6cre1a8HySUrVde6G0pONSokm7Mcc05Tjdyt74ZHplFz99FPFAYk6QtQ/WtJJdaFBCVYzt0pBgObo6+RM98XyfKbSDD7ZZlGABTj+lKUNDe2MQkvFEXxXBZV2RO8hW+EAgFOYQxtPO3D2gjYpeHmntw2zo+PwkdHx789Pnj2vNuvd72vvG+8sRd633vPvJ+9U+/MI4M/B38N/h78M3w4fDn8Y4ja0L07Hea+17uG9D9KGz9n</latexit>

(w !c)
previous hidden, and squashed. c̃m+1 = tanh(⇥ <latexit sha1_base64="mPogIB68lNAu0jq2CMnk2uEHZIk=">AAAGcXicfVTdbts2FFbj1Wu1n7bbVdEbroExa+kCKy2wAUWBotvFLpYs25q0QOgKFEVZdElJIKlEjsYH3QvsBfYCO5TkJoq9yrB0dHi+73znHFJxKbg2s9nft3ZGn9wef3rnrv/Z5198ee/+g69OdVEpyk5oIQr1NiaaCZ6zE8ONYG9LxYiMBXsTv//Jrb85Z0rzIn9tViWbS7LIecopMeCKHuz8M8GxbGqLXqCzOmr498/sc9QaT9fGwdoI7dyf4NcZM+RdM62xKdBlYMElicmUbHSRGklqO8Xo/36BP0mnGxR1cEXyJ5ActiTrqEsXtQrsJURd+pN6q4bj6UXEoYglwpIn4AMbp4rQBrO6nGLjENHyGh2mSWEcuMG6klhwyY2OmuW3mOfo1F6HgdNuQ/qTi65l/uSPo6Np63eJ1yWirGsb2msbaIO19A64FwIQQpau+46gjpZPHGYJmOBmmVnb6nbkzRE5X70SFbN9iQkTXbbQ2vVb3b5Bv6IQcEyI6ACeCyIlpHclQ54XoX33q46WkB+C0qiRoMl1TvOFJFcjyJyANLBZJKGWoS5w1z1wD8VRCsL5x4n4diI+JOKBjw0XCWuoveIzJM9u0NENuove3dMFvh/d353tz9oLbRphb+x6/XUMJ0PhpKCVZLmhgmh9Fs5KM2+IMpwKZn1caVYS+p4s2BmYOZFMz5t2PBZNwJOgtFDwzw1qvdcRDZFar2QMkW7b65trzrlt7awy6Y/zhudlZVhOu0RpJRDU6443Srhi1IgVGIQqDloRzQjsEQMfARj7tTQZE+fMDAuhct7otM0+kBRLOwTXXaE+VixnF7SATZUn3zU4JZKLVcJSUgnjTla6trf160lyzkvdt+4DpWAGF4oveE6EYKnB7jZ0wyMzuL37+GcGA1LsEFT/VjJFTKFACVEL9y2CgS3wN9iZH4vk+YdIMIdlNa0AKMb1pShZ3tjWpKLQDMcLVVTlQPAGvhUKBCSFMXTxbAjrImCXhjf35KZxerAfPt0/+P3Z7stX/X694z3yHntTL/R+8F56v3jH3olHR4cjPfprZG//O344RuPHXejOrR7ztTe4xnv/AZrHKxw=</latexit>
(h!c)
hm + ⇥ xm+1 )

■ Next hidden state is a combination of

output and memory cell. hm+1 = om+1 · tanh(cm+1 ) <latexit sha1_base64="O1W6ddent1LYQqiPFOe0L6iPxcY=">AAAHG3icfVTNbttGEGZsV03Yvzg5BgXYGmqkOjVEJ0ALFAGCtIceGtdtYyeAVyFWy6W4yi6X4C5tyot9h7xAnqa3otce+iS9dpakbFFSTUHkaDjfN/PNjHaSc6b0aPTPra3tnQ96H96+43/08SeffnZ3996pkmVB6AmRXBavJ1hRzjJ6opnm9HVeUCwmnL6avP3BvX91TgvFZPZSz3M6FniasYQRrMEV7W6/66OJMJUNngZnVWTYN0/s90FtPF4YhwsjtGO/j16mVOM3ZlAhLYPLoQWXwDothFEy0QJXdoCC//sM/X4yWKOohtckvwPJi5pkEXXpouZDewlRl36/2ljD8eAiYiBiFiDBYvCBjZICE4NolQ+QdohotkSHSCy1AxukSoE4E0yryMweIpYFp3YZBk67Cen3L5qW+f3fjo4Gtd8lXkgM0qZtwX7dQDtclN4A90MAQsjMdd8RVNHskcPMADNclZnWra5Hbo7w+fw5L6ltJcaUN9lCaxe/qvoX9CsKAUc5jw7hOcVCQHonGfI8De2bn1U0g/wQlERGQE2uc4pNBb4eQeoKSIY2jQRo6dYF7qoF7geTKIHC2c1EbDMR6xIx1wHNeEwNsdeEGmfpCh9Z47to3S0fMJErhlYlkm6IpMaxJddKSr8vbxYjN4uRXTFy6KdXPC1js0aNILIo1I/u7o0ORvUVrBtha+x57XUc7W4VKJakFDTThGOlzsJRrscGF5oRTq2PSkVzTN7iKT0DM8OCqrGpF8kGffDEQSIL+GbQD+ddRhgslJqLCUS6P6hafeecm96dlTr5bmxYlpeaZqRJlJQ8gN64gyiIWUGJ5nMwMCkY1BqQFMM2aziuYPJLaVLKz6nuCiFibFRSZ++UNBG2C64aoT4qaEYviIT1z+KvDUqwYHwe0wSXXLszIFnYm/r1KD5nuWpbd0XJqUayYFOWYc5popG7dd3wSDWq7z76kcKACvoCqv4lpwXWsoBKcDF1pyYMbIq+QM68KZJlV5FgdmWZugAQ4/oic5oZW5uES0XRZFrIMu8UvIavCwUCnMAYmnjahTURsKXh6k6uG6eHB+Hjg8Nfn+w9e97u623vgfelN/BC71vvmfeTd+ydeGT7353Pd77aedh73/uj92fvryZ061aLue91rt7f/wE9LmXH</latexit>

How much context do you need? Is it worth the additional computation? 34

 Does it parallelize?
Stacking RNNs
■ Just as we can add many hidden layers to feed-forward networks, can also add
many RNN layers: stacked RNNs wi+1
<latexit sha1_base64="fXUrGibuPWDoPErC1D8zjEiH0LQ=">AAAFEXicfVNdb9MwFM1CgRG+NnjkxTBFpDCmppsEEkKagAdeNgbsS6pL5DhO685OItvZ0kb5FTzyS3hDvPIL+DfYaQvtWnDV9PT6nOvrc2/CjFGpWq1fK/aVxtVr11dvODdv3b5zd2393rFMc4HJEU5ZKk5DJAmjCTlSVDFymgmCeMjISXj2xuyfnBMhaZocqmFGuhz1EhpTjJQOBesrX10Y8rKowCvQKYKSPtupXoIabE9Bewr8quu48LBPFPpcegVUKRg1Kx3iSPUFL2UaK46KyoPgX5+m48beQoqi+TfJJ51kr04yZY0Ma9isRpo1ctxiaQ0H3kVA9SUGAHIa6ZjGMBYIl5AUmQeVUQSDmXQQR6ky4hLKnENGOVUyKAePIU3AcTUr08FqmdJxL8aWOe7H/X2vjpuDp1cE/bFt4OnUwOa0eMcon/paqTkDY7/JUASDTSMa1NRgbaO11aoXWAT+BGxYk3UQrNtdGKU45yRRmCEpO34rU90SCUUxI5UDc0kyhM9Qj3Q0TBAnslvWY1QBV0ciEKdCfxMF6uisokRcyiEPNdP0Sl7eM8Fle51cxS+6JU2yXJEEjw+Kcwa0m2YmQUQFwYoNNUBYUF0rwH2ke6f05GrDZo7pE3ZO1PxFMO+WMq5Pnysp5Pq/IAm5wCnnKImelDBGnLJhRGKUM2U6H0/xMms2o3OayYlLxdgmBzKiYCpojyaIMRIraB7zYf3TV7B+OvAt0b0QZE8X+D4jAqlU6EqQ6Jl3RfemBx9CA//HpMkfpobz1yrrAvRljAVpRpKyqiFmqSQw7Ik0z+YKXtDXheoEKNaOj/lkXjZm6IH0L4/fIjhub/nbW+0POxu7ryejuWo9sB5ZnuVbz61d6511YB1Z2LZtz/btduNL41vje+PHmGqvTDT3rbnV+Pkbys+zMQ==</latexit>

softmax(
softmax(
<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>

<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>
))

… layer L

…
…

… layer 2

… layer 1

… breathless runners approached the

wi
<latexit sha1_base64="tqdvN1Co9lflZKGA9FYLbBE3SQc=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1OC+X1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4Dx3yZxA==</latexit>
3 wi
<latexit sha1_base64="cCatZM/QvTiGjI99S0wQVTzzBoU=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1O6+P1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DxGqZww==</latexit>
2 wi
<latexit sha1_base64="CH1jSvOpKNyBxULr16kKG8gaFmQ=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1Oy+L1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DwViZwg==</latexit>
1 wi
<latexit sha1_base64="5EwDIiByx9xSWpoPXUBPQBRnzvo=">AAAFD3icfVNdb9MwFM1KgRG+NnjkxTAVWhij3SaBhJAm4IGXjQH7kuoSOY7TurOTyHa2tFF+BI/8Et4Qr/wE/g3XaQvtWnDV9PT6nOvrc2/8RHBtms1fS5VL1ctXri5fc6/fuHnr9srqnSMdp4qyQxqLWJ34RDPBI3ZouBHsJFGMSF+wY//0jd0/PmNK8zg6MIOEdSTpRjzklBgIeatLX2vYl3lWoFeonXk5f7pdvEQl2JqAzQloFR23hg96zJDPeT3DJkbDRgEhSUxPyVzHoZEkK+oY/evTcGthfS5F1vib5BMk2S2TTFhDyxo0iiGwhm4tW1jDfv3c43CJPsKSBxADjENFaI5ZltSxsQqvP5UO0yA2VpxjnUosuORGe3n/EeYROiqmZRAsFind2vnIMrf2cW+vXsbtwZMrot7INvRkYmBjUrxrlaADRt+ab/WZ11+3kn5J9FbWmhvNcqF50BqDNWe89r3VSgcHMU0liwwVROt2q5mYTk6U4VSwwsWpZgmhp6TL2gAjIpnu5OUQFagGkQCFsYJvZFAZnVbkRGo9kD4wbaf0xT0bXLTXTk34opPzKEkNi+jooDAVCLy0E4kCrhg1YgCAUMWhVkR7BDpnYG7BrqljekycMTN7ESo7uQ7L02dK8iX8Vyxi5zSWkkTB4xyHRHIxCFhIUmFs38MJXmTNenDGEz12KRvZ5GLBDI4V7/KICMFCg+1jNgw/PYPLp4vfMuiFYrtQ4PuEKWJiBZUQ1bVvCvSmi+9jC//H5NEfJsDZa+VlAXAZa0GcsCgvSkhFrBn2uypOk5mC5/RloZCAhOD4iM9mZSMGDGTr4vjNg6PNjdbWxuaH7bWd1+PRXHbuOQ+cutNynjs7zjtn3zl0aMWpPKw8qzSrX6rfqt+rP0bUytJYc9eZWdWfvwFWlrLB</latexit>
35
 Bidirectional RNNs
■ In language it’s also often useful to model past and future context
■ Can also run an RNN in the backward direction (reverse reading order)
■ Combining both directions works best!

wi+1
<latexit sha1_base64="fXUrGibuPWDoPErC1D8zjEiH0LQ=">AAAFEXicfVNdb9MwFM1CgRG+NnjkxTBFpDCmppsEEkKagAdeNgbsS6pL5DhO685OItvZ0kb5FTzyS3hDvPIL+DfYaQvtWnDV9PT6nOvrc2/CjFGpWq1fK/aVxtVr11dvODdv3b5zd2393rFMc4HJEU5ZKk5DJAmjCTlSVDFymgmCeMjISXj2xuyfnBMhaZocqmFGuhz1EhpTjJQOBesrX10Y8rKowCvQKYKSPtupXoIabE9Bewr8quu48LBPFPpcegVUKRg1Kx3iSPUFL2UaK46KyoPgX5+m48beQoqi+TfJJ51kr04yZY0Ma9isRpo1ctxiaQ0H3kVA9SUGAHIa6ZjGMBYIl5AUmQeVUQSDmXQQR6ky4hLKnENGOVUyKAePIU3AcTUr08FqmdJxL8aWOe7H/X2vjpuDp1cE/bFt4OnUwOa0eMcon/paqTkDY7/JUASDTSMa1NRgbaO11aoXWAT+BGxYk3UQrNtdGKU45yRRmCEpO34rU90SCUUxI5UDc0kyhM9Qj3Q0TBAnslvWY1QBV0ciEKdCfxMF6uisokRcyiEPNdP0Sl7eM8Fle51cxS+6JU2yXJEEjw+Kcwa0m2YmQUQFwYoNNUBYUF0rwH2ke6f05GrDZo7pE3ZO1PxFMO+WMq5Pnysp5Pq/IAm5wCnnKImelDBGnLJhRGKUM2U6H0/xMms2o3OayYlLxdgmBzKiYCpojyaIMRIraB7zYf3TV7B+OvAt0b0QZE8X+D4jAqlU6EqQ6Jl3RfemBx9CA//HpMkfpobz1yrrAvRljAVpRpKyqiFmqSQw7Ik0z+YKXtDXheoEKNaOj/lkXjZm6IH0L4/fIjhub/nbW+0POxu7ryejuWo9sB5ZnuVbz61d6511YB1Z2LZtz/btduNL41vje+PHmGqvTDT3rbnV+Pkbys+zMQ==</latexit>

softmax(
softmax(
<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>

<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>
))

… breathless runners approached the

wi
<latexit sha1_base64="5EwDIiByx9xSWpoPXUBPQBRnzvo=">AAAFD3icfVNdb9MwFM1KgRG+NnjkxTAVWhij3SaBhJAm4IGXjQH7kuoSOY7TurOTyHa2tFF+BI/8Et4Qr/wE/g3XaQvtWnDV9PT6nOvrc2/8RHBtms1fS5VL1ctXri5fc6/fuHnr9srqnSMdp4qyQxqLWJ34RDPBI3ZouBHsJFGMSF+wY//0jd0/PmNK8zg6MIOEdSTpRjzklBgIeatLX2vYl3lWoFeonXk5f7pdvEQl2JqAzQloFR23hg96zJDPeT3DJkbDRgEhSUxPyVzHoZEkK+oY/evTcGthfS5F1vib5BMk2S2TTFhDyxo0iiGwhm4tW1jDfv3c43CJPsKSBxADjENFaI5ZltSxsQqvP5UO0yA2VpxjnUosuORGe3n/EeYROiqmZRAsFind2vnIMrf2cW+vXsbtwZMrot7INvRkYmBjUrxrlaADRt+ab/WZ11+3kn5J9FbWmhvNcqF50BqDNWe89r3VSgcHMU0liwwVROt2q5mYTk6U4VSwwsWpZgmhp6TL2gAjIpnu5OUQFagGkQCFsYJvZFAZnVbkRGo9kD4wbaf0xT0bXLTXTk34opPzKEkNi+jooDAVCLy0E4kCrhg1YgCAUMWhVkR7BDpnYG7BrqljekycMTN7ESo7uQ7L02dK8iX8Vyxi5zSWkkTB4xyHRHIxCFhIUmFs38MJXmTNenDGEz12KRvZ5GLBDI4V7/KICMFCg+1jNgw/PYPLp4vfMuiFYrtQ4PuEKWJiBZUQ1bVvCvSmi+9jC//H5NEfJsDZa+VlAXAZa0GcsCgvSkhFrBn2uypOk5mC5/RloZCAhOD4iM9mZSMGDGTr4vjNg6PNjdbWxuaH7bWd1+PRXHbuOQ+cutNynjs7zjtn3zl0aMWpPKw8qzSrX6rfqt+rP0bUytJYc9eZWdWfvwFWlrLB</latexit>
wi
<latexit sha1_base64="CH1jSvOpKNyBxULr16kKG8gaFmQ=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1Oy+L1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DwViZwg==</latexit>
1 wi
<latexit sha1_base64="cCatZM/QvTiGjI99S0wQVTzzBoU=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1O6+P1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DxGqZww==</latexit>
2 wi
<latexit sha1_base64="tqdvN1Co9lflZKGA9FYLbBE3SQc=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1OC+X1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4Dx3yZxA==</latexit>
3 36
 Bidirectional RNNs
■ In language it’s also often useful to model past and future context
■ Can also run an RNN in the backward direction (reverse reading order)
■ Combining both directions works best!

… …
… …

as the breathless runners approached the

wi
<latexit sha1_base64="5EwDIiByx9xSWpoPXUBPQBRnzvo=">AAAFD3icfVNdb9MwFM1KgRG+NnjkxTAVWhij3SaBhJAm4IGXjQH7kuoSOY7TurOTyHa2tFF+BI/8Et4Qr/wE/g3XaQvtWnDV9PT6nOvrc2/8RHBtms1fS5VL1ctXri5fc6/fuHnr9srqnSMdp4qyQxqLWJ34RDPBI3ZouBHsJFGMSF+wY//0jd0/PmNK8zg6MIOEdSTpRjzklBgIeatLX2vYl3lWoFeonXk5f7pdvEQl2JqAzQloFR23hg96zJDPeT3DJkbDRgEhSUxPyVzHoZEkK+oY/evTcGthfS5F1vib5BMk2S2TTFhDyxo0iiGwhm4tW1jDfv3c43CJPsKSBxADjENFaI5ZltSxsQqvP5UO0yA2VpxjnUosuORGe3n/EeYROiqmZRAsFind2vnIMrf2cW+vXsbtwZMrot7INvRkYmBjUrxrlaADRt+ab/WZ11+3kn5J9FbWmhvNcqF50BqDNWe89r3VSgcHMU0liwwVROt2q5mYTk6U4VSwwsWpZgmhp6TL2gAjIpnu5OUQFagGkQCFsYJvZFAZnVbkRGo9kD4wbaf0xT0bXLTXTk34opPzKEkNi+jooDAVCLy0E4kCrhg1YgCAUMWhVkR7BDpnYG7BrqljekycMTN7ESo7uQ7L02dK8iX8Vyxi5zSWkkTB4xyHRHIxCFhIUmFs38MJXmTNenDGEz12KRvZ5GLBDI4V7/KICMFCg+1jNgw/PYPLp4vfMuiFYrtQ4PuEKWJiBZUQ1bVvCvSmi+9jC//H5NEfJsDZa+VlAXAZa0GcsCgvSkhFrBn2uypOk5mC5/RloZCAhOD4iM9mZSMGDGTr4vjNg6PNjdbWxuaH7bWd1+PRXHbuOQ+cutNynjs7zjtn3zl0aMWpPKw8qzSrX6rfqt+rP0bUytJYc9eZWdWfvwFWlrLB</latexit>
wi
<latexit sha1_base64="CH1jSvOpKNyBxULr16kKG8gaFmQ=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1Oy+L1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DwViZwg==</latexit>
1 wi
<latexit sha1_base64="cCatZM/QvTiGjI99S0wQVTzzBoU=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1O6+P1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DxGqZww==</latexit>
2 wi
<latexit sha1_base64="tqdvN1Co9lflZKGA9FYLbBE3SQc=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1OC+X1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4Dx3yZxA==</latexit>
3 37
 Bidirectional RNNs
■ In language it’s also often useful to model past and future context. Combining
both directions works best!

softmax(
softmax( ))
Does it parallelize?
<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>

<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>

concatenate
… …
… …

as the breathless runners approached the

wi
<latexit sha1_base64="5EwDIiByx9xSWpoPXUBPQBRnzvo=">AAAFD3icfVNdb9MwFM1KgRG+NnjkxTAVWhij3SaBhJAm4IGXjQH7kuoSOY7TurOTyHa2tFF+BI/8Et4Qr/wE/g3XaQvtWnDV9PT6nOvrc2/8RHBtms1fS5VL1ctXri5fc6/fuHnr9srqnSMdp4qyQxqLWJ34RDPBI3ZouBHsJFGMSF+wY//0jd0/PmNK8zg6MIOEdSTpRjzklBgIeatLX2vYl3lWoFeonXk5f7pdvEQl2JqAzQloFR23hg96zJDPeT3DJkbDRgEhSUxPyVzHoZEkK+oY/evTcGthfS5F1vib5BMk2S2TTFhDyxo0iiGwhm4tW1jDfv3c43CJPsKSBxADjENFaI5ZltSxsQqvP5UO0yA2VpxjnUosuORGe3n/EeYROiqmZRAsFind2vnIMrf2cW+vXsbtwZMrot7INvRkYmBjUrxrlaADRt+ab/WZ11+3kn5J9FbWmhvNcqF50BqDNWe89r3VSgcHMU0liwwVROt2q5mYTk6U4VSwwsWpZgmhp6TL2gAjIpnu5OUQFagGkQCFsYJvZFAZnVbkRGo9kD4wbaf0xT0bXLTXTk34opPzKEkNi+jooDAVCLy0E4kCrhg1YgCAUMWhVkR7BDpnYG7BrqljekycMTN7ESo7uQ7L02dK8iX8Vyxi5zSWkkTB4xyHRHIxCFhIUmFs38MJXmTNenDGEz12KRvZ5GLBDI4V7/KICMFCg+1jNgw/PYPLp4vfMuiFYrtQ4PuEKWJiBZUQ1bVvCvSmi+9jC//H5NEfJsDZa+VlAXAZa0GcsCgvSkhFrBn2uypOk5mC5/RloZCAhOD4iM9mZSMGDGTr4vjNg6PNjdbWxuaH7bWd1+PRXHbuOQ+cutNynjs7zjtn3zl0aMWpPKw8qzSrX6rfqt+rP0bUytJYc9eZWdWfvwFWlrLB</latexit>
wi
<latexit sha1_base64="CH1jSvOpKNyBxULr16kKG8gaFmQ=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1Oy+L1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DwViZwg==</latexit>
1 wi
<latexit sha1_base64="cCatZM/QvTiGjI99S0wQVTzzBoU=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1O6+P1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4DxGqZww==</latexit>
2 wi
<latexit sha1_base64="tqdvN1Co9lflZKGA9FYLbBE3SQc=">AAAEu3icfVJbb9MwFE5GgRFuGzzyEpgqWgRTu00CCU2auEi8TAyxC1IdKsc5ad3ZcWQ7W1or/4hfwxuCH4OTttCsBUdxvhx/5/g7lzBlVOlO54e7dq1x/cbN9Vve7Tt3793f2HxwqkQmCZwQwYT8EmIFjCZwoqlm8CWVgHnI4Cw8f1uen12AVFQkx3qcQsDxIKExJVhbU3/Tfd9EITd54e/7vbxv6Iu94rVfgd052JmDbhF4TXQ8BI2/mlaOtPAn7cKaONZDyY0SseY4L1rI/9fT9ppxaylE3v4b5LMNclgFmbMmJWvcLiaWNfGa+UoNR63LPrVJjHzEaWRtFqNYYmIQ5GkL6dKjP1oIh0gkdOlskMo4YpRTrfpm9BTRxD8tFt2ssVjl6V1OC+X1N7Y6251q+cugOwNbzmwd9TfXAhQJknFINGFYqV63k+rAYKkpYVB4KFOQYnKOB9CzMMEcVGCqhhd+01oiPxbSvon2K+uih8FcqTEPLbOsqrp6VhpXnfUyHb8KDE3STENCphfFGfNt3uX0+BGVQDQbW4CJpFarT4bYVlnbGbN9WbhmCOwCdD0RwgOj4ur2mqSQ238JCVwSwTlOomcGxZhTNo4gxhnTZY/iOV5VmufRBU3VrEr5tEweYqCRkHRAE8wYxBqVW91sP0ONqt1D78D2QsKhFfgxBYm1kFYJloNyqm1vBugxKuH/mDT5w7SwnpapBNhkyhKIFBJTVJAwoQCFAymytCZ4yb8SagPg2FZ8yoe625RhB7J7dfyWwenOdnd3e+fT3tbBm9lorjuPnCdOy+k6L50D54Nz5Jw4xP3mfnd/ur8a+w3SGDXYlLrmznweOrXVyH4Dx3yZxA==</latexit>
3 38
RNNs
Good: Needs improvement:

■ Input size ■ Parallelization

■ Parameter count ■ Long-distance dependencies
■ Contextualized word
representations

■ Inductive bias

39
 Attention in neural networks
■ Neural attention mechanism: a weighted average over (word) representations,
where the weights are learned, computed as a function of the input.

■ Additive attention: ψ is a feed-forward layer with concatenated [k;q] as input.

■ Multiplicative attention: ψ is a dot product of linear projections of k, q.
■ First popularized in neural machine translation, where the network would attend over
the source sentence to make each
18.3.prediction in theTRANSLATION
NEURAL MACHINE target sentence.

Output

activation ↵

Query ↵

Key Value

40
Figure 18.6: A general view of neural attention. The dotted box indicates that each ↵m
Transformer self-attention

Q
K
V

Nobel committee awards Strickland who advanced optics

41
Transformer self-attention

Q
K
V

Nobel committee awards Strickland who advanced optics

42
Transformer self-attention

X
+ =

Q
K
V

Nobel committee awards Strickland who advanced optics

43
Transformer self-attention

Q
K
V

Nobel committee awards Strickland who advanced optics

44
Transformer self-attention

Q
K
V

Nobel committee awards Strickland who advanced optics

45
Transformer self-attention

Q
K
V

Nobel committee awards Strickland who advanced optics

46
Transformer self-attention

Q
K
V

Nobel committee awards Strickland who advanced optics

47
Transformer self-attention

Q
K
V

Nobel committee awards Strickland who advanced optics

48
Transformer self-attention

optics
advanced
who
Strickland
awards
committee
Nobel

Q
K
V

Nobel committee awards Strickland who advanced optics

49
Transformer self-attention

optics
advanced
who
Strickland
awards
committee
Nobel
A
Q
K
V

Nobel committee awards Strickland who advanced optics

50
Transformer self-attention

optics
advanced
who
Strickland
awards
committee
Nobel
A
Q
K
V

Nobel committee awards Strickland who advanced optics

51
Transformer self-attention

optics
advanced
who
Strickland
awards
committee
Nobel
A
Q
K
V

Nobel committee awards Strickland who advanced optics

52
Transformer self-attention

optics
advanced
who
Strickland
awards
committee
Nobel
A
Q
K
V

Nobel committee awards Strickland who advanced optics

53
Transformer self-attention

optics
advanced
who
Strickland
awards
committee
Nobel
A
Q
K
V

Nobel committee awards Strickland who advanced optics

54
Multi-head self-attention

MH
M1

optics
advanced
who
Strickland
awards
committee
Nobel
A
Q
K
V

Nobel committee awards Strickland who advanced optics

55
Multi-head self-attention
concat
MH
M1

optics
advanced
who
Strickland
awards
committee
Nobel
A
Q
K
V

Nobel committee awards Strickland who advanced optics

56
Transformer layer
Feed Feed Feed Feed Feed Feed Feed
Forward Forward Forward Forward Forward Forward Forward

MH
M1

+ optics
advanced
who
Strickland
awards
committee
Nobel
A
Q
K
V
Layer p
p+1
Nobel committee awards Strickland who advanced optics
57
Transformer layer Does it parallelize???

Nobel committee awards Strickland who advanced optics 58

Transformer architecture

Layer J Multi-head self-attention + feed forward

Layer p Multi-head self-attention + feed forward

Layer 1 Multi-head self-attention + feed forward

Nobel committee awards Strickland who advanced optics 59

 Transformer encoder-decoder
llegó la bruja verde
decoder

the Masked self-attention + cross-attention + feed forward Layer K

green
witch
arrived

ve a
la

e
gó

uj
rd
lle

br
Masked self-attention + cross-attention + feed forward Layer 1

encoder

Layer J Multi-head self-attention + feed forward

Layer 1 Multi-head self-attention + feed forward

the green witch arrived 60

 Transformer encoder-decoder

Add & Norm

Figure: Vaswani, et al. Attention is All You Need. NeurIPS 2017. 61

Residual connections and layer normalization
On Layer Normalizat

■ Residual connections are another way to ease optimization in

deep NNs by mitigating vanishing gradients. In Transformer, also
adds inductive bias that the representation at each timestep
should remain tied to the input.

■ Layer normalization (Ba et al. 2016) also aids optimization by

reducing extreme values, thereby keeping gradients under
control. Normalize the inputs to each layer:
d
<latexit sha1_base64="iV96+700p7Uuxwr1CkAnFjlvlXo=">AAACIHicbZC7TsMwFIYdrqXcAowsFhUSU5UgVFgqVbAwFolepCZEjuO0Vu0ksh2kKsrKgzCzwjOwIUZ4A94CN81AW45k6df/n+Njf37CqFSW9WWsrK6tb2xWtqrbO7t7++bBYVfGqcCkg2MWi76PJGE0Ih1FFSP9RBDEfUZ6/vhmmvceiZA0ju7VJCEuR8OIhhQjpS3PhA5PYRM6oUA4s/MsyB2Zci8bN+38IRh5Y8+sWXWrKLgs7FLUQFltz/xxghinnEQKMyTlwLYS5WZIKIoZyatOKkmC8BgNyUDLCHEi3az4SQ5PtRPAMBb6RAoW7t+JDHEpJ9zXnRypkVzMpuZ/2SBV4ZWb0ShJFYnwbFGYMqhiOMUCAyoIVmyiBcKC6rdCPEIaitLw5m7C3M1kWCzSbOxFEsuie163G/XG3UWtdV1SqoBjcALOgA0uQQvcgjboAAyewAt4BW/Gs/FufBifs9YVo5w5AnNlfP8Cn1KkBQ==</latexit>

1 X
µ= hk
d
k=1
h µ
<latexit sha1_base64="aPUNLZl53sg1h5n3XdO+2FG1ooU=">AAACOnicbZBBS9xAFMcnq22tte3WHr0MLgWldEmk2FIoiF48SFHo7gqbsLzMvmQHZ5Iw8yIuId/FD+LZa7322pv06gfo7JqDq30w8Of/f2/ezC8ulLTk+7+91tLys+cvVl6uvlp7/eZt+9163+alEdgTucrNaQwWlcywR5IUnhYGQccKB/HZwSwfnKOxMs9+0rTASEOayUQKIGeN2t9CwgsyujqCKZofudH11mSbf+dhClpDmBgQ1eRTqMu6Cq1MNdT8Iw9jJBi1O37Xnxd/KoJGdFhTx6P2bTjORakxI6HA2mHgFxRVYEgKhfVqWFosQJxBikMnM9Boo2r+x5p/cM6YJ7lxJyM+dx9OVKCtnerYdWqgiX2czcz/ZcOSkq9RJbOiJMzE/aKkVJxyPgPGx9KgIDV1AoSR7q1cTMBhIYd14Saho8om80WOTfCYxFPR3+kGu93dk8+dvf2G0grbYJtsiwXsC9tjh+yY9Zhgl+ya/WI33pX3x7v1/t63trxm5j1bKO/uH6oUrq0=</latexit>

mean
LayerNorm(h) = +
d
<latexit sha1_base64="DdJYa17hM++4Ya68hp6x5zDLKqk=">AAACL3icbZDLSsQwFIZT73erLt0EB0EXju0goxtBdONSwVFh2ilpmo6hSVqSdGAIfQwfxLVbfQZxI259CzN1Ft4OBH7+/5yc5IsLRpX2vFdnYnJqemZ2bn5hcWl5ZdVdW79WeSkx6eCc5fI2RoowKkhHU83IbSEJ4jEjN3F2NspvBkQqmosrPSxIyFFf0JRipK0VufuBon2Oei14DINUImz8yiRVoEoemezYr3rJziDK4B4MeLnba0Vuw2t6dcG/wh+LBhjXReR+BEmOS06Exgwp1fW9QocGSU0xI9VCUCpSIJyhPulaKRAnKjT1xyq4bZ0Eprm0R2hYu98nDOJKDXlsOznSd+p3NjL/y7qlTo9CQ0VRaiLw16K0ZFDncEQJJlQSrNnQCoQltW+F+A5ZPNqy/HET5qFRab3IsvF/k/grrltNv91sXx40Tk7HlObAJtgCO8AHh+AEnIML0AEY3INH8ASenQfnxXlz3r9aJ5zxzAb4Uc7HJ5xTqPQ=</latexit>

1 X
2 2
= (vk µ)
scaling params d
k=1
standard dev
62
Figure: Xiong, et al. On Layer Normalization in the Transformer Architecture. ICML 2020.
 Transformer encoder-decoder

Positional encoding

Figure: Vaswani, et al. Attention is All You Need. NeurIPS 2017. 63

Modeling context
■ Like RNNs, each token observes context from the entire sequence at each layer.
■ But, representations at each layer can be computed entirely in parallel (like CNNs).
■ What is lost? Markov(ish) assumption. optics
advanced
who
Strickland
awards
committee
Nobel

Nobel committee awards Strickland Nobel committee awards Strickland

RNN Transformer
64
 Figures from: Transformer Architecture: The Positional Encoding

Positional encoding
■ Positional embeddings encode a token’s position in the sentence
p1 p2 … pn
w1 + w2 + wn +

■ Sinusoidal position encodings generalize to any sequence length.

with
position

pt position position 65
 Transformer encoder-decoder

Inputs & outputs

Figure: Vaswani, et al. Attention is All You Need. NeurIPS 2017. 66
 2019)
tasks, with we find different tokenizations.
a surprising performance On downstream
gap, with number
batching ofnoninterfering
potential merges considered,
merges and usingincluding
strong
components:
which takes aacorpus
vocabulary of construction
(potentially procedure,
pre-split) vocabulary,
text a consistent pruning it to the desired size:
tasks,
the unigram we find LM a surprising
method providing performance gap, withim- batching noninterfering
priors to filter out unlikely merges
merges. andAsusingtheir strong
imple-
which takesaavocabulary
and returns corpusthe of (potentially
with
unigram the
LM pre-split)
desired
method size,
providing text
and a consistent im- 2 Unigram
priors to filter outpublic,
unlikely 1 we merges. As their imple-
provement over BPE of up to 1% depending on the
Algorithm mentation is not
language modeling are (Kudo,
unable to make a
and returns a vocabulary
a tokenization procedure,
provement with
which
overthe desired
takes
BPE the
of upsize,
built
to
task, indicating the benefits of this technique. 1%and
vo-depending on
Algorithm the mentation
2 Unigram is
comparisonlanguage not public,
to this method.
1 we are unable to make a
modeling (Kudo,
Sub-word representations
2018)
acabulary
tokenization procedure,
and applies towhich
task,itindicating new takes the
the benefits
text, built
returning vo-
of this a technique.
2018) comparison to this method.
1: Input: set ofThe unigram
strings D, LM method
target vocab (Kudo,
size k 2018), in con-
cabulary
sequence and applies
of tokens. ittheory,
to new
2 InAlgorithms thesetext,two returning
steps cana The unigram LM method (Kudo, 2018),of in BPE
con-
2 InAlgorithms 1: Input: set of
trast strings
to
2: procedure U NIGRAM LM(D, k)the target
bottom-up
D, vocab
construction size kprocess
sequence of tokens.
be independent, although theory, these two steps can
Subwordwe will see that for the consist trast
and to the bottom-up
WordPiece, begins construction
with a supersetprocessof of BPE
the final
■ Word embeddings suffer from UNK/OOV, but character embeddings are slow.
be independent,
algorithms although
we examine Subword the we
tokenization
will
tokenization
tokenization see
components: a vocabulary construction
algorithms
that for
procedure
algorithms the consist
2:
3: procedure
of
of
procedure,
3:
4:
two
two
V
V
allU NIGRAM
substrings
and WordPiece,
vocabulary,
all substrings
once in D
LM(D,
occurring
pruning
(not
begins
occurringit
crossing
k)
towith
more
the adesired
more than
words)
than
superset of the final
size:
algorithms we examine
is tightly coupled to thethe
components:
which tokenization
vocabulary
takes aacorpus ofprocedure
construction
vocabulary construction
(potentially procedure,
pre-split) text vocabulary, pruning it to the desired size:
4: while once in kDdo (not crossing words)
5: unigram |V | > LM [Kudo, . Prune
2018] tokens
■ Solution: sub-word units.
is tightly
procedure.
procedure.
coupled
A BPE vocabulary
to
which
and
the vocabulary
takes
isreturns
a tokenization
a corpus
a vocabulary
constructed
construction
procedure,
of (potentially
and returns a vocabulary with the desired5:size, and
with
as follows: which thetakes
pre-split)
desired 6:
thesize,
text
builtand
whileFit|V | > k do
Algorithm
unigram
vo- Algorithm
2 Unigram
LM ✓
2 Unigram
to D . Prune tokens
language modeling (Kudo,
language modeling (Kudo,
6:
7:built vo- 2018)
Fit unigram LM ✓ to
for t 2 V do . Estimate token ‘loss’D
A BPE vocabulary acabulary
tokenization
is constructed procedure,
as which
follows: takes the
BPE [Gage, 1994;cabulary Sennrich and
and
et
applies
applies
al., it
it
to
to
2016]
new
new
text,
text,
returning
7:
returning
8:
a
a
2018)
for 1:L
t2 Input:
V do
p set
(D) of
. Estimate
strings
p (D)D, token
target ‘loss’
vocab size k
Algorithm 1 Byte-pair sequence of tokens.
encoding In theory,
(Sennrich etthese
al., two steps can 1:
t
Input:
✓
set ofUstrings
✓ 0
target vocab
D,LM(D, size k
sequence of tokens. In theory, these two 8:
steps can 2:
L procedure
where p 0 (D)
✓ is the LM NIGRAM
p (D)
✓ without token t k)
Algorithm 1
2016; Gage, 1994) Byte-pair
be encoding
independent, (Sennrich
although we et al.,
will see 9:
that for the 2:
t ✓
procedure0 isall Usubstrings
0
NIGRAM LM(D, k) more than
be independent, although we will see that
9: for the 3:
where V ✓ the LM withoutoccurring
token t
2016; Gage, 1994) algorithms we examine the tokenization procedure
10: end 3:
for V all substrings occurring more than
1: Input: set of strings algorithms D, target
we examinevocab size k procedure end 4: once in
for min(|V | k, b↵|V |c) of theD (not crossing words)
is tightly coupled to thethe tokenization
vocabulary 10:
construction
11: Remove once in kDdo (not crossing words)
2: Input: set ofBPE(D,
strings target vocab size
1: D, k 4: while Prune tokens
procedure is tightly
procedure. k) coupled to the vocabulary construction
11:
5:
Remove min(|V |V | >
|| >k, b↵|V |c)V of .
the
2: procedure BPE(D, k) 12: tokens
5:
6:
t with
while Fithighest
|Vunigram kL do
t from
LM ✓ to D, . Prune tokens
3: V all unique
procedure.
A BPE characters
vocabulary in D is constructed as follows: 12: tokens t with highest LtLM from VD ,
3: V all unique characters in D 13: where
6:
7: ↵ 2 [0,Fit
for 1]t is
unigram
2 aV hyperparameter
do .✓ to
Estimate token ‘loss’
4: A BPE vocabulary
(about 4,000 in English Wikipedia) is constructed as follows: where
13:
14: end while
7:
8:
↵ 2 [0, for1]Lt is
2 aV hyperparameter
do
p (D) . Estimate
p 0 (D) token ‘loss’
4:
5: while(about
|V | < 4,000
k do in 1English
Algorithm Byte-pair Wikipedia)
. Merge encoding
tokens(Sennrich 14:
et al.,
end while
t ✓ ✓

5: while |V | Algorithm
<2016;
k do Gage, 11994)
Byte-pair
. Merge encoding
tokens 15:
(Sennrich et al., Fit final8:
9: unigram LM L
where
t ✓ to p
✓ D
✓0 (D)
is the p
LM ✓ 0 (D)
without token t
6: tL , tR Most frequent bigram in D 15: Fit final9: unigram LM where✓ to D0 is the LM without token t
2016; Gage, 1994) 16: return 10:V, ✓ end for ✓
6:
7: ttL ,t
NEWR tLMost
1: + tRfrequent
Input: set .ofMake bigram
strings new in
D, target D vocab size k
token 16: return 10:
11:V, ✓ end
Remove for min(|V | k, b↵|V |c) of the
7: t 1:
t
2: Input:
procedure
+ t set of
. strings
Make
BPE(D, new
D,
k) target
token vocab 17:
sizeendk procedure
8: V NEW V + [tNEW ]
L R 17: end procedure 11:
12: Remove
tokens t min(|V
with | k,Lb↵|V
highest from |c)V of
, the
2: procedure BPE(D, k) t
8:
9: V V 3:
+
Replace eachNEW [t occurrence of tL , tR in in D
V ] all unique characters 12:
13: tokens
where t with
↵ 2 [0, highest
1] is a Lt from V ,
hyperparameter
Replace 3:
each
4: V
occurrence all unique
(about 4,000
of characters
in English
in in D
Wikipedia)
9:
10: D with t t L , t R 13:
14: end where
while ↵ 2 [0, 1] is a hyperparameter
4: NEW
while (about 4,000do in English Wikipedia)
Merge tokens LM
Unigram 15: tokenization takes the vocabulary
10: D with5: t |V | < k . 14: end while
11: end while 5: NEWwhile |V | < k do Merge tokens Fit final unigram LM ✓ to D
11: end while 6: t L , tR Most frequent . bigram
V Unigram
andin D LM
unigram 15: tokenization
LM parameters
Fit
returnfinal takes
unigram ✓ the
and
LM vocabulary
performs
✓ to D
12: return V 6: ttL ,t tLMost+ tRfrequent . Makebigram
new in D
token
16: V, ✓
12: return V
7: NEWR V and
Viterbi unigram
inference16:
17:
LM
to
end parameters
return
decode
procedure the
V, ✓ ✓ and
segmentation performs with
13: end procedure 7: tVNEW V + tL [t+NEW tR] . MakeViterbi new token
13: end procedure 8:
8:
maximum inference17: to
likelihood end decode
procedure
under the
✓. segmentation
This method with
closely
V V for
+ [tLanguage] 67
Figures from: Bostrom and Durrett. Byte Pair Encoding 9: Replace
is Suboptimal each occurrence
NEW Model ofmaximum
tL , tR in likelihood
Pretraining. EMNLP Findings, under2020. ✓. This method closely
 Sub-word representations Impact on e ciency?

■ Word embeddings suffer from UNK/OOV, but character embeddings are slow.
■ Solution: sub-word units.
■ BPE and unigram LM both implemented in the SentencePiece library

Original:furiously
Original: furiously Original:tricycles
Original: tricycles
(a) (a) BPE:BPE: fur fur iously
iously (b) (b) BPE:BPE: t tric ricy yclescles
Original: furiously Original: tricycles
Unigram
Unigram LM: LM: fur furiousiously ly Unigram
Unigram LM:LM: tri tricyclecycles s
(a) BPE: fur iously (b) BPE: t ric y cles
Original:
Unigram
Original: Completely
LM:Completely
fur ious preposterous
ly
preposterous suggestions
Unigram LM:
suggestions tri cycle s
(c) (c) BPE: Comple
BPE: Complet tely ely prepprepost osterous erous suggestsuggestionsions
Original: Completely preposterous suggestions
Unigram
Unigram LM:LM: Complete
Completely ly pre prepostposter erous ous suggestion
suggestions s
(c) BPE: Comple t ely prep ost erous suggest ions
Unigram LM: Complete ly pre post er ous suggestion s
Example tokenizations.
mple tokenizations. TheThe character
character ‘ ’aisspecial
‘ ’ is a special start-of-word
start-of-word marker
marker inserted
inserted during
during tokenizat
tokenization
r recovery
covery of of
wordword boundaries.
boundaries. Additional
Additional examples
examples are are included
included in in
the the supplementary
supplementary material.
material.
Example tokenizations. The character ‘ ’ is a special start-of-word marker inserted during tokeniza 68
Figures from: Bostrom and Durrett. Byte Pair Encoding is Suboptimal for Language Model Pretraining. EMNLP Findings, 2020.
ffi
 Transformer encoder-decoder

Figure: Vaswani, et al. Attention is All You Need. NeurIPS 2017. 69

 Transformer LLMs
■ Encoder-only: (Ro)BERT(a), etc.
decoder
llegó la bruja verde

■ Encoder-decoder: T5, BART, … Masked self-attention + cross-attention + feed forward Layer K

■ Decoder-only: GPT, Llama, …

Masked self-attention + cross-attention + feed forward Layer 1

encoder

Layer J Multi-head self-attention + feed forward

Layer 1 Multi-head self-attention + feed forward

the green witch arrived 70

BERT
■ Non-autoregressive (encoder-only) Transformer language model,
trained with masked language modeling (MLM) objective.

Layer J Multi-head self-attention + feed forward

Layer p Multi-head self-attention + feed forward

Layer 1 Multi-head self-attention + feed forward

Nobel committee awards Strickland who advanced optics

71
BERT si: which sequence
is this token from?
Next sentence prediction:
si
does the second sequence follow the rst? +
xi = pi
predict masked tokens +
wi
IsNext committee Strickland

Multi-head self-attention + feed forward

Multi-head self-attention + feed forward

Multi-head self-attention + feed forward

[CLS] Nobel committee

[MASK] awards Strickland
[MASK] who advanced optics [SEP] She …
NSP mask 15% input: two 128-512 token sequences 72
fi
 Finetuning pretrained MLMs
■ Finetuning recipe:
1. Initialize model parameters with weights of pretrained LM.
2. Add a softmax classi er “head” (randomly initialized).
3. Perform supervised training on end-task data. Can update all weights, or just train
the new classi er weights, but updating all weights almost always works best.

73
Figure from: Devlin et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. ACL 2019.
fi
fi
 Fine-tuning pretrained MLMs
■ Sentence classi cation:
■ Provide single sentence as input.
■ Use nal representation at [CLS] position as features for a linear classi er.
ŷ = positive
<latexit sha1_base64="flW23ooh6V4cbOzWIpWNQ0wJlNg=">AAACCHicbVDLSgMxFM3UV62vqks3wSK4KjMi1Y1QdOOygn1gO5RMmmlDk8yQ3BHK0B9w7Va/wZ249S/8BP/CdDoLWz0QOJxzXzlBLLgB1/1yCiura+sbxc3S1vbO7l55/6BlokRT1qSRiHQnIIYJrlgTOAjWiTUjMhCsHYxvZn77kWnDI3UPk5j5kgwVDzklYKWH3ohAOpniK9wvV9yqmwH/JV5OKihHo1/+7g0imkimgApiTNdzY/BTooFTwaalXmJYTOiYDFnXUkUkM36aXTzFJ1YZ4DDS9inAmfq7IyXSmIkMbKUkMDLL3kz8z+smEF76KVdxAkzR+aIwERgiPPs+HnDNKIiJJYRqbm/FdEQ0oWBDWphEpZ+aMFtks/GWk/hLWmdVr1at3Z1X6td5SkV0hI7RKfLQBaqjW9RATUSRQs/oBb06T86b8+58zEsLTt5ziBbgfP4AnuOacg==</latexit>

softmax(
softmax( ) )
<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>

<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>

<latexit sha1_base64="8pv/+ZuGd1Uyqg6nzbMYaugyxmE=">AAACEnicbVC7TsMwFHV4lvIKZWSxqJDKUiUIFcYKFsYi9SU1oXJcp7HqOJHtIKKof8HMCt/Ahlj5AT6Bv8BNM9CWI1k6Oue+fLyYUaks69tYW9/Y3Nou7ZR39/YPDs2jSldGicCkgyMWib6HJGGUk46iipF+LAgKPUZ63uR25vceiZA04m2VxsQN0ZhTn2KktDQ0K047IAo9ZLUAOiqC6fl0aFatupUDrhK7IFVQoDU0f5xRhJOQcIUZknJgW7FyMyQUxYxMy04iSYzwBI3JQFOOQiLdLL99Cs+0MoJ+JPTjCubq344MhVKmoacrQ6QCuezNxP+8QaL8azejPE4U4Xi+yE8Y1J+cBQFHVBCsWKoJwoLqWyEOkEBY6bgWJuHQzaSfL9LZ2MtJrJLuRd1u1Bv3l9XmTZFSCZyAU1ADNrgCTXAHWqADMHgCL+AVvBnPxrvxYXzOS9eMoucYLMD4+gXvMZ3F</latexit>

(h!y )
⇥
hCLS
<latexit sha1_base64="AoSMBCiwdDTRgOfmaQCti6eEde8=">AAACBXicbVDLTgIxFL2DL8QX6tJNIzFxRWaMQZdENi5cYJRHAhPSKR1oaDuTtmNCJqxdu9VvcGfc+h1+gn9hGVgIeJImJ+fcV08Qc6aN6347ubX1jc2t/HZhZ3dv/6B4eNTUUaIIbZCIR6odYE05k7RhmOG0HSuKRcBpKxjVpn7riSrNIvloxjH1BR5IFjKCjZVaw15au3uY9Iolt+xmQKvEm5MSzFHvFX+6/YgkgkpDONa647mx8VOsDCOcTgrdRNMYkxEe0I6lEguq/TQ7d4LOrNJHYaTskwZl6t+OFAutxyKwlQKboV72puJ/Xicx4bWfMhknhkoyWxQmHJkITf+O+kxRYvjYEkwUs7ciMsQKE2MTWphEhJ/qMFtks/GWk1glzYuyVylX7i9L1Zt5Snk4gVM4Bw+uoAq3UIcGEBjBC7zCm/PsvDsfzuesNOfMe45hAc7XL7Q5mW4=</latexit>

Multi-head self-attention + feed forward

Multi-head self-attention + feed forward

[CLS] I like green eggs

74
fi
fi
fi
 Fine-tuning pretrained MLMs
■ Sentence pair classi cation:
■ Provide both sentences as input, separated by [SEP]. This is called a cross-encoder.
■ Input embeddings indicate sentence identity.
■ Use nal representation at [CLS] position as features for a linear classi er.
ŷ = contradiction
<latexit sha1_base64="flW23ooh6V4cbOzWIpWNQ0wJlNg=">AAACCHicbVDLSgMxFM3UV62vqks3wSK4KjMi1Y1QdOOygn1gO5RMmmlDk8yQ3BHK0B9w7Va/wZ249S/8BP/CdDoLWz0QOJxzXzlBLLgB1/1yCiura+sbxc3S1vbO7l55/6BlokRT1qSRiHQnIIYJrlgTOAjWiTUjMhCsHYxvZn77kWnDI3UPk5j5kgwVDzklYKWH3ohAOpniK9wvV9yqmwH/JV5OKihHo1/+7g0imkimgApiTNdzY/BTooFTwaalXmJYTOiYDFnXUkUkM36aXTzFJ1YZ4DDS9inAmfq7IyXSmIkMbKUkMDLL3kz8z+smEF76KVdxAkzR+aIwERgiPPs+HnDNKIiJJYRqbm/FdEQ0oWBDWphEpZ+aMFtks/GWk/hLWmdVr1at3Z1X6td5SkV0hI7RKfLQBaqjW9RATUSRQs/oBb06T86b8+58zEsLTt5ziBbgfP4AnuOacg==</latexit>

softmax(
softmax(
<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>

<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>
) )
<latexit sha1_base64="8pv/+ZuGd1Uyqg6nzbMYaugyxmE=">AAACEnicbVC7TsMwFHV4lvIKZWSxqJDKUiUIFcYKFsYi9SU1oXJcp7HqOJHtIKKof8HMCt/Ahlj5AT6Bv8BNM9CWI1k6Oue+fLyYUaks69tYW9/Y3Nou7ZR39/YPDs2jSldGicCkgyMWib6HJGGUk46iipF+LAgKPUZ63uR25vceiZA04m2VxsQN0ZhTn2KktDQ0K047IAo9ZLUAOiqC6fl0aFatupUDrhK7IFVQoDU0f5xRhJOQcIUZknJgW7FyMyQUxYxMy04iSYzwBI3JQFOOQiLdLL99Cs+0MoJ+JPTjCubq344MhVKmoacrQ6QCuezNxP+8QaL8azejPE4U4Xi+yE8Y1J+cBQFHVBCsWKoJwoLqWyEOkEBY6bgWJuHQzaSfL9LZ2MtJrJLuRd1u1Bv3l9XmTZFSCZyAU1ADNrgCTXAHWqADMHgCL+AVvBnPxrvxYXzOS9eMoucYLMD4+gXvMZ3F</latexit>

(h!y )
⇥
hCLS
<latexit sha1_base64="AoSMBCiwdDTRgOfmaQCti6eEde8=">AAACBXicbVDLTgIxFL2DL8QX6tJNIzFxRWaMQZdENi5cYJRHAhPSKR1oaDuTtmNCJqxdu9VvcGfc+h1+gn9hGVgIeJImJ+fcV08Qc6aN6347ubX1jc2t/HZhZ3dv/6B4eNTUUaIIbZCIR6odYE05k7RhmOG0HSuKRcBpKxjVpn7riSrNIvloxjH1BR5IFjKCjZVaw15au3uY9Iolt+xmQKvEm5MSzFHvFX+6/YgkgkpDONa647mx8VOsDCOcTgrdRNMYkxEe0I6lEguq/TQ7d4LOrNJHYaTskwZl6t+OFAutxyKwlQKboV72puJ/Xicx4bWfMhknhkoyWxQmHJkITf+O+kxRYvjYEkwUs7ciMsQKE2MTWphEhJ/qMFtks/GWk1glzYuyVylX7i9L1Zt5Snk4gVM4Bw+uoAq3UIcGEBjBC7zCm/PsvDsfzuesNOfMe45hAc7XL7Q5mW4=</latexit>

Multi-head self-attention + feed forward

Multi-head self-attention + feed forward

[CLS] I like eggs [SEP] Eggs are the worst 75

fi
fi
fi
 Fine-tuning pretrained MLMs
■ Sequence labeling:
■ Use nal representations at each timestep as features for a linear classi er.

ŷ = O O B-Food
<latexit sha1_base64="flW23ooh6V4cbOzWIpWNQ0wJlNg=">AAACCHicbVDLSgMxFM3UV62vqks3wSK4KjMi1Y1QdOOygn1gO5RMmmlDk8yQ3BHK0B9w7Va/wZ249S/8BP/CdDoLWz0QOJxzXzlBLLgB1/1yCiura+sbxc3S1vbO7l55/6BlokRT1qSRiHQnIIYJrlgTOAjWiTUjMhCsHYxvZn77kWnDI3UPk5j5kgwVDzklYKWH3ohAOpniK9wvV9yqmwH/JV5OKihHo1/+7g0imkimgApiTNdzY/BTooFTwaalXmJYTOiYDFnXUkUkM36aXTzFJ1YZ4DDS9inAmfq7IyXSmIkMbKUkMDLL3kz8z+smEF76KVdxAkzR+aIwERgiPPs+HnDNKIiJJYRqbm/FdEQ0oWBDWphEpZ+aMFtks/GWk/hLWmdVr1at3Z1X6td5SkV0hI7RKfLQBaqjW9RATUSRQs/oBb06T86b8+58zEsLTt5ziBbgfP4AnuOacg==</latexit>

softmax(
softmax(
<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>

<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>
) )
<latexit sha1_base64="8pv/+ZuGd1Uyqg6nzbMYaugyxmE=">AAACEnicbVC7TsMwFHV4lvIKZWSxqJDKUiUIFcYKFsYi9SU1oXJcp7HqOJHtIKKof8HMCt/Ahlj5AT6Bv8BNM9CWI1k6Oue+fLyYUaks69tYW9/Y3Nou7ZR39/YPDs2jSldGicCkgyMWib6HJGGUk46iipF+LAgKPUZ63uR25vceiZA04m2VxsQN0ZhTn2KktDQ0K047IAo9ZLUAOiqC6fl0aFatupUDrhK7IFVQoDU0f5xRhJOQcIUZknJgW7FyMyQUxYxMy04iSYzwBI3JQFOOQiLdLL99Cs+0MoJ+JPTjCubq344MhVKmoacrQ6QCuezNxP+8QaL8azejPE4U4Xi+yE8Y1J+cBQFHVBCsWKoJwoLqWyEOkEBY6bgWJuHQzaSfL9LZ2MtJrJLuRd1u1Bv3l9XmTZFSCZyAU1ADNrgCTXAHWqADMHgCL+AVvBnPxrvxYXzOS9eMoucYLMD4+gXvMZ3F</latexit>

(h!y )
⇥

Multi-head self-attention + feed forward

Multi-head self-attention + feed forward

[CLS] I like shakshuka

76
fi
fi
 Fine-tuning pretrained MLMs
■ Sequence labeling:
■ Use nal representations at each timestep as features for a linear classi er.
■ What to do with sub-tokens? pooling: aggregation fn over embeddings
ŷ = B-Food
<latexit sha1_base64="flW23ooh6V4cbOzWIpWNQ0wJlNg=">AAACCHicbVDLSgMxFM3UV62vqks3wSK4KjMi1Y1QdOOygn1gO5RMmmlDk8yQ3BHK0B9w7Va/wZ249S/8BP/CdDoLWz0QOJxzXzlBLLgB1/1yCiura+sbxc3S1vbO7l55/6BlokRT1qSRiHQnIIYJrlgTOAjWiTUjMhCsHYxvZn77kWnDI3UPk5j5kgwVDzklYKWH3ohAOpniK9wvV9yqmwH/JV5OKihHo1/+7g0imkimgApiTNdzY/BTooFTwaalXmJYTOiYDFnXUkUkM36aXTzFJ1YZ4DDS9inAmfq7IyXSmIkMbKUkMDLL3kz8z+smEF76KVdxAkzR+aIwERgiPPs+HnDNKIiJJYRqbm/FdEQ0oWBDWphEpZ+aMFtks/GWk/hLWmdVr1at3Z1X6td5SkV0hI7RKfLQBaqjW9RATUSRQs/oBb06T86b8+58zEsLTt5ziBbgfP4AnuOacg==</latexit>

softmax(
softmax( ) )
<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>

<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>

<latexit sha1_base64="8pv/+ZuGd1Uyqg6nzbMYaugyxmE=">AAACEnicbVC7TsMwFHV4lvIKZWSxqJDKUiUIFcYKFsYi9SU1oXJcp7HqOJHtIKKof8HMCt/Ahlj5AT6Bv8BNM9CWI1k6Oue+fLyYUaks69tYW9/Y3Nou7ZR39/YPDs2jSldGicCkgyMWib6HJGGUk46iipF+LAgKPUZ63uR25vceiZA04m2VxsQN0ZhTn2KktDQ0K047IAo9ZLUAOiqC6fl0aFatupUDrhK7IFVQoDU0f5xRhJOQcIUZknJgW7FyMyQUxYxMy04iSYzwBI3JQFOOQiLdLL99Cs+0MoJ+JPTjCubq344MhVKmoacrQ6QCuezNxP+8QaL8azejPE4U4Xi+yE8Y1J+cBQFHVBCsWKoJwoLqWyEOkEBY6bgWJuHQzaSfL9LZ2MtJrJLuRd1u1Bv3l9XmTZFSCZyAU1ADNrgCTXAHWqADMHgCL+AVvBnPxrvxYXzOS9eMoucYLMD4+gXvMZ3F</latexit>

(h!y )
ŷ = O O
<latexit sha1_base64="flW23ooh6V4cbOzWIpWNQ0wJlNg=">AAACCHicbVDLSgMxFM3UV62vqks3wSK4KjMi1Y1QdOOygn1gO5RMmmlDk8yQ3BHK0B9w7Va/wZ249S/8BP/CdDoLWz0QOJxzXzlBLLgB1/1yCiura+sbxc3S1vbO7l55/6BlokRT1qSRiHQnIIYJrlgTOAjWiTUjMhCsHYxvZn77kWnDI3UPk5j5kgwVDzklYKWH3ohAOpniK9wvV9yqmwH/JV5OKihHo1/+7g0imkimgApiTNdzY/BTooFTwaalXmJYTOiYDFnXUkUkM36aXTzFJ1YZ4DDS9inAmfq7IyXSmIkMbKUkMDLL3kz8z+smEF76KVdxAkzR+aIwERgiPPs+HnDNKIiJJYRqbm/FdEQ0oWBDWphEpZ+aMFtks/GWk/hLWmdVr1at3Z1X6td5SkV0hI7RKfLQBaqjW9RATUSRQs/oBb06T86b8+58zEsLTt5ziBbgfP4AnuOacg==</latexit>

⇥
softmax(
softmax(
<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>

<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>
) ) m
<latexit sha1_base64="OHnlrXy7IVgBBmq0P8doDHCMyTM=">AAAB/3icbVDLSgMxFL1TX7W+qi7dBIvgqsyIVJdFNy5bsA9oh5JJM21okhmSjFCGLly71W9wJ279FD/BvzCdzsK2HggczrmvnCDmTBvX/XYKG5tb2zvF3dLe/sHhUfn4pK2jRBHaIhGPVDfAmnImacsww2k3VhSLgNNOMLmf+50nqjSL5KOZxtQXeCRZyAg2VmqKQbniVt0MaJ14OalAjsag/NMfRiQRVBrCsdY9z42Nn2JlGOF0VuonmsaYTPCI9iyVWFDtp9mhM3RhlSEKI2WfNChT/3akWGg9FYGtFNiM9ao3F//zeokJb/2UyTgxVJLFojDhyERo/ms0ZIoSw6eWYKKYvRWRMVaYGJvN0iQi/FSH2SKbjbeaxDppX1W9WrXWvK7U7/KUinAG53AJHtxAHR6gAS0gQOEFXuHNeXbenQ/nc1FacPKeU1iC8/ULRJaW/g==</latexit>

Examples:
<latexit sha1_base64="8pv/+ZuGd1Uyqg6nzbMYaugyxmE=">AAACEnicbVC7TsMwFHV4lvIKZWSxqJDKUiUIFcYKFsYi9SU1oXJcp7HqOJHtIKKof8HMCt/Ahlj5AT6Bv8BNM9CWI1k6Oue+fLyYUaks69tYW9/Y3Nou7ZR39/YPDs2jSldGicCkgyMWib6HJGGUk46iipF+LAgKPUZ63uR25vceiZA04m2VxsQN0ZhTn2KktDQ0K047IAo9ZLUAOiqC6fl0aFatupUDrhK7IFVQoDU0f5xRhJOQcIUZknJgW7FyMyQUxYxMy04iSYzwBI3JQFOOQiLdLL99Cs+0MoJ+JPTjCubq344MhVKmoacrQ6QCuezNxP+8QaL8azejPE4U4Xi+yE8Y1J+cBQFHVBCsWKoJwoLqWyEOkEBY6bgWJuHQzaSfL9LZ2MtJrJLuRd1u1Bv3l9XmTZFSCZyAU1ADNrgCTXAHWqADMHgCL+AVvBnPxrvxYXzOS9eMoucYLMD4+gXvMZ3F</latexit>

⇥ (h!y ) h1
<latexit sha1_base64="9l/pRVE2yVl6pRnOQGSXq23qwrI=">AAACAXicbVDLSgMxFL1TX7W+qi7dBIvgqsyIVJdFNy4r2ge0Q8mkmTY0yQxJRihDV67d6je4E7d+iZ/gX5hOZ2FbDwQO59xXThBzpo3rfjuFtfWNza3idmlnd2//oHx41NJRoghtkohHqhNgTTmTtGmY4bQTK4pFwGk7GN/O/PYTVZpF8tFMYuoLPJQsZAQbKz2M+l6/XHGrbga0SrycVCBHo1/+6Q0ikggqDeFY667nxsZPsTKMcDot9RJNY0zGeEi7lkosqPbT7NQpOrPKAIWRsk8alKl/O1IstJ6IwFYKbEZ62ZuJ/3ndxITXfspknBgqyXxRmHBkIjT7NxowRYnhE0swUczeisgIK0yMTWdhEhF+qsNskc3GW05ilbQuql6tWru/rNRv8pSKcAKncA4eXEEd7qABTSAwhBd4hTfn2Xl3PpzPeWnByXuOYQHO1y9u9Zed</latexit>

… f … hS
<latexit sha1_base64="21apwD9Lh5QV/p/eeOVSwWzNln8=">AAACAXicbVDLTgIxFL2DL8QX6tJNIzFxRWaMQZdENy4xCJLAhHRKBxrazqTtmJAJK9du9RvcGbd+iZ/gX1iGWQh4kiYn59xXTxBzpo3rfjuFtfWNza3idmlnd2//oHx41NZRoghtkYhHqhNgTTmTtGWY4bQTK4pFwOljML6d+Y9PVGkWyQcziakv8FCykBFsrNQc9Zv9csWtuhnQKvFyUoEcjX75pzeISCKoNIRjrbueGxs/xcowwum01Es0jTEZ4yHtWiqxoNpPs1On6MwqAxRGyj5pUKb+7Uix0HoiAlspsBnpZW8m/ud1ExNe+ymTcWKoJPNFYcKRidDs32jAFCWGTyzBRDF7KyIjrDAxNp2FSUT4qQ6zRTYbbzmJVdK+qHq1au3+slK/yVMqwgmcwjl4cAV1uIMGtIDAEF7gFd6cZ+fd+XA+56UFJ+85hgU4X7+li5e/</latexit>

<latexit sha1_base64="a+FfmhKkkkOjs8mK4QdtY8ubLZo=">AAACGnicbVDLSsNAFJ3UV62vqDvdDBbBVUlEqptC0Y3LivYBbRom00k77UwSZiZCCQU/xLVb/QZ34taNn+BfOE2zsK0HLhzOuXfu3ONFjEplWd9GbmV1bX0jv1nY2t7Z3TP3DxoyjAUmdRyyULQ8JAmjAakrqhhpRYIg7jHS9EY3U7/5SISkYfCgxhFxOOoH1KcYKS255hHvJsMJrMCOjLmb0Io96d7DgUu7Q9csWiUrBVwmdkaKIEPNNX86vRDHnAQKMyRl27Yi5SRIKIoZmRQ6sSQRwiPUJ21NA8SJdJL0hgk81UoP+qHQFSiYqn8nEsSlHHNPd3KkBnLRm4r/ee1Y+VdOQoMoViTAs0V+zKAK4TQQ2KOCYMXGmiAsqP4rxAMkEFY6trmXMHcS6aeLdDb2YhLLpHFessul8t1FsXqdpZQHx+AEnAEbXIIquAU1UAcYPIEX8ArejGfj3fgwPmetOSObOQRzML5+AR15oQ4=</latexit>

S
X j
sum pooling m = j
hi
i=1
Multi-head self-attention + feed forward
S
<latexit sha1_base64="DfT1g7GaEVSyisaKyACb5nfY4kg=">AAACJXicbZDLTgIxFIY7eEO8oS7dNBISV2TGGHRDQnTjEoOACQyTTulAoZ2ZtB0T0swT+CCu3eozuDMmrlz7FpbLQsCTNPnz/+f0tJ8fMyqVbX9ZmbX1jc2t7HZuZ3dv/yB/eNSUUSIwaeCIReLBR5IwGpKGooqRh1gQxH1GWv7oZpK3HomQNArv1TgmLkf9kAYUI2UsL1/kXT1MYQV2AoGwdlJdTzsy4Z6mFSft1uHAo92hly/YJXtacFU4c1EA86p5+Z9OL8IJJ6HCDEnZduxYuRoJRTEjaa6TSBIjPEJ90jYyRJxIV0+/k8KicXowiIQ5oYJT9++ERlzKMfdNJ0dqIJeziflf1k5UcOVqGsaJIiGeLQoSBlUEJ2xgjwqCFRsbgbCg5q0QD5DhogzBhZswd7UMposMG2eZxKponpeccql8d1GoXs8pZcEJOAVnwAGXoApuQQ00AAZP4AW8gjfr2Xq3PqzPWWvGms8cg4Wyvn8BVJ6l6A==</latexit>

1 X
mean pooling m =
j j
hi
Multi-head self-attention + feed forward
S
i=1

S
<latexit sha1_base64="Nwpb9+QfGKPshd2TUwL/Rm/701Y=">AAACGnicbVC7TsMwFHV4lvIKsMFiUSExVQlChaVSBQtjEfQhtWnkuE7r1k4i20FUUSU+hJkVvoENsbLwCfwFbpqBthzpSkfn3Ovre7yIUaks69tYWl5ZXVvPbeQ3t7Z3ds29/boMY4FJDYcsFE0PScJoQGqKKkaakSCIe4w0vOH1xG88ECFpGNyrUUQcjnoB9SlGSkuuecg7yWAMy7DN0aOb0LI97tzBvks7A9csWEUrBVwkdkYKIEPVNX/a3RDHnAQKMyRly7Yi5SRIKIoZGefbsSQRwkPUIy1NA8SJdJL0hjE80UoX+qHQFSiYqn8nEsSlHHFPd3Kk+nLem4j/ea1Y+ZdOQoMoViTA00V+zKAK4SQQ2KWCYMVGmiAsqP4rxH0kEFY6tpmXMHcS6aeLdDb2fBKLpH5WtEvF0u15oXKVpZQDR+AYnAIbXIAKuAFVUAMYPIEX8ArejGfj3fgwPqetS0Y2cwBmYHz9AgRyoP8=</latexit>

j
max pooling m = j
max hi
i=1
[CLS] _I _like _sh ak sh uka 77
fi
fi
 Fine-tuning pretrained MLMs
■ Reading comprehension:
■ Use nal representations at each timestep as features to score every token as
start or end of a span. Take the highest-scoring pair.
argmax
<latexit sha1_base64="tTEivhUuoFATxS4vwjnh2DuoBho=">AAACCXicbVDLSgMxFM3UV62vqks3wSLUTZkRqW6EohuXFewDpkPJpJk2NI8hyQhl6Be4dqvf4E7c+hV+gn9hOp2FbT0QOJxzXzlhzKg2rvvtFNbWNza3itulnd29/YPy4VFby0Rh0sKSSdUNkSaMCtIy1DDSjRVBPGSkE47vZn7niShNpXg0k5gEHA0FjShGxkp+s6oNUuYc3sB+ueLW3AxwlXg5qYAczX75pzeQOOFEGMyQ1r7nxiZI7TiKGZmWeokmMcJjNCS+pQJxooM0O3kKz6wygJFU9gkDM/VvR4q41hMe2kqOzEgvezPxP89PTHQdpFTEiSECzxdFCYNGwtn/4YAqgg2bWIKwovZWiEdIIWxsSguTMA9SHWWLbDbechKrpH1R8+q1+sNlpXGbp1QEJ+AUVIEHrkAD3IMmaAEMJHgBr+DNeXbenQ/nc15acPKeY7AA5+sXZbGaQQ==</latexit>

P(start) = 0.004 0.005 0.034 0.981 0.004 0.082

<latexit sha1_base64="y5EJfp4jnV50neibdRTykyCWAB0=">AAACB3icbVDLSgMxFM34rPVVdekmWIS6KTMi1Y1QdOOygn1IO5RMeqcNTTJDkhHK0A9w7Va/wZ249TP8BP/CdNqFbT0QOJxzXzlBzJk2rvvtrKyurW9s5rby2zu7e/uFg8OGjhJFoU4jHqlWQDRwJqFumOHQihUQEXBoBsPbid98AqVZJB/MKAZfkL5kIaPEWOmxVgLZO8PXuFsoumU3A14m3owU0Qy1buGn04toIkAayonWbc+NjZ8SZRjlMM53Eg0xoUPSh7alkgjQfpodPManVunhMFL2SYMz9W9HSoTWIxHYSkHMQC96E/E/r52Y8MpPmYwTA5JOF4UJxybCk9/jHlNADR9ZQqhi9lZMB0QRamxGc5Oo8FMdZotsNt5iEsukcV72KuXK/UWxejNLKYeO0QkqIQ9doiq6QzVURxQJ9IJe0Zvz7Lw7H87ntHTFmfUcoTk4X7+G/pk2</latexit>

P(end) = 0.001 0.023 0.002 0.001 0.078 0.735

softmax(
softmax(
<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>
<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>
) )
<latexit sha1_base64="8pv/+ZuGd1Uyqg6nzbMYaugyxmE=">AAACEnicbVC7TsMwFHV4lvIKZWSxqJDKUiUIFcYKFsYi9SU1oXJcp7HqOJHtIKKof8HMCt/Ahlj5AT6Bv8BNM9CWI1k6Oue+fLyYUaks69tYW9/Y3Nou7ZR39/YPDs2jSldGicCkgyMWib6HJGGUk46iipF+LAgKPUZ63uR25vceiZA04m2VxsQN0ZhTn2KktDQ0K047IAo9ZLUAOiqC6fl0aFatupUDrhK7IFVQoDU0f5xRhJOQcIUZknJgW7FyMyQUxYxMy04iSYzwBI3JQFOOQiLdLL99Cs+0MoJ+JPTjCubq344MhVKmoacrQ6QCuezNxP+8QaL8azejPE4U4Xi+yE8Y1J+cBQFHVBCsWKoJwoLqWyEOkEBY6bgWJuHQzaSfL9LZ2MtJrJLuRd1u1Bv3l9XmTZFSCZyAU1ADNrgCTXAHWqADMHgCL+AVvBnPxrvxYXzOS9eMoucYLMD4+gXvMZ3F</latexit>

(h!y )
⇥
…
Multi-head self-attention + feed forward

Multi-head self-attention + feed forward

[CLS] what are cilia for [SEP] comb jellies use … _lo com otion
78
fi
 N. Reimers and I. Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, EMNLP 2019.

SBERT: Sentence embeddings

■ Single [CLS] token is insuf cient for cramming the entire &%#@ sentence.
(Often worse than just averaging GloVe embeddings!)

■ Instead: Use two BERTs in a bi-encoder.

■ Train w/ softmax classi er. softmax(
softmax(
<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>
<latexit sha1_base64="/4jVRd01VWx7s0GMy/AAWIkSfBo=">AAAD43icfVJbb9MwFHabAaPcOnjkJVBV2hBUbTcJJIQ0AQ+8IIa0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u55HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgiit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2BB7FQLZ1RiBZnnuzj817M3ara6nW65wk3QW4EWWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ55BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFPP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ==</latexit>
) )
u v u-v
u concat v

mean mean
pool pool
Multi-head self-attention + feed forward Multi-head self-attention + feed forward
tied parameters
Multi-head self-attention + feed forward Multi-head self-attention + feed forward

what are cilia for comb jellies use … 79

fi
fi
 N. Reimers and I. Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, EMNLP 2019.

SBERT: Sentence embeddings

■ Single [CLS] token is insuf cient for cramming the entire &%#@ sentence.
(Often worse than just averaging GloVe embeddings!)

■ Instead: Use two BERTs in a bi-encoder.

■ Train w/ contrastive (triplet) loss. Minimize: max(||u-vp|| - ||u-vn|| + m, 0)
negative example
vn
u vp

mean mean
pool pool
Multi-head self-attention + feed forward
Multi-head self-attention + feed forward Multi-head self-attention + feed forward
tied parameters
Multi-head self-attention + feed forward
Multi-head self-attention + feed forward Multi-head self-attention + feed forward

what are cilia for comb jellies use … 80

fi
 Transformer LLMs
■ Encoder-only: (Ro)BERT(a), etc.
decoder
llegó la bruja verde

■ Encoder-decoder: T5, BART, … Masked self-attention + cross-attention + feed forward Layer K

■ Decoder-only: GPT, Llama, …

Masked self-attention + cross-attention + feed forward Layer 1

encoder

Layer J Multi-head self-attention + feed forward

Layer 1 Multi-head self-attention + feed forward

the green witch arrived 81

 response
Encoder-decoder LLMs
six people injured …
■ Same transformer encoder-decoder decoder
model you know and love, now with
different inputs and outputs. Masked self-attention + cross-attention + feed forward Layer K

Masked self-attention + cross-attention + feed forward Layer 1

encoder

Layer J Multi-head self-attention + feed forward

Layer 1 Multi-head self-attention + feed forward

summarize: severe ooding hit … prompt 82

fl
 Pretraining encoder-decoder LLMs
■ Autoregressive decoding is not very parallelizable & thus inef cient on
accelerators (GPUs, TPUs).

■ Further, what happens when the decoder makes bad predictions?

■ During training, use teacher forcing: provide correct tokens, not predicted, at
each decoder step.
error propagation & slow convergence

D . 1 2 3 4

Encoder Decoder

<s> D . 1 2 3
pre x: A B C

original: A B C D . E F .
83
fi
fi
 Pretraining encoder-decoder LLMs
■ Autoregressive decoding is not very parallelizable & thus inef cient on
accelerators (GPUs, TPUs).

■ Further, what happens when the decoder makes bad predictions?

■ During training, use teacher forcing: provide correct tokens, not predicted, at
each decoder step.

D . 1 F . </s>

Encoder Decoder

<s> D . E F .
pre x: A B C

original: A B C D . E F .
84
fi
fi
Decoder-only LLMs
■ Can we do even better than encoder-decoder to align training objective with
good zero-shot performance at test time?

■ Yes! Use just the decoder.

D . E F . </s>

Encoder Decoder

<s> D . E F .
A B C

85
Decoder-only LLMs
■ Can we do even better than encoder-decoder to align training objective with good
zero-shot performance at test time? Yes! Use just the decoder.
■ State-of-the-art for in-context learning, basis for all the latest & greatest LLMs.
■ Training: No denoising, pre xes necessary; Train on sequences using teacher forcing.
■ Inference: For input/prompt tokens, use teacher forcing & compute embeddings only.
Training Inference

A B C D . E F . </s> D . E F . </s>

Decoder: Masked MHA + FF Decoder: Masked MHA + FF

<s> A B C D . E F . <s> A B C D . E F .
86
fi
Keywords

• Feature representations.

• Input / output dimensionality.

• CNNs: Layers (norms, pooling, etc). 1D, 2D, 3D.

• Transformers: Self-attention, encoder vs decoder.

• Mapping, real valued control, …

Questions

• How much information does a pixel, a spectrogram, a word contain?

• How much do I care about parameter size vs model size?

• Does my task require a large vocabulary (why?), high resolution (why?), …

• What interactions are required in the input?

(Which pixels or words need to talk to each other?)