A GPT model is trained using **causal language modeling**. During training, for a sequence of T = 4
tokens, which of the following correctly represents the **attention mask matrix** applied to the
attention logits?
A.B.C.D.
A published solution is not available for this question yet.
In Transformer models that use relative position embeddings (such as in Transformer-XL or T5),
clipping is often applied to the relative position indices. Which of the following statements about
clipping in relative position embeddings are correct?
A.
Clipping ensures that very large relative distances are mapped to a fixed
maximum distance.
B.
Without clipping, the model would require embeddings for every possible
relative distance, which is infeasible for long sequences.
C.
Clipping increases the precision of embeddings for large relative distances.
D.
Clipping introduces an upper bound k such that all distances greater than k
are mapped to the same index.
A published solution is not available for this question yet.
For the input “you enjoy tea often”, compute the final representation of the word “enjoy” after the
attention layer (i.e., after applying WQ, WK, WV , attention weights, and WO). Enter the sum of the
elements in the resulting vector.
A published solution is not available for this question yet.
Suppose the input sentence is “tea you enjoy often”. Using the same matrices and processing
method, what is the attention score (i.e., softmax entry from A) for the query word “tea” attending
to key word “often”?
A published solution is not available for this question yet.