Large Language Models · End Term · Jan 2024 · FN

BERT stands for

The statement that, in general, position information can also be injected into the attention layers of a transformer model instead of the embedding of input tokens is

Suppose a GPT model is pre-trained using the Causal Language Modelling (CLM) objective and fine-tuned on a text summarization task. The context length T of the model was kept at 1024 during both pre-training and fine-tuning. Assume that the model uses Absolute Position Embedding (APE) with **learnable** parameters. Based on the above data, answer the given subquestions.

A user runs inference on the model N times with an input text that contains 712 tokens (including the special [start] token), then which of the following could be the possible length of summarized text

Suppose a GPT model is pre-trained using the Causal Language Modelling (CLM) objective and fine-tuned on a text summarization task. The context length T of the model was kept at 1024 during both pre-training and fine-tuning. Assume that the model uses Absolute Position Embedding (APE) with **learnable** parameters. Based on the above data, answer the given subquestions.

A user runs inference on the model N times with an input text that contains 712 tokens (including the special [start] token). However, the user observed that in each run the model generated the exact same summary. Based on this observation, select the decoding strategy that the model might be using

Based on the above data, answer the given subquestions.

Suppose the input word to the tokenizer is “SomeThing”. Assume that input text is normalized before applying pre-tokenization. Enter the number of tokens returned by the tokenizer. Ignore the special token used to represent the boundary of the word/token.

Based on the above data, answer the given subquestions.

Suppose the input word to the tokenizer is “1000000”.Enter the number of tokens returned by the tokenizer.

Based on the above data, answer the given subquestions.

Suppose the number of encoder layers N = 3. What is the total number of learnable parameters, including the parameters of the input and position embedding layers, in the model? Enter the final answer in thousands. For example, if the answer is 12,238, then enter 12 as the answer.

Based on the above data, answer the given subquestions.

Suppose we increase the context length from T = 256 to T = 512 keeping number of layers same (N = 3). How many parameters does the model need to learn **additionally** compared to the previous setting?

Compute the representation for the word “coffee” and enter the sum of the elements in it.

Suppose the input sentence is “coffee i like much“. What is the attention score between the word “coffee” with “much”?

Based on the above data, answer the given subquestions.

What is the input sequence length that would have generated this matrix?

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.