Large Language Models · End Term · May 2025 · FN

Which of the following statements about subword tokenizers (e.g., Byte Pair Encoding, SentencePiece) is **true**?

A GPT-based model is fine-tuned for story generation. During evaluation, you test different decoding strategies on the same prompt: 1. **Method A:** Every run produces the exact same story, even when executed multiple times. 2. **Method B:** The story changes slightly in each run, but overall structure and key events remain similar. 3. **Method C:** The story is wildly different each time, sometimes even changing the genre. Based on the observations, select the most likely decoding strategies for each method:

Which of the following are valid design choices when building a transformer-based language model?

Which of the following statements about Top-k and Nucleus (Top-p) sampling are correct?

Which of the following preprocessing steps are commonly required when preparing data for transformer models like BERT?

For the input “you enjoy tea often”, compute the final representation of the word “tea” after the attention layer (i.e., after applying WQ, WK, WV , attention weights, and WO). Enter the sum of the elements in the resulting vector.

Suppose the input sentence is “tea you enjoy often”. Using the same matrices and processing method, what is the attention score (i.e., softmax entry from A) for the query word “tea” attending to key word “often”?

What percentage of entries in the attention matrix are zero (i.e., the sparsity)?

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.

Based on the above data, answer the given subquestions.