Introduction to Deep Learning and Generative AI · End Term · May 2026 · FN

**1. Recall the original minimax GAN generator loss:** In the original GAN formulation (Goodfellow et al., 2014), the generator \(G\) minimizes the saturating loss: \[L_G = \mathbb{E}_{z \sim p_z}\big[\log(1 - D(G(z)))\big]\] where \(D(x) \in (0, 1)\) is the discriminator's estimated probability that \(x\) is real. **2. Analyze what happens when \(D\) becomes nearly perfect early in training:** When the discriminator \(D\) easily distinguishes fake samples from real ones, it assigns \(D(G(z)) \approx 0\) to every generated sample \(G(z)\). In the region where \(D(G(z)) \approx 0\), the function \(\log(1 - D)\) is nearly flat (\(\log(1 - 0) = 0\)), and the derivative of the sigmoid output \(D(G(z))\) saturates near \(0\). Consequently, the gradient of \(\log(1 - D(G(z)))\) with respect to the generator's parameters **vanishes (becomes extremely small)**, causing the generator to stop learning! **Answer: E — The generator stops learning because it receives very small (vanishing) gradients.** A: **Incorrect:** When \(D(G(z)) \approx 0\), the saturating loss \(\log(1 - D(G(z)))\) has a near-zero slope, starving the generator of gradient signal. B: **Incorrect:** If the generator is still poor while the discriminator is perfect, the GAN has not converged to the Nash equilibrium (\(p_g = p_{\text{data}}\)). C: **Incorrect:** Although \(\log(1 - D(G(z))) \approx \log(1) = 0\), this reflects a completely failed generator (rejected with 100% confidence by \(D\)), not perfect image generation. D: **Incorrect:** Mode collapse occurs when \(G\) maps many latent codes to a single realistic output, whereas an overpowering \(D\) early in training causes vanishing gradients. E: **Correct:** When \(D(G(z)) \to 0\), the minimax objective \(\log(1 - D(G(z)))\) saturates in a flat region, producing vanishing gradients for the generator.

**1. Distinguish between Discriminative and Generative models:** Given input features \(x\) (images) and class labels \(y\) (cat vs. dog): - **Discriminative model** (e.g., Logistic Regression, ResNet classifier, SVM): Directly models the conditional probability distribution **\(P(y \mid x)\)** (or learns the decision boundary separating classes given \(x\)) without modeling how the inputs \(x\) are generated. - **Generative model** (e.g., Naive Bayes, GMM, class-conditional VAE/diffusion): Models the joint probability distribution **\(P(x, y) = P(x \mid y) P(y)\)** (or \(P(x)\) in unsupervised generation), allowing it both to generate new samples \(x\) and to compute \(P(y \mid x)\) via Bayes' rule. **Answer: B — The discriminative model learns P(y|x); the generative model learns P(x,y).** A: **Incorrect:** This swaps the two models: \(P(y \mid x)\) is learned by the discriminative model, while \(P(x)\) or \(P(x, y)\) is learned by the generative model. B: **Correct:** A discriminative classifier directly models the posterior \(P(y \mid x)\), whereas a supervised generative model learns the joint distribution \(P(x, y)\). C: **Incorrect:** Discriminative models do not model the distribution of inputs \(x\), so they do not learn \(P(x, y)\). D: **Incorrect:** Learning only a decision boundary is a property of discriminative models, not generative models.

**1. Understand how embedding-based retrieval works in Retrieval-Augmented Generation (RAG):** In a RAG pipeline, both document chunks and the user's query are passed through a dense **embedding model** (such as a bi-encoder transformer) that maps text into a continuous, high-dimensional **semantic vector space**. **2. Explain why different words with the same meaning still match:** Because the embedding model is trained so that paraphrases, synonyms, and conceptually related sentences map to **nearby points** in the vector space (measured by cosine similarity or inner product), retrieval depends on **semantic meaning** rather than exact keyword overlap (unlike lexical BM25/TF-IDF matching). **Answer: A — Embeddings represent semantic information in a vector space, allowing similarity to be measured beyond exact word matching.** A: **Correct:** Dense text embeddings map semantically similar phrases to close vectors in embedding space, enabling retrieval even when the surface words differ. B: **Incorrect:** Standard dense embedding retrieval encodes the query directly into a vector without requiring an LLM to extract keyword synonyms. C: **Incorrect:** FAISS is a vector similarity search library that compares numerical vectors; it does not rewrite text queries. D: **Incorrect:** In RAG, the vector index retrieves top-\(k\) chunks first, rather than having the LLM scan the entire raw corpus directly.

**1. Trace the tensor shapes in PyTorch step by step:** - `x = torch.tensor([1, 2, 3])` Creates a 1D tensor of 3 elements with shape `torch.Size([3])`. - `x.unsqueeze(0)` Inserts a new dimension of size `1` at dimension index `0`, changing the shape from `(3,)` to **`(1, 3)`**. - `y = torch.cat((x.unsqueeze(0), x.unsqueeze(0)), dim=0)` Concatenates two tensors of shape `(1, 3)` along dimension `0` (`dim=0`), adding their sizes along dimension `0` (\(1 + 1 = 2\)) while keeping dimension `1` equal to `3`. Thus, `y` has shape **`torch.Size([2, 3])`**. **Answer: A — torch.Size([2, 3]).** A: **Correct:** `x.unsqueeze(0)` has shape `(1, 3)`, and concatenating two `(1, 3)` tensors along `dim=0` produces shape `torch.Size([2, 3])`. B: **Incorrect:** `torch.Size([1, 6])` would result from concatenating two `(1, 3)` tensors along `dim=1`, not `dim=0`. C: **Incorrect:** `torch.Size([6])` would result from `torch.cat((x, x), dim=0)` without calling `.unsqueeze(0)`. D: **Incorrect:** Both tensors have matching shapes `(1, 3)`, so `torch.cat` executes without any error.

**1. Recall the receptive field recurrence formula for a CNN:** Let \(R_l\) be the receptive field size and \(j_l\) be the cumulative stride (jump) at layer \(l\), starting from the input layer with \(R_0 = 1\) and \(j_0 = 1\). For a layer with kernel size \(k_l\) and stride \(s_l\): \[R_l = R_{l-1} + (k_l - 1) \times j_{l-1}, \quad j_l = j_{l-1} \times s_l\] **2. Compute the receptive field after `Conv1` and `MaxPool`:** - **Layer 1 (`Conv1`: \(k_1 = 3,\; s_1 = 1\)):** \[R_1 = 1 + (3 - 1) \times 1 = 3, \quad j_1 = 1 \times 1 = 1\] (Each `Conv1` neuron sees a \(3 \times 3\) input patch.) - **Layer 2 (`MaxPool`: \(k_2 = 2,\; s_2 = 2\)):** \[R_2 = R_1 + (k_2 - 1) \times j_1 = 3 + (2 - 1) \times 1 = 4\] (Intuitively, a \(2 \times 2\) pooling window combines two adjacent \(3 \times 3\) `Conv1` windows shifted by stride 1 along each axis, covering \(3 + 1 = 4\) pixels in height and \(3 + 1 = 4\) pixels in width!) Thus, the receptive field after pooling is **4x4**. **Answer: C — 4x4.** A: **Incorrect:** `2x2` is only the kernel size of the MaxPool layer relative to the Conv1 feature map, not the input image. B: **Incorrect:** `3x3` is the receptive field after `Conv1` alone, before the `2x2` pooling layer. C: **Correct:** Combining a \(2 \times 2\) neighborhood of `Conv1` neurons (each viewing \(3 \times 3\) input pixels with stride 1) covers a \((3 + 1) \times (3 + 1) = 4 \times 4\) region of the input. D: **Incorrect:** A `5x5` receptive field would require a second `3x3` layer (\(3 + (3-1)\times 1 = 5\)), whereas MaxPool is `2x2`.

**1. Understand the output shapes of `torch.nn.RNN` in PyTorch:** We initialize: `rnn = nn.RNN(input_size=5, hidden_size=8, num_layers=3, batch_first=True)` and pass input `x = torch.randn(4, 7, 5)` where: - Batch size \(B = 4\) (first dimension because `batch_first=True`) - Sequence length \(L = 7\) - Input feature size \(H_{\text{in}} = 5\) **2. Check the shape of the final hidden state `h` in `output, h = rnn(x)`:** In PyTorch, even when `batch_first=True` is set (which makes `output` have shape `(B, L, H_out) = (4, 7, 8)`), the **hidden state tensor `h_n`** (`h`) returned by `nn.RNN` **always** has shape: \[(\text{num\_layers} \times \text{num\_directions},\; \text{batch\_size},\; \text{hidden\_size})\] Substituting `num_layers = 3` (unidirectional, so `num_directions = 1`), `batch_size = 4`, and `hidden_size = 8` gives: `torch.Size([3, 4, 8])`. **Answer: A — torch.Size([3, 4, 8]).** A: **Correct:** In PyTorch `nn.RNN`, the final hidden state `h` always has shape `(num_layers, batch_size, hidden_size) = (3, 4, 8)`, regardless of `batch_first=True`. B: **Incorrect:** Because `num_layers=3`, the first dimension of `h` is `3`, not `1`. C: **Incorrect:** `batch_first=True` only affects the input `x` and sequence `output` tensor, **not** the hidden state `h`, whose first dimension is always `num_layers` (`3`). D: **Incorrect:** `h` stores only the final time step's hidden state for each of the 4 batch items, so sequence length `7` does not appear in `h.shape`.

**1. Understand Teacher Forcing vs. Student Forcing (Autoregressive feeding):** During the training of a sequence-to-sequence decoder: - **Teacher Forcing:** Always feeds the **ground-truth target token** `target[t]` as the next input to the decoder, regardless of what the model predicted. - **Student Forcing (Free running):** Feeds the model's **own predicted token** `top1 = output.argmax(1)` as the next input to the decoder. **2. Analyze Code A and Code B:** - **Code A:** Unconditionally sets `input = target[t]`. Since it always feeds the ground-truth token, **Code A performs pure Teacher Forcing**. - **Code B:** Flips a fair coin `teacher_force = random.random() < 0.5` and sets `input = target[t] if teacher_force else top1`. With 50% probability it uses the ground-truth `target[t]` (Teacher Forcing), and with 50% probability it uses its own prediction `top1` (Student Forcing). Thus, **Code B performs a mixture of Teacher Forcing and Student Forcing** (scheduled sampling). **Answer: B — Code A -> Teacher Forcing; Code B -> A mixture of Teacher Forcing and Student Forcing.** A: **Incorrect:** Code B uses `target[t]` 50% of the time and `top1` 50% of the time, so it is a mixture rather than pure Student Forcing. B: **Correct:** Code A always feeds the ground-truth `target[t]` (Teacher Forcing), while Code B randomly alternates between `target[t]` and the model's prediction `top1`. C: **Incorrect:** Code A feeds `target[t]` (Teacher Forcing), not `top1`. D: **Incorrect:** Neither snippet is pure Student Forcing. E: **Incorrect:** Code B feeds `top1` whenever `teacher_force` is `False`.

**1. Understand how causal / padding masks must be applied in Self-Attention:** In masked self-attention: \[\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + M\right) V\] To prevent a token from attending to masked positions (`mask == 0`) while ensuring that the remaining valid attention weights in each row still **sum to 1.0**: - We must set the masked logits (`scores`) to **\(-\infty\)** (or a large negative number like `-1e9`) **BEFORE** applying `softmax`. - Because \(e^{-10^9} \approx 0\), `softmax` assigns **0** weight to masked positions and properly normalizes the unmasked positions so their probabilities sum to `1`. **2. Compare Student A and Student B:** - **Student A:** Applies `softmax` *first*, and then zeroes out `mask == 0` afterward (`weights.masked_fill(mask == 0, 0)`). Zeroing out entries *after* `softmax` means the remaining attention weights in a row **no longer sum to 1** (and during backpropagation, masked keys still affected the softmax denominator!). Thus, Student A is **incorrect**. - **Student B:** Masks `scores` with `-1e9` **before** calling `torch.softmax(scores, dim=-1)`. Thus, **only Student B is correct**. **Answer: B — Only Student B.** A: **Incorrect:** Zeroing out probabilities *after* `softmax` breaks the probability normalization (the remaining weights no longer sum to 1) and leaks masked scores into the softmax denominator. B: **Correct:** Student B sets masked attention logits to `-1e9` *before* `softmax`, so `exp(-1e9) -> 0` and the unmasked probabilities properly sum to 1. C: **Incorrect:** Student A's post-softmax masking does not produce normalized attention distributions. D: **Incorrect:** Student B's pre-softmax masking with `-1e9` is the standard PyTorch implementation of masked attention.

**1. Recall the attention blocks inside a standard Transformer Encoder and Decoder:** In the original Transformer architecture (Vaswani et al., 2017): - **Encoder layer:** Contains **bidirectional (unmasked) Self-Attention**, allowing every input token to attend to all tokens in the input sequence. - **Decoder layer:** Contains two attention sub-layers: 1. **Masked (Causal) Self-Attention:** Prevents each target position from attending to future target positions during autoregressive generation. 2. **Cross-Attention (Encoder-Decoder Attention):** Queries come from the decoder's previous sub-layer, while Keys and Values come from the **encoder's output**, allowing the decoder to attend to the encoded source sequence. **Answer: E — Encoder: Self-Attention; Decoder: Masked Self-Attention + Cross-Attention.** A: **Incorrect:** The encoder uses unmasked (bidirectional) self-attention, while the decoder uses *masked* self-attention. B: **Incorrect:** The encoder only attends to the source sequence itself (self-attention), not cross-attention. C: **Incorrect:** Cross-attention is located in the decoder (to attend to encoder outputs), not in the encoder. D: **Incorrect:** The decoder also requires a Masked Self-Attention layer in addition to Cross-Attention. E: **Correct:** The Encoder uses bidirectional Self-Attention, and the Decoder uses both Masked Self-Attention (for causal target modeling) and Cross-Attention (over the encoder representations).

**1. Understand the two terms in the Variational Autoencoder (VAE) ELBO loss:** The VAE loss function is: \[\mathcal{L}_{\text{VAE}} = \mathcal{L}_{\text{recon}}(x, \hat{x}) + D_{\text{KL}}\big(q(z \mid x) \,\|\, p(z)\big)\] 1. **First term (\(\mathcal{L}_{\text{recon}}\)):** Reconstruction loss, which encourages the decoder to accurately reconstruct the input \(x\) from its latent code \(z \sim q(z \mid x)\). 2. **Second term (\(D_{\text{KL}}(q(z \mid x) \,\|\, p(z))\)):** The Kullback-Leibler (KL) divergence between the encoder's approximate posterior distribution \(q(z \mid x)\) and the prior distribution \(p(z) = \mathcal{N}(0, I)\). **2. Identify the role of the KL divergence term:** The KL term acts as a **regularizer on the latent space**, forcing the encoded distributions \(q(z \mid x)\) to stay close to the smooth prior \(p(z) = \mathcal{N}(0, I)\) so that the model does not simply memorize isolated points and we can generate coherent new samples by sampling \(z \sim p(z)\). **Answer: B — Regularize the latent space to stay close to the prior p(z).** A: **Incorrect:** Reconstruction accuracy is enforced by the first term \(\mathcal{L}_{\text{recon}}(x, \hat{x})\); the KL term actually trades off against exact reconstruction. B: **Correct:** The KL divergence \(D_{\text{KL}}(q(z|x) \| p(z))\) regularizes the encoder's latent distribution to match the prior \(p(z)\), ensuring a smooth, continuous latent space. C: **Incorrect:** VAEs are known to produce slightly blurrier images than GANs; the KL term regularizes latent distributions rather than sharpening pixels. D: **Incorrect:** Classifying real vs. fake images is the role of the discriminator in a GAN, not a VAE.

**1. Understand why Scaled Dot-Product Attention divides by \(\sqrt{d_k}\):** In self-attention, if the components of query \(q\) and key \(k\) (of dimension \(d_k = \text{head\_size}\)) have zero mean and unit variance, then their dot product \(q \cdot k = \sum_{i=1}^{d_k} q_i k_i\) has mean \(0\) and variance **\(d_k\)** (standard deviation \(\sqrt{d_k}\)). **2. Analyze what happens when `weights /= math.sqrt(d_k)` is removed:** As \(d_k\) increases, the unscaled dot products \(q \cdot k\) grow large in magnitude. When large positive and negative numbers are passed into `softmax`, the largest value dominates exponentially, pushing the `softmax` output into **saturated regions** (nearly a one-hot `[1, 0, ..., 0]` distribution) where the Jacobian/gradients of `softmax` become **extremely small (vanishing gradients)**, making training difficult. **Answer: E — The dot products tend to grow larger in magnitude, pushing softmax into saturated regions where the output is nearly one-hot and gradients through the attention weights become very small, making learning harder.** A: **Incorrect:** Softmax is NOT scale-invariant; multiplying logits by a large factor makes the output distribution much sharper (closer to one-hot) and shrinks its gradients. B: **Incorrect:** Causal masking (`-inf`) still zeroes out future positions regardless of whether the remaining finite logits are scaled. C: **Incorrect:** Dividing by a scalar \(\sqrt{d_k}\) does not change tensor shapes, so removing it causes no shape mismatch. D: **Incorrect:** Softmax mathematically always outputs values in \([0, 1]\) that sum to \(1.0\); large logits cause gradient saturation, not probabilities > 1. E: **Correct:** Without dividing by \(\sqrt{d_k}\), the variance of \(q \cdot k\) grows as \(d_k\), producing large logits that saturate `softmax` into nearly one-hot outputs with vanishing gradients.

**1. Understand autoregressive next-token prediction data preparation:** We are given a sequence of 5 tokens: `tokens = [10, 25, 7, 19, 4]` In next-token prediction, at each position \(t\), the input \(x_t\) is the current token and the target \(y_t\) is the **immediately following token** in the sequence: - Input `x`: all tokens except the last one \(\implies\) `tokens[:-1] = [10, 25, 7, 19]` - Target `y`: all tokens except the first one \(\implies\) `tokens[1:] = [25, 7, 19, 4]` Thus, `10 -> 25`, `25 -> 7`, `7 -> 19`, and `19 -> 4`. **Answer: B — x = tokens[:-1], y = tokens[1:].** A: **Incorrect:** `x = tokens[1:]` and `y = tokens[:-1]` shifts in reverse, predicting the *previous* token instead of the *next* token. B: **Correct:** `tokens[:-1]` takes the first \(N-1\) tokens as inputs and `tokens[1:]` takes the corresponding next tokens (indices `1` to `N-1`) as targets. C: **Incorrect:** `tokens[:]` has length 5 while `tokens[1:]` has length 4, causing a length mismatch and misaligned pairs. D: **Incorrect:** `tokens[:-1]` has length 4 while `tokens[:]` has length 5.

**1. Calculate the per-head dimension \(d_k\):** We are given: - Total embedding dimension: \(d_{\text{model}} = 20\) - Number of attention heads: \(h = 2\) - Sequence length: \(L = 8\) tokens In multi-head attention, the embedding dimension is split equally across the \(h\) heads: \[d_k = \frac{d_{\text{model}}}{h} = \frac{20}{2} = 10\] **2. Determine the shapes of `query`, `key`, and `scores` for a single attention head:** - **`query` matrix \(Q\):** For \(L = 8\) tokens and head dimension \(d_k = 10\), shape is **`(8, 10)`**. - **`key` matrix \(K\):** For \(L = 8\) tokens and head dimension \(d_k = 10\), shape is **`(8, 10)`**. - **Attention `scores` matrix \(Q K^T\):** Multiplying \(Q\) of shape `(8, 10)` by \(K^T\) of shape `(10, 8)` produces an \(L \times L\) pairwise token-to-token score matrix of shape **`(8, 8)`**. **Answer: E — query: (8, 10), key: (8, 10), scores: (8, 8).** A: **Incorrect:** `(8, 20)` is the full embedding before splitting across the 2 heads, and `scores` is \(L \times L = (8, 8)\), not `(20, 20)`. B: **Incorrect:** `query` has shape `(seq_len, head_dim) = (8, 10)`, not `(10, 8)`. C: **Incorrect:** Both `query` and `key` have shape `(8, 10)`, and `scores` is `(8, 8)`. D: **Incorrect:** For a single head, `key` also has dimension \(d_k = 20 / 2 = 10\), not `20`. E: **Correct:** With \(L = 8\) tokens and per-head dimension \(d_k = 20 / 2 = 10\), both `query` and `key` are `(8, 10)`, and `scores = Q @ K.T` is `(8, 8)`.

**1. Recall the computational complexity of Transformer Self-Attention with respect to sequence length \(n\):** In standard self-attention, every token attends to every token in the sequence of length \(n\). Computing the \(n \times n\) attention matrix \(Q K^T\) and multiplying by \(V\) takes time proportional to the square of the sequence length: \[\text{Cost} \propto n^2 \cdot d\] **2. Calculate the factor change when \(n\) is doubled from 128 to 256:** Since \(n_{\text{new}} = 2 n_{\text{old}}\): \[\frac{\text{Cost}_{\text{new}}}{\text{Cost}_{\text{old}}} = \left(\frac{256}{128}\right)^2 = 2^2 = 4\] Thus, doubling the sequence length **quadruples** the computational cost of self-attention. **Answer: C — Quadruples.** A: **Incorrect:** Self-attention scales quadratically \(O(n^2)\) with sequence length, not linearly \(O(n)\). B: **Incorrect:** Doubling \(n\) multiplies \(n^2\) by \(2^2 = 4\), not \(3\). C: **Correct:** Because self-attention computes pairwise interactions across all \(n \times n\) token pairs, doubling \(n\) from 128 to 256 multiplies the cost by \(2^2 = 4\) (quadruples). D: **Incorrect:** Self-attention cost depends directly on sequence length \(n\) via \(O(n^2 d)\).

**1. Understand how `ReduceLROnPlateau(optimizer, factor=0.1, patience=2)` works in PyTorch:** With `mode='min'` (default) and `patience=2`, `ReduceLROnPlateau` tracks the best (minimum) loss seen so far and counts the number of consecutive epochs with no improvement (`num_bad_epochs`): - Whenever a new strictly lower loss is observed, `best` is updated and `num_bad_epochs` resets to `0`. - When a loss is not better than `best`, `num_bad_epochs` increments by `1`. - The learning rate is multiplied by `factor=0.1` only when `num_bad_epochs > patience` (i.e., when `num_bad_epochs` reaches `patience + 1 = 3` bad epochs!). **2. Trace the loop over `losses = [4, 3, 2, 2, 2, 2]` with initial `lr = 0.01`:** 1. **Step 1 (`loss = 4`):** Initial best \(= 4\), `num_bad_epochs = 0` \(\implies\) prints **`0.01`**. 2. **Step 2 (`loss = 3`):** Improved! New best \(= 3\), `num_bad_epochs = 0` \(\implies\) prints **`0.01`**. 3. **Step 3 (`loss = 2`):** Improved! New best \(= 2\), `num_bad_epochs = 0` \(\implies\) prints **`0.01`**. 4. **Step 4 (`loss = 2`):** Not `< 2` (1st bad epoch: `num_bad_epochs = 1 <= 2`) \(\implies\) prints **`0.01`**. 5. **Step 5 (`loss = 2`):** Not `< 2` (2nd bad epoch: `num_bad_epochs = 2 <= 2`) \(\implies\) prints **`0.01`**. 6. **Step 6 (`loss = 2`):** Not `< 2` (3rd bad epoch: `num_bad_epochs = 3 > 2`, patience exceeded!) \(\implies\) learning rate is reduced to \(0.01 \times 0.1 = 0.001\), and prints **`0.001`**. Thus, the printed sequence is five `0.01`s followed by one `0.001`. **Answer: B — 0.01, 0.01, 0.01, 0.01, 0.01, 0.001.** A: **Incorrect:** At the 4th step (`loss = 2`), it is only the 1st epoch without improvement after reaching `2` at step 3, whereas `patience=2` waits for 2 bad epochs before reducing on the 3rd bad epoch. B: **Correct:** Losses improve through step 3 (`4 -> 3 -> 2`), then stagnate at `2` for steps 4 and 5 (2 bad epochs tolerated by `patience=2`), and trigger the reduction to `0.001` on step 6. C: **Incorrect:** The loss is actively decreasing on steps 2 and 3, so the learning rate is not reduced early. D: **Incorrect:** The learning rate is reduced only once (at step 6), not on step 5 and step 6.

**1. Evaluate each statement about activation functions:** - **Statement A:** The sigmoid function \(\sigma(x) = \frac{1}{1+e^{-x}}\) has a maximum derivative of \(0.25\) and saturates at \(0\) and \(1\) for large \(|x|\), which **causes** (rather than avoids) the **vanishing gradient problem** in deep networks. (Incorrect.) - **Statement B:** \(\text{ReLU}(x) = \max(0, x)\) is a **piecewise linear, non-linear function** (its slope changes from \(0\) for \(x < 0\) to \(1\) for \(x > 0\)), which is precisely how it introduces non-linearity into deep neural networks! (Incorrect.) - **Statement C:** The output of **sigmoid** lies strictly in \((0, 1)\) (always positive, so **not zero-centered**), whereas the output of **tanh** lies in \((-1, 1)\) with \(\tanh(0) = 0\) (**zero-centered**). (Correct!) - **Statement D:** For all \(x \in \mathbb{R}\), the derivative of sigmoid is \(\sigma'(x) = \sigma(x)(1 - \sigma(x)) > 0\) and the derivative of tanh is \(\tanh'(x) = 1 - \tanh^2(x) > 0\). Thus, **both derivatives are always non-negative** (never negative). (Correct!) **Answer: C, D — Sigmoid is not zero-centered whereas tanh is zero-centered, and the derivatives of sigmoid and tanh are never negative.** A: **Incorrect:** Sigmoid suffers heavily from vanishing gradients because its derivative is at most \(0.25\) and approaches \(0\) in both tails. B: **Incorrect:** \(\text{ReLU}(x) = \max(0, x)\) is non-linear over \(\mathbb{R}\) because it bends at \(x = 0\) (violating \(f(a+b) = f(a)+f(b)\)). C: **Correct:** Sigmoid outputs values in \((0, 1)\) (not zero-centered), whereas \(\tanh(x)\) outputs values symmetrically in \((-1, 1)\) (zero-centered). D: **Correct:** Both \(\sigma(x)\) and \(\tanh(x)\) are strictly increasing functions on \(\mathbb{R}\), so their derivatives \(\sigma(1-\sigma)\) and \(1 - \tanh^2\) are never negative.

**1. Evaluate each statement about Transfer Learning:** - **Statement A:** By reusing rich feature representations learned from a large source dataset, transfer learning significantly **reduces the amount of labeled training data required** on the target task. (Correct!) - **Statement B:** **Freezing layers** (e.g., setting `param.requires_grad = False` in PyTorch) stops gradients from updating those layers' weights during backpropagation, keeping the pre-trained weights fixed. (Correct!) - **Statement C:** Transfer learning is widely used across **NLP** (BERT, GPT), **audio/speech**, **reinforcement learning**, and **computer vision** (detection, segmentation, classification)—not just image classification. (Incorrect.) - **Statement D:** When the source domain is unrelated to the target domain (**negative transfer**) or abundant target data is available, a pre-trained model is not guaranteed to always beat training from scratch. (Incorrect.) **Answer: A, B — Transfer learning reduces the amount of labeled data required, and freezing layers prevents their weights from being updated during backpropagation.** A: **Correct:** Pre-trained representations allow models to generalize well on downstream tasks even with small labeled datasets. B: **Correct:** Freezing a layer disables weight updates for its parameters during gradient descent. C: **Incorrect:** Transfer learning is foundational in NLP (e.g., fine-tuning pre-trained language models), audio, and many other domains. D: **Incorrect:** Domain mismatch (negative transfer) can cause a pre-trained model to underperform a model trained from scratch.

**1. Check how each change affects the number of trainable parameters in a Transformer:** - **Option A (Increasing vocabulary size \(V\)):** The token embedding matrix has shape \(V \times d_{\text{model}}\) (and the output language-modeling head has \(d_{\text{model}} \times V\) parameters). Increasing \(V\) directly adds new trainable embedding rows, **increasing the trainable parameters**. (Correct!) - **Option B (Using fixed sinusoidal positional embeddings):** Fixed sinusoidal positional encodings are precomputed deterministic sine/cosine values (`requires_grad=False`) and have **zero trainable parameters**. (Incorrect.) - **Option C (Increasing the number of layers \(L\)):** Each Transformer layer contains its own trainable attention projection matrices (\(W_Q, W_K, W_V, W_O\)), feed-forward network weights, and layer normalization parameters. Adding more layers directly **increases the trainable parameters**. (Correct!) **Answer: A, C — Increasing vocabulary size and increasing the number of layers.** A: **Correct:** Increasing the vocabulary size \(V\) adds rows to the trainable token embedding matrix (\(V \times d_{\text{model}}\)). B: **Incorrect:** Fixed sinusoidal positional encodings are non-trainable constants and add zero trainable parameters. C: **Correct:** Each additional Transformer block adds its own set of trainable self-attention and MLP weights. D: **Incorrect:** Both A and C increase the number of trainable parameters.

**1. Recall the architecture and pre-training objectives of BERT (Bidirectional Encoder Representations from Transformers):** - **Architecture:** BERT is an **Encoder-Only** Transformer model built from stacked bidirectional **Self-Attention layers** (Statement C is **correct**; Statement A is false because BERT is not an Encoder-Decoder model). - **Attention directionality:** BERT computes attention **bidirectionally** across both left and right context simultaneously, not unidirectionally (Statement D is false). - **Pre-training objectives:** BERT was pre-trained using two self-supervised tasks: 1. **Masked Language Modeling (MLM)** (not Causal Language Modeling, so Statement B is false). 2. **Next Sentence Prediction (NSP)** (Statement E is **correct**!). **Answer: C, E — Self Attention layers are present, and Next Sentence Prediction is one of the training objectives.** A: **Incorrect:** While BERT uses Masked Language Modeling, it has an **Encoder-Only** architecture, not an Encoder-Decoder architecture. B: **Incorrect:** BERT uses Masked Language Modeling (MLM), whereas Causal Language Modeling (CLM) is used by autoregressive Decoder-Only models like GPT. C: **Correct:** BERT consists of a stack of Transformer encoder blocks powered by multi-head self-attention layers. D: **Incorrect:** BERT's self-attention is **bidirectional** (each token attends to both left and right context), not unidirectional. E: **Correct:** Original BERT pre-training combines Masked Language Modeling (MLM) with Next Sentence Prediction (NSP).

**1. Analyze the three L1-ball (diamond) sets in \(\mathbb{R}^2\):** Recall that the L1 norm \(\|(x, y)\|_1 = |x| + |y|\) is a convex function, and every sublevel set of a convex function is a **convex set**: - \(F = \{(x, y) : |x| + |y| \le 1\}\) is the closed L1 diamond of radius \(1\) centered at \((0, 0)\) (**convex**). - \(G = \{(x, y) : |x - 1| + |y| \le 1\}\) is the closed L1 diamond of radius \(1\) centered at \((1, 0)\) (**convex**). - \(H = \{(x, y) : |x| + |y| \le 0.5\}\) is the closed L1 diamond of radius \(0.5\) centered at \((0, 0)\) (**convex**). **2. Check each statement:** - **Option A:** While \(F\) is convex, the boundary set \(\{(x, y) : |x| + |y| = 1\}\) is a hollow diamond loop and is **not convex** (e.g., \((-1, 0)\) and \((1, 0)\) are on the boundary, but their midpoint \((0, 0)\) has \(|0|+|0| = 0 \neq 1\)). (**Incorrect.**) - **Option B:** \(H\) is convex (an L1 ball), and since the intersection of any two convex sets is always convex, \(F \cap G\) is also **convex**! (**Correct!**) - **Option C:** Notice that \(H \subseteq F\) (since \(|x| + |y| \le 0.5 \implies |x| + |y| \le 1\)). Therefore, \(F \cup H = F\), which **is a convex set**! (**Incorrect.**) - **Option D:** Let us find \(F \cap G\) explicitly. For any \((x, y) \in F \cap G\), we have \(0 \le x \le 1\) (since if \(x < 0\), \(|x-1| = 1-x > 1\), and if \(x > 1\), \(|x| > 1\)). For \(x \in [0, 1]\), notice that: \[|x - 0.5| + |y| \le 0.5 \iff \max(x - 0.5,\; 0.5 - x) + |y| \le 0.5 \iff \max(x + |y|,\; 1 - x + |y|) \le 1 \iff |x| + |y| \le 1 \text{ and } |x - 1| + |y| \le 1\] Thus, \(F \cap G = \{(x, y) : |x - 0.5| + |y| \le 0.5\}\), which is the L1 diamond of radius \(0.5\) centered at \((0.5, 0)\) and is convex! (**Correct!**) **Answer: B, D — H and F cap G are convex, and {(x, y) : |x - 0.5| + |y| <= 0.5} is convex and equals F cap G.** A: **Incorrect:** The boundary \(\{(x, y) : |x| + |y| = 1\}\) is hollow and does not contain the midpoint \((0, 0)\) between \((-1, 0)\) and \((1, 0)\), so it is not convex. B: **Correct:** \(H\) is a norm ball (convex), and the intersection \(F \cap G\) of two convex norm balls is always convex. C: **Incorrect:** Since \(H \subseteq F\), the union \(F \cup H\) is simply \(F\), which is convex. D: **Correct:** For \(x \in [0, 1]\), requiring both \(x + |y| \le 1\) and \((1 - x) + |y| \le 1\) is algebraically identical to \(|x - 0.5| + |y| \le 0.5\).

**1. Calculate the patch dimensions and number of patches in the Vision Transformer (ViT):** We are given: - Input RGB image size: \(H \times W \times C = 96 \times 96 \times 3\) - Patch size: \(P \times P = 8 \times 8\) (non-overlapping) - Embedding dimension: \(D = 128\) 1. **Number of patches \(N\):** \[N = \left(\frac{96}{8}\right) \times \left(\frac{96}{8}\right) = 12 \times 12 = 144 \text{ patches}\] (So Option A claiming `156` is false.) 2. **Flattened length of each RGB patch:** Since the image is colourful (RGB, \(C = 3\) channels), each \(8 \times 8\) patch contains: \[P^2 \cdot C = 8 \times 8 \times 3 = 192 \text{ values}\] (So Option D claiming length `64` ignores the 3 RGB channels and is false.) **2. Calculate the number of weights in the shared linear projection layer (ignoring biases):** In ViT, a **single shared linear projection matrix** \(E \in \mathbb{R}^{(P^2 C) \times D}\) is applied to every patch vector of length \(192\) to map it to dimension \(D = 128\): \[\text{Number of weights} = 192 \times 128 = 24,576\] Because the linear projection layer is **shared across all 144 patches**, the parameter count does *not* get multiplied by the number of patches. **Answer: B — The projection layer contains 24,576 weights across all patches.** A: **Incorrect:** A \(96 \times 96\) image divided into \(8 \times 8\) patches yields \(12 \times 12 = 144\) patches, not `156`. B: **Correct:** Each flattened \(8 \times 8 \times 3\) RGB patch has dimension \(192\), and the single shared linear projection to dimension \(128\) has \(192 \times 128 = 24,576\) weights. C: **Incorrect:** The linear projection layer is shared across all 144 patches, so its weight count is \(24,576\), not \(144 \times 24,576 = 3,538,944\). D: **Incorrect:** Because the image has 3 RGB channels, each \(8 \times 8\) patch flattens to \(8 \times 8 \times 3 = 192\) values, not `64`. E: **Incorrect:** \(8,192 = 64 \times 128\) forgets to multiply by the 3 color channels. F: **Incorrect:** \(18,432\) does not match \(192 \times 128 = 24,576\).

**1. Recall the formula for the Inception Score (IS):** The Inception Score is defined as: \[\text{IS} = \exp\Big( \mathbb{E}_{x \sim p_g} \big[ D_{\text{KL}}(p(y \mid x) \,\|\, p(y)) \big] \Big)\] **2. Evaluate the KL divergence when \(p(y \mid x) = p(y)\) for every generated image \(x\):** A fundamental property of the Kullback-Leibler (KL) divergence is that \(D_{\text{KL}}(P \,\|\, Q) = 0\) whenever the two distributions \(P\) and \(Q\) are identical. Here, since \(p(y \mid x) = p(y)\) for every \(x\): \[D_{\text{KL}}(p(y \mid x) \,\|\, p(y)) = 0 \implies \mathbb{E}_{x}[0] = 0\] **3. Compute the resulting Inception Score:** \[\text{IS} = \exp(0) = e^0 = 1\] **Answer: 1.**

**1. Find the embedding dimension \(d_{\text{model}}\) from the positional embedding matrix:** In an LLM with trainable positional embeddings and context length \(L = 20\), the positional embedding matrix has shape \(L \times d_{\text{model}} = 20 \times d_{\text{model}}\). We are given that the positional embedding matrix contains \(1000\) parameters: \[20 \times d_{\text{model}} = 1000 \implies d_{\text{model}} = \frac{1000}{20} = 50\] **2. Calculate the number of parameters in the word (token) embedding matrix:** The word embedding matrix maps each of the \(V = 64\) vocabulary tokens to a vector of the exact same embedding dimension \(d_{\text{model}} = 50\), so its shape is \(V \times d_{\text{model}} = 64 \times 50\): \[\text{Word embedding parameters} = 64 \times 50 = 3200\] **Answer: 3200.**