Deep Learning · End Term · May 2025 · AN

**1. Separate memorization from prediction:** Zero training loss only says that the fitted model reproduces its training examples under the chosen loss. A highly flexible model can memorize those examples and still make poor predictions elsewhere. **2. Explain the regularization tradeoff:** A penalty or other restriction can prevent that exact memorization, increasing training loss while improving generalization. For example, a smoother fit may ignore training noise that should not be extrapolated. **3. Evaluate both statements:** Statement 1’s guarantee is false; Statement 2 correctly says the favorable tradeoff may occur, rather than promising it for every penalty strength. **Answer: D — Statement 1 is false, but Statement 2 is true..** A: **Incorrect:** The claimed guarantee in Statement 1 is false. B: **Incorrect:** Training success does not guarantee unseen-data performance, while the second tradeoff is possible. C: **Incorrect:** The listed option D gives the correct truth values. D: **Correct:** The first is false and the second is true for the reasons stated.

**1. Describe layerwise pretraining:** Train one layer at a time to learn structure in unlabeled inputs, then use its representations as the next layer’s input. This gives each shallow training problem a relatively short gradient path. **2. Connect to supervised fine-tuning:** Stacking those learned layers supplies a data-informed initialization. Fine-tuning starts from useful intermediate representations instead of requiring a very deep, randomly initialized network to discover all of them through weak end-to-end signals. **3. State the scope:** This can alleviate optimization difficulty; it does not mathematically guarantee nonvanishing gradients in every later network. It does not require adding temporary skip connections or replacing activation functions. **Answer: A — It allows the network to learn a better representation of the data in each layer, which leads to better-.** A: **Correct:** It supplies useful representations and a better initialization for subsequent supervised training. B: **Incorrect:** Temporary skip connections are not the defining mechanism of layerwise pretraining. C: **Incorrect:** Changing sigmoid to ReLU is a separate architectural choice. D: **Incorrect:** Its principal role is useful feature initialization, not simply shrinking weights to prevent explosion.

**1. Identify when a sample can update a weight:** For linear prediction and squared loss, \[\frac{\partial L_i}{\partial w_j}=2(\hat y_i-y_i)x_{ij}.\] A zero feature guarantees zero data gradient for that sample, irrespective of its residual. **2. Count nonzero entries in each feature column:** Across the four samples, columns w₀,w₁,w₂,w₃,w₄ are nonzero 4,3,1,2,1 times respectively. The bias column is all ones; w₂ appears only in sample 2 and w₄ only in sample 3. **3. Choose the sparsest update opportunities:** The minimum count is one, attained by w₂ and w₄. These weights are structurally reached by the fewest one-example updates. **Answer: C, E — w2; w4.** **Convention note:** This is the intended feature-sparsity count, assuming generic nonzero residuals at the relevant updates and no extra momentum or weight decay. An exact zero residual can suppress an otherwise possible update; the unspecified learning rate does not determine such accidental cancellations. A: **Incorrect:** Its bias feature is nonzero in all four samples. B: **Incorrect:** Its feature is nonzero in three samples. C: **Correct:** Its feature is nonzero only in the second sample. D: **Incorrect:** Its feature is nonzero in two samples. E: **Correct:** Its feature is nonzero only in the third sample.

**1. Check whether the final ReLU is active:** Using stride-1 unflipped CNN cross-correlation, the given image and kernel yield \(A=\begin{pmatrix}3&-1\\4&4\end{pmatrix}\). Averaging its four entries gives B=(3−1+4+4)/4=2.5>0. ReLU therefore has derivative 1 here. **2. Differentiate the shared center weight:** The input values under the kernel center in the four windows are 1,−1,2,1. Their average is \[\frac{\partial B}{\partial K_{11}}=\frac{1-1+2+1}{4}=\frac34.\] **3. Apply the supplied upstream derivative:** With ∂L/∂ŷ=2, \[\frac{\partial L}{\partial K_{11}}=2\cdot1\cdot\frac34=1.50.\] All four windows contribute because they share the same center weight. **Answer: 1.50.** **Convention note:** K₁₁ denotes the center kernel coefficient by the question’s explicit zero-based indexing. Pooling averages A before ReLU, so the negative entry is not clipped before taking the mean.

**1. Apply the additive score formula:** With identity U,W, vector v=(1,1)ᵀ and decoder state s=(1,2)ᵀ, add s to encoder states (1,0),(0,1),(1,1). The vectors are (2,2),(1,3),(2,3), giving \[e\approx(1.928055,1.756649,1.959082).\] **2. Extract the first score:** \(e_1=2\tanh2=2(0.964027580)=1.928055160\). To two decimal places, enter 1.93. **Answer: 1.93.**

**1. Apply the additive score formula:** With identity U,W, vector v=(1,1)ᵀ and decoder state s=(1,2)ᵀ, add s to encoder states (1,0),(0,1),(1,1). The vectors are (2,2),(1,3),(2,3), giving \[e\approx(1.928055,1.756649,1.959082).\] **2. Form the normalized weights:** \(\alpha_j=e^{e_j}/\sum_ke^{e_k}\), so \(\alpha\approx(0.347948,0.293139,0.358913)\). Normalization converts scores to positive weights summing to one. **3. Round the first weight:** αₜ₁≈0.347947986 rounds to 0.35. The first state’s score is not the largest, but its normalized weight is still around one third. **Answer: 0.35.**

**1. Apply the additive score formula:** With identity U,W, vector v=(1,1)ᵀ and decoder state s=(1,2)ᵀ, add s to encoder states (1,0),(0,1),(1,1). The vectors are (2,2),(1,3),(2,3), giving \[e\approx(1.928055,1.756649,1.959082).\] **2. Form the normalized weights:** \(\alpha_j=e^{e_j}/\sum_ke^{e_k}\), so \(\alpha\approx(0.347948,0.293139,0.358913)\). Normalization converts scores to positive weights summing to one. **3. Compute the weighted context:** \[c_t=\alpha_1(1,0)^T+\alpha_2(0,1)^T+\alpha_3(1,1)^T\approx(0.706861,0.652052)^T.\] **4. Sum with an independent check:** The coordinate sum is 1.358913057, giving 1.36. It also equals \(\alpha_1+\alpha_2+2\alpha_3=1+\alpha_3\), which verifies the result. **Answer: 1.36.**

**1. Track the per-head matrix shapes:** There are t=32 tokens, model dimension 64 and h=2 heads, so dₖ=d_q=64/2=32. For one head, QᵀK is 32×32; rowwise softmax preserves that shape. **2. Multiply the value matrix:** The specified dᵥ=16 means Vᵀ is 32×16. Hence \(\operatorname{softmax}(Q^TK/\sqrt{d_k})V^T\) has shape 32×16. **3. Count elements:** \[32\cdot16=512.\] This is one head’s output, not the concatenation across two heads. **Answer: 512.**

**1. Count one head’s projections:** With d_model=64,dₖ=d_q=32,dᵥ=16, the query and key matrices each have 32×64=2048 weights. The value matrix has 16×64=1024. Per-head total is 5120. **2. Include both heads:** Two independently parameterized heads require 2×5120=10240 weights. **3. Add the output projection:** Concatenated value width is h dᵥ=32. Wₒ maps this to model width 64, giving 64×32=2048 weights. **4. Sum without biases:** \[P=10240+2048=12288.\] Sequence length controls the number of attention scores and operations, not these projection parameter counts. **Answer: 12288.**

**1. Count the expansion map:** W₁ is 64×256, with 16384 weights, and b₁ has 256 entries. **2. Count the contraction map:** W₂ is 256×64, also with 16384 weights, and b₂ has 64 entries. ReLU adds no trainable parameters. **3. Add the shared positionwise parameters:** \[P=64(256)+256+256(64)+64=32768+320=33088.\] The same feedforward parameters act at every token position, so do not multiply by sequence length. **Answer: 33088.**

**1. Select the good context vector:** From C, the column for good is h=(1,1)ᵀ. Dot it with W’s good,bad,ugly columns (1,−1),(0,1),(−1,0), giving logits (0,1,−1). **2. Normalize all vocabulary scores:** \[P(bad\mid good)=\frac{e^1}{e^0+e^1+e^{-1}}=\frac{2.718282}{1+2.718282+0.367879}.\] **3. Evaluate and round:** The probability is 0.665240956, giving 0.67 to two decimal places. **Answer: 0.67.**

**1. Generate from good:** Its context vector (1,1) gives logits (0,1,−1), so bad has the highest probability and is word1. **2. Feed the selected word back:** The context vector of bad is (−1,−1), giving logits (0,−1,1). Thus word2 is ugly. The context vector of ugly is (1,−1), giving logits (2,−1,−1), so word3 is good. **3. Return exactly the three generated words:** The supplied seed good is not included in the requested sentence, which is bad ugly good. No sampling is performed because each step retains the maximum-probability word. **Answer: A — bad ugly good.** A: **Correct:** It follows the three successive argmax transitions good→bad→ugly→good. B: **Incorrect:** It incorrectly includes the seed as the first generated word. C: **Incorrect:** After bad, the highest-scoring next word is ugly, not bad. D: **Incorrect:** Good is not the highest-scoring output when the context is good. E: **Incorrect:** After ugly, the highest-scoring output is good rather than bad.

**1. Distinguish word and context parameters:** The context good selects h=(1,1)ᵀ from C. The output word embedding for good is W_good=(1,−1)ᵀ. Its logit is 0, and \(p_{good}=1/(1+e+e^{-1})=0.244728471\). **2. Differentiate cross-entropy for the target good:** The target indicator for good is 1, so \[\nabla_{W_{good}}L=(p_{good}-1)h=(-0.755271529,-0.755271529)^T.\] **3. Apply η=1 using the old forward values:** \[W_{good,new}=(1,-1)^T-\nabla L=(1.755271529,-0.244728471)^T.\] The closest listed vector is (1.76,−0.24). This is the output embedding, not the context embedding. **Answer: A — (1.76, -0.24).** A: **Correct:** It rounds the computed updated output word embedding. B: **Incorrect:** It moves in the wrong direction or by the wrong gradient. C: **Incorrect:** It fails to retain the original second component’s −1 starting value. D: **Incorrect:** It incorrectly makes the second component more negative despite the negative gradient there. E: **Incorrect:** It does not match the shared positive-coordinate increment from the gradient.

**1. Follow the scalar recurrence:** The model is \(s_t=\sigma(ux_t+ws_{t-1})\), \(\hat y=vs_2\), with u=w=1,v=2,s₀=0,x₁=−ln3,x₂=−1/4. Therefore \(s_1=\sigma(-\ln3)=1/(1+3)=1/4\), and \(s_2=\sigma(-1/4+1/4)=\sigma(0)=1/2\). **2. Report the first hidden value:** The question asks for s₁, which is exactly 1/4=0.25. The second input does not affect this earlier state. **Answer: 0.25.**

**1. Follow the scalar recurrence:** The model is \(s_t=\sigma(ux_t+ws_{t-1})\), \(\hat y=vs_2\), with u=w=1,v=2,s₀=0,x₁=−ln3,x₂=−1/4. Therefore \(s_1=\sigma(-\ln3)=1/(1+3)=1/4\), and \(s_2=\sigma(-1/4+1/4)=\sigma(0)=1/2\). **2. Report the second hidden value:** The two contributions to its pre-activation cancel exactly. A zero sigmoid logit gives s₂=0.5, not zero. **Answer: 0.5.**

**1. Follow the scalar recurrence:** The model is \(s_t=\sigma(ux_t+ws_{t-1})\), \(\hat y=vs_2\), with u=w=1,v=2,s₀=0,x₁=−ln3,x₂=−1/4. Therefore \(s_1=\sigma(-\ln3)=1/(1+3)=1/4\), and \(s_2=\sigma(-1/4+1/4)=\sigma(0)=1/2\). **2. Apply the linear output weight:** \(\hat y=vs_2=2(1/2)=1\). The output neuron is linear, so no additional sigmoid is applied after multiplying by v. **Answer: 1.**

**1. Follow the scalar recurrence:** The model is \(s_t=\sigma(ux_t+ws_{t-1})\), \(\hat y=vs_2\), with u=w=1,v=2,s₀=0,x₁=−ln3,x₂=−1/4. Therefore \(s_1=\sigma(-\ln3)=1/(1+3)=1/4\), and \(s_2=\sigma(-1/4+1/4)=\sigma(0)=1/2\). **2. Differentiate the recurrent path:** Since s₀=0, \(ds_1/dw=\sigma\prime(ux_1+ws_0)s_0=0\). At the second step, \[\frac{ds_2}{dw}=s_2(1-s_2)\left(s_1+w\frac{ds_1}{dw}\right)=\tfrac12\tfrac12\tfrac14=\tfrac1{16}.\] **3. Include output and loss derivatives:** ŷ=1,y=0 and v=2. Therefore \[\frac{\partial L}{\partial w}=(\hat y-y)v\frac{ds_2}{dw}=1\cdot2\cdot\tfrac1{16}=\tfrac18=0.125.\] Although w is shared across time, the earlier contribution is exactly zero here. **Answer: 0.125.**

**1. Write the first residual block:** Each skip connection adds its input to the ReLU output. At x=1 and w₁=1, \(r_1=x+\operatorname{ReLU}(w_1x)=1+1=2\). **2. Evaluate the second residual branch:** With w₂=−5, its ReLU receives −5(2)=−10 and outputs zero. **3. Include the bypass:** The final output is \(f(1)=r_1+\operatorname{ReLU}(w_2r_1)=2+0=2\). The skip path preserves the positive signal even though the second nonlinear branch is inactive. **Answer: 2.**