Deep Learning · End Term · May 2026 · AN

**1. Separate shapes from trainable parameters:** A convolution uses \(P=(k_hk_wC_{in}+1)C_{out}\) parameters: kernel weights across all input channels plus one bias per filter. Its spatial output side is \(\lfloor(N+2p-k)/s\rfloor+1\). A dense layer uses \((n_{in}+1)n_{out}\) parameters. Pooling has none. **2. First convolution and pooling:** The 32×32×3 input becomes 30×30×16 after a valid 3×3, stride-1 convolution. Its parameters are \((3\cdot3\cdot3+1)16=448\). A 2×2, stride-2 pool produces 15×15×16. **3. Second convolution and pooling:** The next valid 3×3 convolution produces 13×13×32 and has \((3\cdot3\cdot16+1)32=4640\) parameters. Pooling gives \(\lfloor(13-2)/2\rfloor+1=6\) positions per side, hence 6×6×32. **4. Flatten and count dense layers:** Flattening gives \(6\cdot6\cdot32=1152\) inputs. The 64-unit dense layer has \((1152+1)64=73792\) parameters. The 10-unit output layer has \((64+1)10=650\). **5. Add each trainable layer once:** \[P_{total}=448+4640+73792+650=79530.\] Spatial reuse of convolutional weights is why they are not multiplied by the number of image positions. **Answer: 79530.**

**1. Follow the chain rule:** A hidden-layer error signal is \(\delta^{(l)}=((W^{(l+1)})^T\delta^{(l+1)})\odot\sigma\prime(z^{(l)})\). Here δ is the derivative of the loss with respect to pre-activation, and ⊙ means elementwise multiplication. Each hidden layer therefore depends on the signal arriving from the layer above. **2. Explain vanishing gradients:** For sigmoid, \[\sigma\prime(z)=\sigma(z)(1-\sigma(z))\le\tfrac14.\] Repeated multiplication by small derivatives can shrink gradients through depth, particularly in saturated units. Weight matrices also affect the total product, so this is a possible failure rather than an unconditional outcome. **3. Explain instability and small-gradient limits:** Gradient descent uses \(W\leftarrow W-\eta\nabla_WL\). A very large gradient can make the update unstable. A small gradient may indicate saturation, a stationary point or a plateau; it does not guarantee that loss decreases on every update. **Answer: A, B, C — chain-rule propagation, possible vanishing gradients and instability from large gradients.** **Convention note:** Sigmoid with binary cross-entropy simplifies the output pre-activation derivative to prediction minus target. Hidden-layer sigmoid derivatives still appear, so that output cancellation does not remove the depth-related issue. A: **Correct:** The chain rule propagates the next layer’s error signal into the current hidden layer. B: **Correct:** Sigmoid derivatives are bounded by 1/4, and their repeated factors can contribute to vanishing gradients. C: **Correct:** A large gradient times the learning rate can create an excessively large parameter change. D: **Incorrect:** Small magnitude alone gives no guarantee of progress; the gradient may even be zero.

**1. Calculate the two hidden pre-activations:** With x=(1,2)ᵀ and zero biases, \[z=\begin{pmatrix}1&-2\\-1&1\end{pmatrix}\begin{pmatrix}1\\2\end{pmatrix}=\begin{pmatrix}-3\\1\end{pmatrix}.\] **2. Apply ReLU:** ReLU(z)=max(0,z) separately for each component, so h=(0,1)ᵀ. The negative first pre-activation is suppressed. **3. Compute the linear output:** Output weights are (1,1)ᵀ and the output bias is zero. Thus \(\hat y=1(0)+1(1)=1\). No sigmoid or output ReLU is specified for this final linear neuron. **Answer: 1.**

**1. Define Maxout for this network:** Each unit selects the larger of its two affine-function values. Unlike ReLU, its alternative branch need not be the constant zero function. **2. Evaluate both hidden units at x=(1,2)ᵀ:** Unit 1 receives \(1(1)-2(2)=-3\) and \(-1(1)+1(2)=1\), so \(h_1=\max(-3,1)=1\). Unit 2 receives \(-1(1)+2(2)=3\) and \(1(1)-1(2)=-1\), so \(h_2=3\). **3. Sum at the linear output:** With output weights both equal to 1 and zero bias, \(\hat y=h_1+h_2=1+3=4\). **Answer: 4.** **Convention note:** The diagram groups the two affine rows separately for each Maxout unit. Its repeated matrix superscript is a labeling typo; the branch grouping shown in the diagram determines the calculation.

**1. Identify the weight’s computational path:** In Scenario A, w₁₁ contributes to the first hidden pre-activation \(z_1=w_{11}x_1+w_{12}x_2=1(1)-2(2)=-3\). **2. Write the loss-gradient chain:** \[\frac{\partial L}{\partial w_{11}}=\frac{\partial L}{\partial\hat y}\frac{\partial\hat y}{\partial h_1}\operatorname{ReLU}\prime(z_1)x_1.\] The final factors are output weight 1, ReLU derivative at −3, and input x₁=1. **3. Apply the inactive ReLU derivative:** Since z₁<0, ReLU′(z₁)=0. Therefore the complete product is zero, regardless of the unspecified finite upstream loss derivative. **Answer: 0.**

**1. Find the winning branch:** The first Maxout unit receives −3 and 1 and outputs 1. The maximum is unique, so backpropagation routes its incoming gradient to the branch producing 1 and gives zero to the losing branch. **2. Compare with the inactive ReLU:** A ReLU at pre-activation −3 has local derivative zero. The winning Maxout affine branch instead has derivative 1 with respect to its own affine value, so a nonzero incoming loss gradient can reach that branch’s weights. **3. Interpret the intended answer:** Yes: this unit is capable of receiving a nonzero weight gradient rather than being blocked by an inactive ReLU. It does not mean every branch receives a gradient. **Answer: A — Yes.** **Convention note:** The loss and target are not specified. A nonzero incoming loss derivative is implicit in the intended Yes answer; at an already stationary loss the weight gradient could still be zero. A: **Correct:** The winning affine branch transmits the incoming gradient to its weights. B: **Incorrect:** The whole unit is not blocked; only the losing affine branch receives zero through the maximum.

**1. Interpret the feature-map axes:** The supplied convolutional map is 7×7×2048. The first two axes identify spatial positions; the third gives the feature vector at each position. **2. Count attention locations:** One encoder state is made from each spatial position, so \[n_{states}=7\cdot7=49.\] Channels determine state dimensionality, not the number of positions. **Answer: B — 49.** A: **Incorrect:** Seven counts only one spatial axis. B: **Correct:** There is one state for each of the 49 spatial locations. C: **Incorrect:** Adding 7+7 does not count positions on a two-dimensional grid. D: **Incorrect:** 2048 is the size of each state vector, not the number of states.

**1. Locate one encoder state:** In a 7×7×2048 map, fixing a row and column leaves all 2048 channel activations at that location. **2. State the vector dimension:** Because each spatial location becomes one encoder state, \(c_j\in\mathbb R^{2048}\). There are 49 such vectors, each with 2048 components. **Answer: 2048.**

**1. Calculate additive attention scores:** The formula is \(e_j=v^T\tanh(Us+Wc_j)\), where s=(0,1)ᵀ, U=W=I and v=(1,1)ᵀ. Thus s+c₁=(1,1), s+c₂=(−1,2), and s+c₃=(2,0). Using the provided tanh values gives \(e_1=1.5232,e_2=0.2024,e_3=0.9640\). **2. Normalize with softmax:** \[\alpha_j=\frac{e^{e_j}}{e^{e_1}+e^{e_2}+e^{e_3}}.\] The supplied exponentials are 4.587, 1.224 and 2.622, whose sum is 8.433. The weights are approximately (0.543935,0.145144,0.310921). **3. Identify the largest weight:** Softmax preserves score ordering because the exponential is increasing and the denominator is common. Score e₁ is largest, so encoder state c₁ receives the highest weight. **Answer: A — encoder state c₁.** A: **Correct:** Its score 1.5232 and weight about 0.544 are the largest. B: **Incorrect:** Its score 0.2024 gives the smallest weight, about 0.145. C: **Incorrect:** Its score 0.9640 gives an intermediate weight, about 0.311. D: **Incorrect:** The three attention scores differ, so the normalized weights are not equal.

**1. Calculate additive attention scores:** The formula is \(e_j=v^T\tanh(Us+Wc_j)\), where s=(0,1)ᵀ, U=W=I and v=(1,1)ᵀ. Thus s+c₁=(1,1), s+c₂=(−1,2), and s+c₃=(2,0). Using the provided tanh values gives \(e_1=1.5232,e_2=0.2024,e_3=0.9640\). **2. Normalize with softmax:** \[\alpha_j=\frac{e^{e_j}}{e^{e_1}+e^{e_2}+e^{e_3}}.\] The supplied exponentials are 4.587, 1.224 and 2.622, whose sum is 8.433. The weights are approximately (0.543935,0.145144,0.310921). **3. Report the requested weight:** \(\alpha_{1t}=4.587/8.433=0.54393454\ldots\). Rounding only at the end gives 0.544 to three decimal places. **Answer: 0.544.**

**1. Calculate additive attention scores:** The formula is \(e_j=v^T\tanh(Us+Wc_j)\), where s=(0,1)ᵀ, U=W=I and v=(1,1)ᵀ. Thus s+c₁=(1,1), s+c₂=(−1,2), and s+c₃=(2,0). Using the provided tanh values gives \(e_1=1.5232,e_2=0.2024,e_3=0.9640\). **2. Normalize with softmax:** \[\alpha_j=\frac{e^{e_j}}{e^{e_1}+e^{e_2}+e^{e_3}}.\] The supplied exponentials are 4.587, 1.224 and 2.622, whose sum is 8.433. The weights are approximately (0.543935,0.145144,0.310921). **3. Form the weighted context vector:** Attention returns \(c_t=\sum_j\alpha_jc_j\). The first coordinates of c₁,c₂,c₃ are 1,−1,2, so \[(c_t)_1=\frac{4.587(1)+1.224(-1)+2.622(2)}{8.433}=\frac{8.607}{8.433}=1.020633\ldots.\] To two decimal places, enter 1.02. **Answer: 1.02.**

**1. Identify the one-step Jacobian:** The stipulated recurrence gives \(J_j=\operatorname{diag}(\sigma\prime(a_j))W\). With derivative factors fixed at 0.5 and W=0.8I₂, each step has \(J_j=0.4I_2\). **2. Count the transitions:** From s₁ to s₆ there are five transitions: 1→2→3→4→5→6. The chain rule multiplies five Jacobians, \[\frac{\partial s_6}{\partial s_1}=(0.4I_2)^5=0.01024I_2.\] **3. Add diagonal entries:** The trace is 2(0.01024)=0.02048, which rounds to 0.020 at three decimal places. **Answer: 0.020.** **Convention note:** The passage stipulates a logistic-sigmoid derivative of 0.5, although a true logistic derivative never exceeds 0.25. This is a formal calculation using the explicitly prescribed Jacobian factors, not a realizable logistic operating point.

**1. Form the stipulated local derivative:** Each step has \(J=\operatorname{diag}(0.5,0.5)(3I_2)=1.5I_2\). **2. Multiply over five transitions:** \[\frac{\partial s_6}{\partial s_1}=(1.5I_2)^5=7.59375I_2.\] The exponent is five because the source state is s₁ and the target state is s₆. **3. Sum and round:** The diagonal sum is 2(7.59375)=15.1875, giving 15.19 to two decimal places. **Answer: 15.19.** **Convention note:** As stipulated in the question, use derivative factors 0.5. Actual logistic-sigmoid derivatives are at most 0.25, so the numerical premise is hypothetical.

**1. Measure the complete local multiplier:** Temporal gradients depend on the product of activation derivatives and recurrent weights. A weight magnitude alone does not determine whether the complete product shrinks or grows. **2. Compare the two prescribed networks:** With derivative factor 0.5 and recurrent weight 0.8, the multiplier is 0.4, so a k-step derivative is proportional to 0.4ᵏ and tends to zero. With recurrent weight 3, the multiplier is 1.5, so it grows as 1.5ᵏ. **3. Name the effects:** The first is vanishing gradients, meaning distant-time signals become tiny. The second is exploding gradients, meaning those signals can become excessively large. **Answer: C — The first network exhibits vanishing gradients while the second exhibits .** **Convention note:** This follows the supplied hypothetical derivative factor 0.5; for an actual logistic sigmoid the derivative bound would need to be enforced. A: **Incorrect:** The first contracts but the second has a complete multiplier greater than one and expands. B: **Incorrect:** It reverses the contraction and expansion in the two cases. C: **Correct:** The first multiplier is 0.4 and the second is 1.5, giving vanishing then exploding behavior. D: **Incorrect:** The first network’s temporal Jacobian contracts rather than expands.

**1. Use the one-hot lookup:** The input word sat is the second vocabulary item, so multiplying W_context by its one-hot vector selects column 2. That column is \(h=(0,1)^T\). **2. Calculate each output score:** The columns of W_word are (1,0),(0,1),(1,1),(−1,0),(0,1). Their dot products with h give \[z=W_{word}^Th=(0,1,1,0,1)^T.\] **3. Sum the requested scores:** \(\sum_jz_j=0+1+1+0+1=3\). These are logits before softmax, not probabilities. **Answer: 3.**

**1. Compute the logits for the input sat:** The one-hot lookup gives h=(0,1)ᵀ, and multiplying W_wordᵀh gives logits (0,1,1,0,1) for [he,sat,on,a,chair]. **2. Apply vocabulary softmax:** The word on has logit 1. Its conditional probability is \[P(on\mid sat)=\frac{e^1}{e^0+e^1+e^1+e^0+e^1}=\frac{e}{2+3e}.\] **3. Evaluate and round:** Using e≈2.718281828 gives numerator 2.718281828 and denominator 10.154845485, so the probability is approximately 0.267684. To three decimals, enter 0.268. **Answer: 0.268.**

**1. Write the rank-one factorization:** A 1000×800 matrix is represented as \(\hat X=u_1\sigma_1v_1^T\). Store the 1000-component left vector, the single singular value and the 800-component right vector. **2. Add storage requirements:** \[1000+1+800=1801\text{ numbers}.\] This is factor storage, not storage of the expanded 800,000-entry approximation. **Answer: 1801.** **Convention note:** Count the three explicit SVD factors as requested. Absorbing the singular value into a vector would use a different storage convention.

**1. Match rows to words:** In the truncated factorization \(\hat X=U\Sigma V^T\), U is 1000×50, V is 800×50 and Σ is 50×50. The rows of UΣ provide 50 features for each vocabulary word; rows of V provide 50 features for each context word. **2. Verify the shared reconstruction:** Setting W_word=UΣ and W_context=V gives \(\hat X=W_{word}W_{context}^T\). Numerical SVD algorithms may be iterative, but SVD does not inherently require CBOW-style gradient training. **3. Check the claimed product:** VᵀV=I gives \[\hat X\hat X^T=U\Sigma V^TV\Sigma U^T=U\Sigma^2U^T=W_{word}W_{word}^T.\] It is not W_context W_contextᵀ, which even has the wrong 800×800 shape. **Answer: A, B — UΣ supplies vocabulary embeddings; V supplies context embeddings.** A: **Correct:** Each row of the 1000×50 matrix UΣ is a 50-dimensional vocabulary embedding. B: **Correct:** Each row of the 800×50 matrix V is a context embedding. C: **Incorrect:** SVD is a matrix factorization and need not be trained with a CBOW-like gradient objective. D: **Incorrect:** The product is UΣ²Uᵀ with shape 1000×1000, not VVᵀ with shape 800×800.

**1. Differentiate the full objective:** For \(J(w)=L(w)+\frac\lambda2w^2\), the regularizer contributes λw. Therefore \[J\prime(w_t)=L\prime(w_t)+\lambda w_t.\] **2. Substitute the current values:** With wₜ=3, L′(wₜ)=4 and λ=0.1, the regularized gradient is 4+0.1(3)=4.3. The factor 1/2 in the penalty cancels the derivative’s factor 2. **3. Take one descent step:** With learning rate η=0.05, \[w_{t+1}=3-0.05(4.3)=3-0.215=2.785.\] The L2 term makes the update slightly more strongly shrink a positive weight than the unregularized update would. **Answer: 2.785.**