Introduction to Deep Learning and Generative AI · Quiz 2 · May 2026

**1. Identify the data:** FashionMNIST contains grayscale images, so each image has one channel. ToTensor converts its pixels to floating-point values, normally between 0 and 1. **2. Apply channel-wise normalization:** For a pixel value \(x\), normalization uses \[z=\frac{x-\mu}{\sigma}.\] Here the supplied dataset mean is \(\mu=0.2860\), and the standard deviation is \(\sigma=0.3530\). **3. Supply one value per channel:** Use transforms.Normalize(mean=(0.2860,), std=(0.3530,)). The trailing comma makes each argument a one-element tuple. For example, a pixel equal to the mean becomes 0 after normalization. **Answer: A.** A: **Correct:** One-element mean and standard-deviation tuples match the one grayscale channel. B: **Incorrect:** This uses scalar values rather than the documented per-channel sequences requested by the question; some implementations may broadcast scalars, but the standard channel-wise form is A. C: **Incorrect:** Three mean and standard-deviation entries describe three channels, whereas FashionMNIST has one. D: **Incorrect:** Scale is not the mean-and-standard-deviation normalization operation.

**1. Interpret the scores:** D(real) is the discriminator's estimated probability that a real sample is real; D(fake) is the same estimate for a generated sample. By epoch 20, the discriminator almost perfectly separates the two. **2. Identify the imbalance:** D(real)=0.999 and D(fake)=0.0001 indicate that the discriminator is winning too easily. With the original saturating generator objective, an almost perfect discriminator can provide a very small useful gradient to the generator. **3. Distinguish other failures:** These two scores alone do not show whether generated images repeat a few patterns, so they cannot prove mode collapse. A successful generator would instead make real and fake samples difficult to distinguish. The vanishing-gradient concern is especially associated with the original saturating GAN loss; modern loss choices can reduce it. **Answer: A.** A: **Correct:** The nearly perfect separation is evidence of an overpowering discriminator and the associated risk of weak generator gradients. B: **Incorrect:** Mode collapse means little diversity in generated samples; discriminator scores alone do not establish it. C: **Incorrect:** Matching the data distribution would prevent such reliable separation. D: **Incorrect:** Nothing in the scores permanently disables discriminator updates.

**1. Recall the GAN goal:** The generator should make samples whose distribution matches that of real samples. **2. Evaluate the ideal discriminator:** With equally likely real and generated samples, the optimal discriminator is \[D^*(x)=\frac{p_{data}(x)}{p_{data}(x)+p_g(x)}.\] When \(p_g=p_{data}\), this equals \(1/2\). **3. Interpret the result:** D(real)=D(fake)=0.5 represents uncertainty between the two classes at the desired balanced equilibrium. A constant 0.5 output by an untrained or broken discriminator alone would not prove success, but it is the intended equilibrium among these choices. **Answer: D.** A: **Incorrect:** The discriminator still separates real and fake very strongly. B: **Incorrect:** This still represents substantial separation. C: **Incorrect:** Perfect discrimination is the opposite of the desired indistinguishable distributions. D: **Correct:** At the balanced ideal equilibrium, neither class can be distinguished better than chance.

**1. Understand clamp:** torch.clamp(grads, min=-5.0, max=2.0) replaces values below -5 by -5 and values above 2 by 2. Values inside the interval are unchanged. **2. Check every entry:** The input is [5.0, -5.0, 0.0, 2.0, 5.0]. Its entries become 2.0, -5.0, 0.0, 2.0 and 2.0 respectively. **3. Read the output:** The resulting values are [2.0, -5.0, 0.0, 2.0, 2.0]. This is element-wise clipping, not normalization or gradient computation. **Answer: D.** A: **Incorrect:** The two entries equal to 5 must be clipped to 2. B: **Incorrect:** Zero must remain zero, and the final 5 must be clipped. C: **Incorrect:** The final entry still incorrectly retains 5. D: **Correct:** Every entry lies in the specified interval after the correct clipping operation.

**1. Separate forward memory from backward history:** An RNN computes \(h_t=f(W_xx_t+W_hh_{t-1}+b)\). Thus the carried hidden state can contain information accumulated before the current training window. **2. Explain truncation:** Truncated backpropagation through time detaches the computational graph after a chosen window of \(\tau\) steps. The numerical hidden state can be carried forward, but gradients do not pass through the detached earlier history. **3. State the consequence:** The backward dependency is limited to the window; the forward state need not be reset. This saves graph storage but does not guarantee that all gradient problems disappear. **Answer: B.** A: **Incorrect:** Detaching a graph and resetting the numerical hidden state are different operations. B: **Correct:** The hidden state continues forward while the gradient history is truncated. C: **Incorrect:** Gradients may still shrink within a window; truncation is not a complete cure. D: **Incorrect:** Old graphs can be released, so truncation normally reduces memory requirements.

**1. Read the input dimensions:** x has shape (4,6,8). With batch_first=True these mean batch size 4, sequence length 6 and input features 8. **2. Determine the sequence output:** The RNN has hidden_size=16, so each of the six time steps produces a 16-dimensional hidden vector for each item. The output shape is (4,6,16). **3. Determine the final hidden state:** For one unidirectional layer, h has shape (num_layers,batch,hidden_size)=(1,4,16). batch_first affects the sequence tensors, not this hidden-state convention. **Answer: A.** A: **Correct:** The output is torch.Size([4,6,16]) and the final hidden state is torch.Size([1,4,16]). B: **Incorrect:** The input explicitly uses batch_first=True, so batch and sequence axes are not exchanged. C: **Incorrect:** The output feature dimension is hidden_size=16, not input_size=8. D: **Incorrect:** The sequence output contains one vector per time step, so its time dimension is not removed.

**1. Training inputs:** Standard teacher forcing supplies the previous ground-truth target token to the decoder during training. Feeding its own previous prediction is typical during inference, or under a different training strategy such as scheduled sampling. **2. Joint learning:** The encoder and decoder are usually trained together through the same sequence loss. Gradients flow through the decoder and context representation into the encoder. **3. Padding:** Padding is an artificial device for batching unequal lengths. Padded target positions should be masked or ignored in the loss rather than treated as meaningful training targets. **Answer: D.** A: **Incorrect:** This describes prediction feedback rather than standard teacher forcing. B: **Incorrect:** The standard architecture is trained jointly, not necessarily in two separately completed stages. C: **Incorrect:** Padding targets are normally excluded from the loss. D: **Correct:** All three proposed descriptions are incorrect for the standard training procedure.

**1. Write the given recurrence:** With a constant gradient \(g\), momentum uses \(v_t=\beta v_{t-1}+g\) and \(x_{t+1}=x_t-\eta v_t\), where \(0<\beta<1\). **2. Solve for terminal velocity:** At steady state, \(v_t=v_{t-1}=v\). Therefore \[v=\beta v+g\quad\Rightarrow\quad (1-\beta)v=g\quad\Rightarrow\quad v=\frac{g}{1-\beta}.\] **3. Compare step magnitudes:** Ordinary GD moves by \(\eta |g|\), while momentum approaches \(\eta |g|/(1-\beta)\). For a nonzero constant gradient, their ratio is \(1/(1-\beta)\). For example, \(\beta=0.9\) gives a factor of 10. Here 'flat and constant' must mean a constant slope; if g were exactly zero both steps would be zero and their ratio would be undefined. **Answer: B.** A: **Incorrect:** This ignores accumulation of the repeated gradient. B: **Correct:** Solving the fixed-point recurrence gives exactly this multiplier. C: **Incorrect:** This is the reciprocal of the required multiplier. D: **Incorrect:** Beta controls decay, not the complete accumulated multiplier.

**1. Recognize a weighted chord:** Since \(x_1<x_2<x_3\), write \(x_2=\lambda x_1+(1-\lambda)x_3\), where \(\lambda=(x_3-x_2)/(x_3-x_1)\) lies between 0 and 1. **2. Test convexity:** A convex function must satisfy \(f(x_2)\leq\lambda f(x_1)+(1-\lambda)f(x_3)\). The first displayed strict inequality is in the opposite direction, so f is not convex on the interval. **3. Test concavity:** A concave function must lie on or above every chord. The second triple gives a point strictly below its chord, so f is not concave either. One counterexample is enough to disprove each global property; the inequalities need not fail for every pair. **Answer: C.** A: **Incorrect:** Both global properties are contradicted by the displayed inequalities. B: **Incorrect:** An affine function lies exactly on every chord, whereas these inequalities are strict. C: **Correct:** The first triple disproves convexity and the second disproves concavity. D: **Incorrect:** The information supplies particular violations, not a violation for every pair.

**1. Find the target image range:** ToTensor gives values in [0,1]. Normalize((0.5,),(0.5,)) applies \((x-0.5)/0.5=2x-1\), mapping the range to [-1,1]. **2. Choose the generator output:** Tanh produces values between -1 and 1, matching the normalized image range and allowing both negative and positive pixel values. **3. Choose the discriminator output:** A standard probability-based GAN discriminator uses sigmoid to produce a real/fake probability between 0 and 1. A logits-based loss may incorporate sigmoid internally, but among the listed activation pairs the intended answer is Tanh, Sigmoid. **Answer: A.** A: **Correct:** Tanh matches the image scale and sigmoid matches a probability output. B: **Incorrect:** The generator cannot cover negative normalized pixel values with sigmoid, while tanh is not a probability range. C: **Incorrect:** ReLU excludes negative values and is unbounded above. D: **Incorrect:** The discriminator probability should not be allowed to become negative. E: **Incorrect:** Sigmoid restricts generated pixels to [0,1], missing the negative half of the target range. F: **Incorrect:** ReLU is unsuitable for both the centered image range and a bounded probability.

**1. Recall the recurrent update:** A basic RNN uses \(h_t=\phi(W_{xh}x_t+W_{hh}h_{t-1}+b)\). Current input and previous hidden state jointly determine the new hidden state. **2. Explain parameter sharing:** The same matrices and biases are used at every time step. A longer input repeats the computation more times but does not create new copies of the trainable parameters. **3. Distinguish dimensions:** Sequence length counts time steps. Input size counts features per step. An RNN can process a different number of steps while keeping this feature size fixed. **Answer: B, C.** A: **Incorrect:** Longer sequences increase computation, not the number of shared trainable weights. B: **Correct:** Both terms appear explicitly in the recurrent update. C: **Correct:** Sharing these matrices across time is a defining feature of the RNN. D: **Incorrect:** The per-step feature size does not need to change when the number of steps changes.

**1. Write the convolution-size formula:** For one spatial dimension, dilation 1 and ordinary convolution, \[O=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1,\] where N is input size, P is padding, K is kernel size and S is stride. **2. Vary one quantity at a time:** Increasing padding enlarges the numerator, allowing more output locations. Increasing the kernel uses more input per location and reduces available positions. Increasing stride samples positions less frequently and generally reduces output size. **3. Account for rounding:** The floor can make a small change leave the size unchanged, but padding is the listed parameter whose increase can enlarge the ordinary convolution output. **Answer: B.** A: **Incorrect:** A larger ordinary-convolution kernel generally reduces output size when the other quantities are fixed. B: **Correct:** Extra padding adds border positions and can increase the number of valid kernel placements. C: **Incorrect:** A larger stride generally reduces the number of sampled output locations.

**1. Use the Hessian test:** For a twice-differentiable function on the usual open convex domain, convexity is equivalent to \(z^T\nabla^2 f(x)z\geq0\) for every x and direction z. This is positive semidefiniteness of the Hessian. **2. Test set operations:** If two points belong to both convex sets, their connecting segment belongs to both sets; hence their intersection is convex. A union can fail: the union of two separated intervals has a gap. **3. Use the first-order inequality:** For differentiable convex f, \(f(y)\geq f(x)+\nabla f(x)^T(y-x)\). At a stationary point the gradient is zero, giving \(f(y)\geq f(x)\) for every y. **4. Avoid false converses:** Having no nonglobal local minima does not imply convexity; \(x^3\) on the real line has no local minima yet is not convex on the whole line. Also, \(g(x)=\cos(\pi x)\) has g'(0)=0 and g''(0)=-\pi^2<0, so x=0 is a stationary maximum. **Answer: A, B, D, E.** A: **Correct:** The positive-semidefinite Hessian test characterizes smooth convex functions under the standard domain assumptions. B: **Correct:** Intersection preserves every connecting segment, whereas a union need not. C: **Incorrect:** The property about local minima is not sufficient to prove convexity. D: **Correct:** At a zero gradient, the convex first-order inequality proves global minimality. E: **Correct:** Cosine has stationary maxima as well as minima; x=0 is a concrete counterexample.

**1. Identify activation ranges:** In a standard LSTM, the forget and input gates use sigmoid, so every component is strictly between 0 and 1 for a finite preactivation. The candidate cell uses tanh, so every component is strictly between -1 and 1. The question specifies infinite precision, so numerical saturation cannot justify values outside these ranges. **2. Check dimensions:** The hidden dimension is 3; every displayed gate and candidate vector has three entries. **3. Check each candidate:** A and C satisfy all component-wise range restrictions. B has a forget-gate component 1.05. D has input-gate component -0.05 and candidate-cell component -1.20. These violate their activation ranges. **Answer: A, C.** A: **Correct:** All forget/input entries lie in (0,1), and all candidate entries lie in (-1,1). B: **Incorrect:** The forget-gate value 1.05 cannot be produced by sigmoid. C: **Correct:** All entries satisfy the appropriate strict activation ranges. D: **Incorrect:** A sigmoid gate cannot be negative, and tanh cannot produce -1.20.

**1. Compute the preactivation:** With no bias, the neuron receives \(z=W^TX\). If either vector is zero, every product in the dot product is zero and therefore z=0. **2. Evaluate the activations at zero:** \[\sigma(0)=\frac{1}{1+e^0}=\frac12,\qquad \operatorname{ReLU}(0)=\max(0,0)=0,\qquad \tanh(0)=0.\] **3. Apply the cases:** A zero dot product does not make sigmoid zero. Zero weights or zero inputs followed by ReLU do give zero. Zero weights followed by tanh also give zero, regardless of a Gaussian-distributed input. **Answer: B, C, D.** A: **Incorrect:** Sigmoid at zero is one-half, not zero. B: **Correct:** Zero weights make the dot product zero, and ReLU(0)=0. C: **Correct:** Zero inputs make the dot product zero, and ReLU(0)=0. D: **Correct:** Zero weights make the Gaussian input irrelevant; tanh(0)=0.

**1. Count the elements:** The image tensor has shape (3,128,128), so it contains \(3\times128\times128=49,152\) entries. **2. Explain flattening:** Flattening preserves all entries while replacing the three dimensions by one dimension of length 49,152. image.flatten(), image.reshape(-1), and image.view(-1) all achieve this for the contiguous tensor created here. The -1 asks PyTorch to infer the required length. **3. Distinguish dimension insertion/removal:** unsqueeze(0) inserts a new size-one dimension, producing (1,3,128,128). squeeze(0) cannot remove the first dimension because its size is 3; the tensor remains three-dimensional. **Answer: A, B, C.** A: **Correct:** flatten combines all dimensions into a single dimension. B: **Correct:** reshape(-1) infers a one-dimensional shape containing all elements. C: **Correct:** view(-1) is valid for this contiguous tensor and gives the same flattened shape. D: **Incorrect:** unsqueeze adds an axis; it does not flatten existing axes. E: **Incorrect:** squeeze cannot remove an axis of size 3 and does not flatten the tensor.

**1. Adagrad:** For a coordinate, its accumulator is \(G_t=G_{t-1}+g_t^2\). Its effective step coefficient is \(\eta/(\sqrt{G_t}+\epsilon)\). Because each squared gradient is nonnegative, G cannot decrease, and this coefficient cannot increase when the base rate is fixed. **2. Sparse features:** A rarely updated coordinate receives fewer nonzero additions to G. Its learning-rate coefficient therefore tends to decay more slowly than that of a frequently updated coordinate. **3. RMSProp:** Replace the cumulative sum by \(V_t=\rho V_{t-1}+(1-\rho)g_t^2\), with \(0<\rho<1\). Older contributions decay. If new gradients become smaller, V can fall and the effective coefficient can rise again. It is an exponentially weighted average, not the total historical sum. This addresses Adagrad's continually shrinking coefficient; it does not guarantee successful optimization in every problem. **Answer: A, B, C, E.** A: **Correct:** The cumulative sum of nonnegative squares cannot decrease. B: **Correct:** Smaller recent squared gradients can lower the moving average and raise the effective coefficient. C: **Correct:** Sparse coordinates accumulate squared-gradient history less often. D: **Incorrect:** RMSProp discounts old contributions rather than accumulating their full sum forever. E: **Correct:** Using a decaying average instead of the cumulative sum is the central RMSProp change.

**1. Identify the layer dimensions:** The RNN has input_size=12, hidden_size=32, num_layers=3 and bias=False. It is unidirectional. **2. First layer:** Its input-to-hidden matrix has \(32\times12=384\) weights. Its hidden-to-hidden matrix has \(32\times32=1,024\) weights. Total: \(384+1,024=1,408\). **3. Second and third layers:** Each receives 32 features from the preceding layer. Each therefore has \(32\times32+32\times32=2,048\) weights. Together they contribute \(2\times2,048=4,096\). **4. Add the layers:** \(1,408+4,096=5,504\). Biases contribute zero because bias=False. Sequence length and batch size do not multiply the number of shared parameters. **Answer: 5504.**

**1. Count one gate's inputs:** Each gate receives the current input of dimension d=100 and previous hidden state of dimension m=128. The concatenated vector therefore has \(100+128=228\) entries. **2. Count one gate/candidate block:** A block produces 128 values, requiring \(128\times228=29,184\) weights and 128 biases. Thus one block has \(29,184+128=29,312\) parameters. **3. Count all four blocks:** The forget, input and output gates plus candidate cell use independent parameter blocks. Therefore \[N=4m(d+m+1)=4\times128\times(100+128+1)=4\times29,312=117,248.\] This counts one bias per output unit per block, as specified by the concatenated affine-map formulation. A library implementation storing two separate bias vectors per block would have a different raw parameter count. **Answer: 117248.**

**1. Count the words to generate:** The sentence has six words: Mumbai / is / a / beautiful / coastal / city. Mumbai is already supplied, so five words remain. **2. Interpret the vocabulary:** At each generation step, the model computes a conditional distribution over all 20 vocabulary words. Conditioning on the preceding two words changes the inputs to that distribution, not its number of entries. **3. Multiply the counts:** Five steps, each with 20 conditional probabilities, give \(5\times20=100\). We do not multiply by \(20^5\): the model generates one successive path, rather than explicitly evaluating every possible five-word continuation. No extra end-of-sentence step is requested. **Answer: 100.**