Questions and published explanations below are available without starting a test. Some questions may not have a published solution yet.
19 / 19 published solutions
Question 2 NAT · 2.0 marks
Consider the input matrix
and filter
given below:
Suppose
is convolved with
using a padding of 1 and a stride of 2 to produce an output
What will be the value of
?
Published solution
**1. Locate the requested output:** The 3-by-2 filter, padding one and stride two produce a 2-by-3 output. Entry e is output row 2, column 2.
**2. Select its window:** In the padded image, the corresponding original-data window is \(\begin{bmatrix}1&2\\0&1\\3&0\end{bmatrix}\). The filter is \(\begin{bmatrix}1&0\\0&-1\\1&1\end{bmatrix}\).
**3. Multiply and sum:** \[e=1(1)+2(0)+0(0)+1(-1)+3(1)+0(1)=3.\] This uses the cross-correlation convention standard in CNN layers, with no spatial flipping of the supplied filter.
**Answer: 3.**
Consider the optimization problem of minimizing the function
We compare gradient descent and an Adam optimizer starting from an arbitrary point
.
Based on the above data, answer the given subquestions.
Starting at
, run two steps of gradient descent with learning rate
. The iterates
follow:
If
, find
correct to two decimal places.
Published solution
**1. Differentiate the objective:** For \(f(x,y)=3(x^2+y^2)\), gradient is (6x,6y).
**2. Perform one gradient step:** With eta=0.1, \((x_{t+1},y_{t+1})=(x_t,y_t)-0.1(6x_t,6y_t)=0.4(x_t,y_t)\).
**3. Perform two steps:** \((x_2,y_2)=0.4^2(x_0,y_0)=0.16(x_0,y_0)\). Thus the requested common factor alpha is 0.16.
**Answer: 0.16.**
Consider the optimization problem of minimizing the function
We compare gradient descent and an Adam optimizer starting from an arbitrary point
.
Based on the above data, answer the given subquestions.
Starting at
, we now turn to Adagrad. The update rule is defined as follows:
Assume
,
and
. If
, determine whether such a
constant
exists. If yes, enter
correct to two decimal places; otherwise enter
.
Published solution
**1. Inspect adaptive scaling:** Adagrad divides each coordinate’s gradient by the square root of that coordinate’s accumulated squared gradients. The two coordinates are not generally scaled by a shared factor.
**2. Test a counterexample to a universal alpha:** With initial point (1,2), zero accumulator, epsilon=0 and eta=0.1, the first positive-coordinate step subtracts 0.1 from both coordinates, producing (0.9,1.9). In step two the decrements are \(0.1(0.9)/\sqrt{1^2+0.9^2}\) and \(0.1(1.9)/\sqrt{2^2+1.9^2}\), about 0.0669 and 0.0689. The final coordinates are about (0.8331,1.8311), not a common multiple of (1,2).
**3. Conclude:** No starting-point-independent constant alpha describes two general Adagrad steps. The question instructs entering 0 when such a constant does not exist. Special symmetric initial points can still have a common factor; zero coordinates require a safe denominator convention when epsilon is set to zero.
**Answer: 0.**
Consider a neural network with the following architecture with one hidden layer and one output
layer. Hidden layer has three neurons with ReLU as an activation function while the output layer
has one neuron with sigmoid as the activation function. The input
is the true output, and
is the predicted output.
The loss function used is squared loss
. Following are the weights of the
network.
Assume the bias to be zero.
Based on the above data, answer the given subquestions.
Perform one forward pass and compute the loss for the input
.
Enter the answer correct to three decimal places.
Published solution
**1. Evaluate hidden preactivations:** Each hidden row has four unit weights, the input is (-1,-1,-1,-1), and bias is zero. Each preactivation equals -4.
**2. Apply activations:** ReLU sets all three hidden activations to zero. The output preactivation is therefore zero; sigmoid(0)=0.5.
**3. Compute the stated squared loss:** For y=1 and a single example, \[L=\tfrac12(1-0.5)^2=\tfrac12(0.25)=0.125.\]
**Answer: 0.125.**
Consider a neural network with the following architecture with one hidden layer and one output
layer. Hidden layer has three neurons with ReLU as an activation function while the output layer
has one neuron with sigmoid as the activation function. The input
is the true output, and
is the predicted output.
The loss function used is squared loss
. Following are the weights of the
network.
Assume the bias to be zero.
Based on the above data, answer the given subquestions.
Perform backpropagation using the optimizer stochastic gradient descent and find the number of
weights that will get updated in
.
Published solution
**1. Identify the hidden-layer derivative:** All three hidden preactivations are -4, strictly in ReLU’s flat negative region. Each ReLU derivative is zero.
**2. Apply backpropagation:** A first-layer weight gradient contains the hidden derivative as a factor: \(\partial L/\partial W^{(1)}_{ij}=\delta_i^{hidden}x_j\). Here every hidden delta is zero, regardless of the nonzero output error.
**3. Count updates:** All 12 entries of W1 have zero gradient, so none changes under ordinary SGD without an additional weight-decay term. The requested update count is 0.
**Answer: 0.**
Consider a neural network with the following architecture with one hidden layer and one output
layer. Hidden layer has three neurons with ReLU as an activation function while the output layer
has one neuron with sigmoid as the activation function. The input
is the true output, and
is the predicted output.
The loss function used is squared loss
. Following are the weights of the
network.
Assume the bias to be zero.
Based on the above data, answer the given subquestions.
Will there be any change in the answer to the previous question if we change the hidden layer
activation to sigmoid (logistic)?
6406536924441.
Yes
6406536924442.
No
Published solution
**1. Replace the hidden activation:** Sigmoid(-4) is small but strictly positive, and its derivative is \(\sigma(-4)[1-\sigma(-4)]>0\).
**2. Trace nonzero gradients:** The output error is nonzero, output weights are one and all four input coordinates are -1. Thus every hidden delta and every first-layer weight gradient is nonzero in exact arithmetic.
**3. Compare with ReLU:** The prior count was zero. With sigmoid, all 12 first-layer weights receive nonzero SGD updates at a nonzero learning rate, so the answer changes.
**Answer: A — Selected: Yes.**
A: **Correct:** Sigmoid has a nonzero derivative at -4 and allows gradients through all hidden units.
B: **Incorrect:** The ReLU zero-gradient result does not remain true after replacing the activation with sigmoid.
Consider a 2D convolutional network where the input image has three channels. The first
convolution layer is computed using a kernel size of
and has four channels. The second
convolution layer is computed using a kernel size of
and has ten channels. Then, which
among the following are correct?
6406536924443.
The first layer has 108 weights.
6406536924444.
The second layer has 1000 weights.
6406536924445.
The first layer has 36 weights.
6406536924446.
The second layer has 250 weights.
6406536924447.
The first layer has 4 biases.
6406536924448.
The second layer has 12 biases.
Published solution
**1. Count first-layer weights:** A 3-by-3 filter spans three input channels. Four output filters require \(3(3)(3)(4)=108\) weights and four biases.
**2. Count second-layer weights:** Its 5-by-5 filters span the four preceding channels. Ten output filters require \(5(5)(4)(10)=1000\) weights and ten biases.
**3. Select matching statements:** The first-layer weight count, second-layer weight count and first-layer bias count in A,B,E are correct.
**Answer: A, B, E — Selected: The first layer has 108 weights.; The second layer has 1000 weights.; The first layer has 4 biases.**
A: **Correct:** Four full 3-by-3-by-3 filters contain 108 weights.
B: **Correct:** Ten full 5-by-5-by-4 filters contain 1000 weights.
C: **Incorrect:** 36 omits the factor of three input channels.
D: **Incorrect:** 250 omits the factor of four input channels.
E: **Correct:** There is one bias for each of four output channels.
F: **Incorrect:** The second layer has ten output-channel biases, not twelve.
Which of the following techniques prevent the model from overfitting?
6406536924449.
Dropout
6406536924450.
Early stopping
6406536924451.
L2 regularization
6406536924452.
MaxOut
6406536924453.
Increasing model depth
Published solution
**1. Explain dropout:** Randomly zeroing activations during training discourages reliance on particular features and acts as regularization.
**2. Explain stopping and penalties:** Early stopping can halt training before validation performance deteriorates. L2 regularization penalizes large weight magnitudes and can reduce overfitting.
**3. Compare architecture changes:** Maxout adds a flexible activation structure and is not itself a general regularizer, though it is often paired with dropout. Increasing depth generally increases capacity rather than inherently preventing overfitting. None of these methods guarantees perfect generalization.
**Answer: A, B, C — Selected: Dropout; Early stopping; L2 regularization.**
A: **Correct:** Dropout is an explicit regularization technique.
B: **Correct:** Validation-based early stopping limits overtraining.
C: **Correct:** L2 adds a weight-magnitude penalty.
D: **Incorrect:** Maxout by itself is not the named overfitting-control mechanism.
E: **Incorrect:** Greater depth does not automatically regularize a model.
belong to another class. It is not possible to separate these two classes by
using a linear decision boundary. We consider a neural network with:
• Input
• One hidden layer with 2 neurons and ReLU activation
• Output layer: one neuron with linear activation
The network is defined as:
Consider the weight matrices:
Suppose we scale the second layer weights as
. Find the range of
for which the
network correctly classifies all four points using threshold
**1. Compute hidden features:** The given first-layer rows compute \(x_1-x_2\) and \(-x_1+x_2\), followed by ReLU.
**2. Evaluate the four inputs:** At (0,0) and (1,1), both hidden values are zero, so the output is zero. At (1,0) and (0,1), one hidden value is one and the other zero, so scaling output weights to (alpha,alpha) produces output alpha.
**3. Apply the classifier threshold:** Under the intended rule that output at least 0.5 is the positive class, the unequal-input cases are correct exactly when \(\alpha\ge0.5\). The equal-input cases remain below threshold. If equality were assigned to the negative class instead, the strict endpoint would differ.
**Answer: A.**
A: **Correct:** Alpha at least 0.5 is the full valid range under the stated positive-at-threshold convention.
B: **Incorrect:** This omits the valid equality endpoint.
C: **Incorrect:** Values below 0.5 misclassify the unequal-input cases.
D: **Incorrect:** This range puts both positive-class examples below threshold.
Consider the motivation for using Transformers instead of RNN-based sequence models. Which of
the following option is correct?
**Statement I**: Transformers allow parallel computation across tokens in a sequence during
training, whereas RNNs process tokens sequentially across time steps.
**Statement II**: Transformer reduces training time significantly for long sequences compared to
RNN-based models.
6406536924462.
Both Statement I and Statement II are true, and Statement I is the correct
explanation of Statement II
6406536924463.
Both Statement I and Statement II are true, but Statement I is not the correct
explanation of Statement II
6406536924464.
Statement I is true, but Statement II is false
6406536924465.
Statement I is false, but Statement II is true
Published solution
**1. Compare training dependencies:** An RNN hidden state depends on the preceding hidden state, imposing sequential computation along time. Transformer self-attention can process representations of many tokens in parallel during training.
**2. Explain the usual speed benefit:** Reducing the sequential path and using large parallel matrix operations often makes sequence-model training faster on suitable hardware. This supports both statements and the proposed explanation in the intended comparison.
**3. State the scope:** It is not a universal runtime guarantee for every sequence length, implementation or hardware; dense attention has quadratic sequence-length work, and autoregressive generation is still sequential across generated tokens.
**Answer: A — Selected: Both Statement I and Statement II are true, and Statement I is the correct explanation of Statement II.**
A: **Correct:** Token-parallel training is the intended explanation of the usual Transformer training-speed advantage.
B: **Incorrect:** Statement I directly supplies the stated parallelism explanation.
C: **Incorrect:** The intended broad hardware-parallel training comparison treats Statement II as true, with the stated qualifications.
D: **Incorrect:** RNN recurrence is sequential, so Statement I is true.
Consider the role of Multi-Head Attention in the Transformer architecture. Which of the following
options correctly identifies the truth values of the two statements below?
**Statement I**: Multi-Head Attention allows different heads to learn different contextual
relationships or representation subspaces among tokens in the same sequence.
**Statement II**: Multi-Head Attention is used mainly to reduce the sequence length before passing
the output to the feed-forward layer.
6406536924458.
Statement I is True, Statement II is False
6406536924459.
Statement I is False, Statement II is True
6406536924460.
Both Statement I and Statement II are True
6406536924461.
Both Statement I and Statement II are False
Published solution
**1. Interpret multiple heads:** Separate query/key/value projections let heads represent different token relationships or feature subspaces.
**2. Track sequence length:** Attention produces an output representation at each query position. Standard multi-head attention concatenates heads and projects feature width; it does not normally reduce the token count.
**3. Judge the statements:** I is true and II is false. Learning different relationships is possible, rather than a guarantee that every trained head becomes distinct.
**Answer: A — Selected: Statement I is True, Statement II is False.**
A: **Correct:** Separate head projections support different relationships, while token count normally stays unchanged.
B: **Incorrect:** The stated truth values are reversed.
C: **Incorrect:** Sequence-length reduction is not the main role of multi-head attention.
D: **Incorrect:** Statement I correctly describes the representational purpose.
A text corpus contains the following two sentences.
• Machine learning models learn patterns from data and improve with experience.
• Deep learning models learn complex patterns from large data.
Based on the above data, answer the given subquestions.
Build a vocabulary
and enter its size
.
Published solution
**1. Apply word-level normalization:** Use the intended lowercase words with sentence-ending periods excluded. Count repeated words only once.
**2. List the vocabulary:** machine, learning, models, learn, patterns, from, data, and, improve, with, experience, deep, complex, large.
**3. Count:** There are 14 unique words. Treating punctuation or capitalization as distinct tokens would be a different tokenization convention from the word vocabulary intended here.
**Answer: 14.**
A text corpus contains the following two sentences.
• Machine learning models learn patterns from data and improve with experience.
• Deep learning models learn complex patterns from large data.
Based on the above data, answer the given subquestions.
Suppose we build a co-occurrence matrix of size
, where each row corresponds to a word in
the vocabulary and the columns correspond to the context of the word. Which of the following
could be a valid size of the co-occurrence matrix? (select all correct answers)
**1. Fix the word rows:** The previous corpus has 14 vocabulary words. The question says each matrix row corresponds to a vocabulary word, so every valid offered size must have 14 rows.
**2. Allow different context definitions:** Columns may represent all vocabulary words or some other specified/restricted contexts. Their count need not equal 14 unless context is defined as the entire same vocabulary.
**3. Select offered shapes:** 14-by-14, 14-by-13 and 14-by-7 all satisfy the fixed row count and can represent suitable context sets. The other shapes have the wrong number of word rows.
**Answer: B, C, E.**
A: **Incorrect:** Twenty rows do not match the 14-word vocabulary.
B: **Correct:** Fourteen word rows and 14 contexts are possible.
C: **Correct:** Fourteen word rows and a 13-context representation are possible.
D: **Incorrect:** Eleven rows omit vocabulary words.
E: **Correct:** Fourteen word rows and seven chosen context types are possible.
F: **Incorrect:** Six rows do not provide one row per vocabulary word.
G: **Incorrect:** Fifteen rows exceed the stated word vocabulary.
H: **Incorrect:** Twelve rows do not match the required 14 rows.
A character-level RNN is trained to generate passwords using 8 possible characters, each
represented as a one-hot encoded vector of length 8. The RNN uses the following equations at
time step
:
Here,
is the hidden activation function and
is the softmax function.
Assume that
Further assume that the special end-of-sequence (EOS) token is not generated.
Based on the above data, answer the given subquestions.
What is the total number of trainable parameters in the network (including bias terms)?
Published solution
**1. Identify dimensions:** Input one-hot width is 8, hidden-state width is 3 and output vocabulary width is 8.
**2. Count recurrent parameters:** Input-to-hidden U has \(3(8)=24\) weights, hidden-to-hidden W has \(3(3)=9\), and hidden bias b has 3.
**3. Count output parameters:** Hidden-to-output V has \(8(3)=24\) weights and output bias c has 8.
**4. Add:** \(24+9+3+24+8=68\). Parameters are shared across timesteps, so password length does not multiply this count.
**Answer: 68.**
A character-level RNN is trained to generate passwords using 8 possible characters, each
represented as a one-hot encoded vector of length 8. The RNN uses the following equations at
time step
:
Here,
is the hidden activation function and
is the softmax function.
Assume that
Further assume that the special end-of-sequence (EOS) token is not generated.
Based on the above data, answer the given subquestions.
If the network generates a password of length 5 during inference, how many times is the hidden
state vector
computed?
Published solution
**1. Match updates to generated tokens:** Each recurrent step computes a new hidden state and produces a distribution for the next character.
**2. Count the requested characters:** A length-five password requires five such recurrent prediction steps.
**3. Exclude extra assumptions:** The initial hidden state is initialized, not counted as a recurrently computed output here. No extra EOS-generation step is included because the problem explicitly excludes EOS. Thus the hidden state is computed five times.
**Answer: 5.**
Which of the following tasks are appropriate examples of **sequence learning** problems typically
solved using Recurrent Neural Networks (RNNs)? Choose all correct options.
6406536924475.
Predicting the next word in a sentence using previous words
6406536924476.
Sentiment analysis of a movie review using the sequence of words
6406536924477.
Classifying whether an image contains a cat or dog from a single image
6406536924478.
Machine translation from English to French
6406536924479.
Predicting tomorrow's stock price using past daily prices
6406536924480.
Detecting edges in an image
6406536924481.
Speech recognition from an audio signal over time
6406536924482.
Identifying handwritten digits from static images
6406536924483.
Generating captions for a video using the sequence of frames
6406536924484.
Predicting the next frame in a video sequence
Published solution
**1. Identify ordered data:** A sequence task depends on relationships across ordered words, time samples or frames, rather than only a single static item.
**2. Match tasks:** Next-word prediction and sentiment use word order; translation maps one word sequence to another. Stock prediction and speech recognition depend on temporal observations. Video captioning and next-frame prediction depend on ordered frames.
**3. Exclude the static examples as posed:** Single-image cat/dog recognition, edge detection and static digit classification do not inherently require temporal recurrence. Images can be reformulated as sequences, but that is not the usual task framing asked here.
**Answer: A, B, D, E, G, I, J — Selected: Predicting the next word in a sentence using previous words; Sentiment analysis of a movie review using the sequence of words; Machine translation from English to French; Predicting tomorrow's stock price using past daily prices; Speech recognition from an audio signal over time; Generating captions for a video using the sequence of frames; Predicting the next frame in a video sequence.**
A: **Correct:** The next word depends on the preceding word sequence.
B: **Correct:** Word ordering contributes to sentence/review sentiment.
C: **Incorrect:** The given single-image task is static rather than inherently sequential.
D: **Correct:** Translation is a sequence-to-sequence task.
E: **Correct:** Past daily values form a time series.
F: **Incorrect:** Static edge detection is a spatial task as stated.
G: **Correct:** Speech is a temporal sequence of acoustic observations.
H: **Incorrect:** The stated digit task uses a static image.
I: **Correct:** A video supplies a sequence of frames for captioning.
J: **Correct:** Predicting a future frame depends on the preceding frame sequence.
**1. Read the embedding convention:** The sentence has eight tokens. Each token embedding contains 64 features.
**2. Stack token rows:** The passage explicitly uses rows as tokens, so \[H\in\mathbb R^{8\times64}.\]
**3. Check matrix multiplication:** This can multiply WQ of shape 64-by-32 because the inner 64 dimensions agree. Transposing H would break that specified multiplication.
**Answer: A.**
A: **Correct:** Eight rows of 64 features give 8-by-64.
B: **Incorrect:** 64-by-8 reverses the stated token-row convention.
C: **Incorrect:** 8-by-32 is a projected query/key size, not the input size.
D: **Incorrect:** 64-by-64 does not reflect the eight-token sequence.
**1. Identify input shape:** H is 8-by-64. WQ and WK are 64-by-32, while WV is 64-by-16.
**2. Apply matrix shape rules:** \((8\times64)(64\times32)\) gives 8-by-32 for Q and K. \((8\times64)(64\times16)\) gives 8-by-16 for V.
**3. Match:** The ordered shapes are \(Q:8\times32\), \(K:8\times32\), \(V:8\times16\). Token count is unchanged; projected feature dimensions differ.
**Answer: C.**
A: **Incorrect:** The dimensions are transposed relative to the token-row convention.
B: **Incorrect:** The token dimension should remain eight, not 64.
C: **Correct:** These shapes follow directly from the three supplied projection matrices.
D: **Incorrect:** The projection output widths are 32,32,16 rather than all 64.
Suppose you are training a skip-gram model to learn bigram embeddings, i.e., the input tokens
are bigrams, and the context tokens are still unigrams. Suppose we have the vocabulary size
, number of unique bigrams
, and the embedding dimension
.
How many parameters are learned in this model?
Published solution
**1. Count input bigram embeddings:** There are 120,000 unique bigrams, each with width 50. The input table has \(120{,}000(50)=6{,}000{,}000\) parameters.
**2. Count output unigram embeddings:** The context vocabulary has 8,000 unigrams, giving \(8{,}000(50)=400{,}000\) parameters.
**3. Add:** \[N=(120{,}000+8{,}000)(50)=6{,}400{,}000.\] This is the standard two-table skip-gram count with no additional output bias specified.
**Answer: 6400000.**
