Machine Learning Foundations · Quiz 2 · May 2024

**Answer: C** **Topic:** Linear Algebra — *Symmetric Matrix Representation of a Quadratic Form* ### Step-by-Step Solution **Step 1:** Write the general quadratic form in three variables $x = [x_1, x_2, x_3]^T$ as $Q(x) = x^T A x = \sum_{i=1}^3 A_{ii} x_i^2 + 2 \sum_{1 \le i < j \le 3} A_{ij} x_i x_j$, where $A$ is a symmetric $3 \times 3$ matrix ($A_{ij} = A_{ji}$). **Step 2:** Read the diagonal entries directly from the coefficients of the squared terms $x_1^2, x_2^2, x_3^2$ in $2x_1 x_2 + 4x_2 x_3 - x_1^2 - 5x_2^2 - 3x_3^2$: $$A_{11} = -1, \quad A_{22} = -5, \quad A_{33} = -3$$ **Step 3:** Read the off-diagonal entries by halving the coefficients of the cross terms $x_i x_j$: - Coefficient of $x_1 x_2$ is $2 \implies A_{12} = A_{21} = \frac{2}{2} = 1$ - Coefficient of $x_1 x_3$ is $0 \implies A_{13} = A_{31} = 0$ - Coefficient of $x_2 x_3$ is $4 \implies A_{23} = A_{32} = \frac{4}{2} = 2$ **Step 4:** Assemble the symmetric matrix $A$: $$A = \begin{pmatrix} -1 & 1 & 0 \\ 1 & -5 & 2 \\ 0 & 2 & -3 \end{pmatrix}$$ which matches **C**. **Key Takeaway / Tip:** Always divide each cross-term coefficient ($x_i x_j$ for $i \ne j$) by $2$ when placing it into the symmetric matrix $A_{ij} = A_{ji}$. A: **Incorrect:** Option A (

) does not satisfy the conditions derived in the step-by-step analysis for Symmetric Matrix Representation of a Quadratic Form (correct choice: C). B: **Incorrect:** Option B (

) does not satisfy the conditions derived in the step-by-step analysis for Symmetric Matrix Representation of a Quadratic Form (correct choice: C). C: **Correct:** Option C (

) matches our derived result under Symmetric Matrix Representation of a Quadratic Form. Always divide each cross-term coefficient ($x_i x_j$ for $i \ne j$) by $2$ when placing it into the symmetric matrix $A_{ij} = A_{ji}$. D: **Incorrect:** Option D (

) does not satisfy the conditions derived in the step-by-step analysis for Symmetric Matrix Representation of a Quadratic Form (correct choice: C).

**Answer: B** **Topic:** Linear Algebra — *Definiteness of a Symmetric Matrix via Sylvester's Criterion* ### Step-by-Step Solution **Step 1:** Use **Sylvester's Criterion** on the symmetric matrix $A = \begin{pmatrix} -1 & 1 & 0 \\ 1 & -5 & 2 \\ 0 & 2 & -3 \end{pmatrix}$ by computing its leading principal minors $D_1, D_2, D_3$: **Step 2:** Compute each leading principal minor: - First leading principal minor: $$D_1 = \det(-1) = -1 < 0$$ - Second leading principal minor: $$D_2 = \det\begin{pmatrix} -1 & 1 \\ 1 & -5 \end{pmatrix} = (-1)(-5) - (1)(1) = 5 - 1 = 4 > 0$$ - Third leading principal minor ($\det A$): $$D_3 = -1((-5)(-3) - 2(2)) - 1(1(-3) - 2(0)) = -1(11) - 1(-3) = -8 < 0$$ **Step 3:** Because the leading principal minors alternate in sign starting with negative ($D_1 < 0, D_2 > 0, D_3 < 0$), all eigenvalues of $A$ are strictly negative, so $A$ is **negative definite** (**B**). **Key Takeaway / Tip:** By Sylvester's Criterion, a symmetric matrix is positive definite iff $D_k > 0$ for all $k$, and negative definite iff $(-1)^k D_k > 0$ for all $k$ (i.e., $D_1 < 0, D_2 > 0, D_3 < 0$). A: **Incorrect:** Option A (positive definite.) does not satisfy the conditions derived in the step-by-step analysis for Definiteness of a Symmetric Matrix via Sylvester's Criterion (correct choice: B). B: **Correct:** Option B (negative definite.) matches our derived result under Definiteness of a Symmetric Matrix via Sylvester's Criterion. By Sylvester's Criterion, a symmetric matrix is positive definite iff $D_k > 0$ for all $k$, and negative definite iff $(-1)^k D_k > 0$ for all $k$ (i.e., $D_1 < 0, D_2 > 0, D_3 < 0$).

**Answer: A, B, C** **Topic:** Linear Algebra — *Properties of Hermitian Matrices* ### Step-by-Step Solution **Step 1:** Examine the complex matrix $A = \begin{pmatrix} 2 & 1+i & i \\ 1-i & 3 & 2i \\ -i & -2i & 1 \end{pmatrix}$ by taking its conjugate transpose $A^H = (\overline{A})^T$: - Diagonal entries $2, 3, 1$ are real, so $\overline{A_{ii}} = A_{ii}$. - Off-diagonal entries satisfy $\overline{A_{12}} = \overline{1+i} = 1-i = A_{21}$, $\overline{A_{13}} = \overline{i} = -i = A_{31}$, and $\overline{A_{23}} = \overline{2i} = -2i = A_{32}$. Therefore, $A^H = A$, meaning **$A$ is a Hermitian matrix** (**A** is true, and **D** is false because a skew-Hermitian matrix requires $A^H = -A$ with zero or purely imaginary diagonal entries). **Step 2:** By the **Spectral Theorem for Hermitian Matrices**, every Hermitian matrix has **exclusively real eigenvalues** (**C** is true) and is **unitarily diagonalizable** (**B** is true). **Step 3:** Thus, **A, B, and C** are correct. **Key Takeaway / Tip:** A complex matrix with real diagonal entries and conjugate-symmetric off-diagonal entries ($A_{ji} = \overline{A_{ij}}$) is Hermitian, guaranteeing real eigenvalues and unitary diagonalizability. A: **Correct:** Option A (A is a Hermitian matrix.) matches our derived result under Properties of Hermitian Matrices. A complex matrix with real diagonal entries and conjugate-symmetric off-diagonal entries ($A_{ji} = \overline{A_{ij}}$) is Hermitian, guaranteeing real eigenvalues and unitary diagonalizability. B: **Correct:** Option B (A is digonalizable.) matches our derived result under Properties of Hermitian Matrices. A complex matrix with real diagonal entries and conjugate-symmetric off-diagonal entries ($A_{ji} = \overline{A_{ij}}$) is Hermitian, guaranteeing real eigenvalues and unitary diagonalizability. C: **Correct:** Option C (All eigenvalue of A real.) matches our derived result under Properties of Hermitian Matrices. A complex matrix with real diagonal entries and conjugate-symmetric off-diagonal entries ($A_{ji} = \overline{A_{ij}}$) is Hermitian, guaranteeing real eigenvalues and unitary diagonalizability. D: **Incorrect:** Option D (A has eigenvalues that are neither real nor purely imaginary.) does not satisfy the conditions derived in the step-by-step analysis for Properties of Hermitian Matrices (correct choice: A, B, C).

**Answer: B** **Topic:** Singular Value Decomposition — *Left Singular Vectors of a Matrix* ### Step-by-Step Solution **Step 1:** Given $A = \begin{pmatrix} 1 & 0 & 1 \\ 1 & 0 & -1 \end{pmatrix}$ and its Singular Value Decomposition $A = Q_1 \Sigma Q_2^T$, the columns of $Q_1 \in \mathbb{R}^{2 \times 2}$ are the orthonormal eigenvectors of $A A^T$ (or unit columns aligned with the singular vectors in $Q_2$). **Step 2:** Check the candidate orthonormal $2 \times 2$ matrices for $Q_1$ alongside $Q_2 = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \end{pmatrix}$ and $\Sigma = \begin{pmatrix} \sqrt{2} & 0 & 0 \\ 0 & \sqrt{2} & 0 \end{pmatrix}$: $$Q_1 \Sigma Q_2^T = \frac{1}{\sqrt{2}} \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix} \begin{pmatrix} \sqrt{2} & 0 & 0 \\ 0 & \sqrt{2} & 0 \end{pmatrix} \begin{pmatrix} 1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \end{pmatrix} = \begin{pmatrix} 1 & 1 & 0 \\ 1 & -1 & 0 \end{pmatrix} \begin{pmatrix} 1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \end{pmatrix} = \begin{pmatrix} 1 & 0 & 1 \\ 1 & 0 & -1 \end{pmatrix} = A$$ **Step 3:** Verify that this reproduces $A$ exactly! Therefore, $Q_1 = \frac{1}{\sqrt{2}} \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}$, which matches **B**. **Key Takeaway / Tip:** When $A A^T = 2I_2$ is a scalar multiple of the identity, any orthonormal basis of $\mathbb{R}^2$ is an eigenvector basis of $A A^T$, so $Q_1$ and $Q_2$ must be chosen consistently via $A Q_2 = Q_1 \Sigma$. A: **Incorrect:** Option A (

) does not satisfy the conditions derived in the step-by-step analysis for Left Singular Vectors of a Matrix (correct choice: B). B: **Correct:** Option B (

) matches our derived result under Left Singular Vectors of a Matrix. When $A A^T = 2I_2$ is a scalar multiple of the identity, any orthonormal basis of $\mathbb{R}^2$ is an eigenvector basis of $A A^T$, so $Q_1$ and $Q_2$ must be chosen consistently via $A Q_2 = Q_1 \Sigma$. C: **Incorrect:** Option C (

) does not satisfy the conditions derived in the step-by-step analysis for Left Singular Vectors of a Matrix (correct choice: B). D: **Incorrect:** Option D (

) does not satisfy the conditions derived in the step-by-step analysis for Left Singular Vectors of a Matrix (correct choice: B).

**Answer: A** **Topic:** Singular Value Decomposition — *Right Singular Vectors of a Matrix* ### Step-by-Step Solution **Step 1:** Using $A = \begin{pmatrix} 1 & 0 & 1 \\ 1 & 0 & -1 \end{pmatrix}$, $Q_1 = \frac{1}{\sqrt{2}} \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}$, and $\Sigma = \begin{pmatrix} \sqrt{2} & 0 & 0 \\ 0 & \sqrt{2} & 0 \end{pmatrix}$, we need the orthogonal matrix $Q_2 \in \mathbb{R}^{3 \times 3}$ satisfying $A = Q_1 \Sigma Q_2^T$, or equivalently $Q_1^T A = \Sigma Q_2^T$. **Step 2:** Compute $Q_1^T A$: $$Q_1^T A = \frac{1}{\sqrt{2}} \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix} \begin{pmatrix} 1 & 0 & 1 \\ 1 & 0 & -1 \end{pmatrix} = \frac{1}{\sqrt{2}} \begin{pmatrix} 2 & 0 & 0 \\ 0 & 0 & 2 \end{pmatrix} = \begin{pmatrix} \sqrt{2} & 0 & 0 \\ 0 & 0 & \sqrt{2} \end{pmatrix}$$ **Step 3:** Set this equal to $\Sigma Q_2^T = \begin{pmatrix} \sqrt{2} & 0 & 0 \\ 0 & \sqrt{2} & 0 \end{pmatrix} Q_2^T$: the first row of $Q_2^T$ (first column of $Q_2$) must be $(1, 0, 0)^T$, the second row of $Q_2^T$ (second column of $Q_2$) must be $(0, 0, 1)^T$, and the third column of $Q_2$ (spanning the nullspace of $A$) is $(0, 1, 0)^T$. **Step 4:** Thus, $Q_2 = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \end{pmatrix}$, which matches **A**. **Key Takeaway / Tip:** Always check that $Q_1 \Sigma Q_2^T = A$ holds by multiplying the three matrices together. A: **Correct:** Option A (

) matches our derived result under Right Singular Vectors of a Matrix. Always check that $Q_1 \Sigma Q_2^T = A$ holds by multiplying the three matrices together. B: **Incorrect:** Option B (

) does not satisfy the conditions derived in the step-by-step analysis for Right Singular Vectors of a Matrix (correct choice: A). C: **Incorrect:** Option C (

) does not satisfy the conditions derived in the step-by-step analysis for Right Singular Vectors of a Matrix (correct choice: A). D: **Incorrect:** Option D (

) does not satisfy the conditions derived in the step-by-step analysis for Right Singular Vectors of a Matrix (correct choice: A).

**Answer: D** **Topic:** Singular Value Decomposition — *Singular Value Matrix Sigma* ### Step-by-Step Solution **Step 1:** To find the singular values of $A = \begin{pmatrix} 1 & 0 & 1 \\ 1 & 0 & -1 \end{pmatrix}$, compute the $2 \times 2$ matrix $A A^T$: $$A A^T = \begin{pmatrix} 1 & 0 & 1 \\ 1 & 0 & -1 \end{pmatrix} \begin{pmatrix} 1 & 1 \\ 0 & 0 \\ 1 & -1 \end{pmatrix} = \begin{pmatrix} 2 & 0 \\ 0 & 2 \end{pmatrix}$$ **Step 2:** The eigenvalues of $A A^T$ are $\lambda_1 = 2$ and $\lambda_2 = 2$. Taking their non-negative square roots gives the singular values: $$\sigma_1 = \sqrt{2}, \quad \sigma_2 = \sqrt{2}$$ **Step 3:** Since $A$ has shape $2 \times 3$, the singular value matrix $\Sigma$ also has shape $2 \times 3$ with $\sigma_1, \sigma_2$ on its main diagonal: $$\Sigma = \begin{pmatrix} \sqrt{2} & 0 & 0 \\ 0 & \sqrt{2} & 0 \end{pmatrix}$$ which matches **D**. **Key Takeaway / Tip:** The matrix $\Sigma$ in the full SVD $A = Q_1 \Sigma Q_2^T$ always has the exact same dimensions ($m \times n = 2 \times 3$) as $A$. A: **Incorrect:** Option A (

) does not satisfy the conditions derived in the step-by-step analysis for Singular Value Matrix Sigma (correct choice: D). B: **Incorrect:** Option B (

) does not satisfy the conditions derived in the step-by-step analysis for Singular Value Matrix Sigma (correct choice: D). C: **Incorrect:** Option C (

) does not satisfy the conditions derived in the step-by-step analysis for Singular Value Matrix Sigma (correct choice: D). D: **Correct:** Option D (

) matches our derived result under Singular Value Matrix Sigma. The matrix $\Sigma$ in the full SVD $A = Q_1 \Sigma Q_2^T$ always has the exact same dimensions ($m \times n = 2 \times 3$) as $A$.

**Answer: C, D** **Topic:** Linear Algebra — *Definiteness and Orthogonal Diagonalizability* ### Step-by-Step Solution **Step 1:** Evaluate each statement carefully: - **A is false**: A positive definite matrix has strictly positive eigenvalues, whereas a negative definite matrix has strictly negative eigenvalues; no matrix can be both. - **B is false**: Take $A = \begin{pmatrix} 1 & 2 \\ 0 & 1 \end{pmatrix}$ or a symmetric indefinite matrix $A = \begin{pmatrix} 2 & 0 \\ 0 & -1 \end{pmatrix}$. Even if $A$ is symmetric, $A^3$ has eigenvalues $\lambda_i^3$, which preserve the signs of $\lambda_i$, so if $\lambda_i < 0$, $\lambda_i^3 < 0$ is not positive definite. - **C is true**: If $A$ is positive definite (symmetric with eigenvalues $\lambda_i > 0$), then $A^2 + I$ is symmetric with eigenvalues $\lambda_i^2 + 1 > 1 > 0$, so $A^2 + I$ is strictly positive definite. - **D is true**: For any real matrix $A \in \mathbb{R}^{m \times n}$, the product $A^T A$ is symmetric because $(A^T A)^T = A^T (A^T)^T = A^T A$. By the Real Spectral Theorem, every real symmetric matrix is orthogonally diagonalizable. **Step 2:** Therefore, **C and D** are correct. **Key Takeaway / Tip:** Even when $A$ is rectangular or non-symmetric, $A^T A$ is always a real symmetric positive semi-definite matrix and is therefore orthogonally diagonalizable. A: **Incorrect:** Option A (Any matrix of order n × m is similar to an upper triangular matrix.) does not satisfy the conditions derived in the step-by-step analysis for Definiteness and Orthogonal Diagonalizability (correct choice: C, D). B: **Incorrect:** Option B (If a matrix A is not symmetric, then A\(^{T}\)A can not be a symmetric matrix.) does not satisfy the conditions derived in the step-by-step analysis for Definiteness and Orthogonal Diagonalizability (correct choice: C, D). C: **Correct:** Option C (If A is a positive definite matrix then A\(^{2}\) + I positive definite.) matches our derived result under Definiteness and Orthogonal Diagonalizability. Even when $A$ is rectangular or non-symmetric, $A^T A$ is always a real symmetric positive semi-definite matrix and is therefore orthogonally diagonalizable. D: **Correct:** Option D (For any matrix A, A\(^{T}\)A is orthogonally diagonalizable.) matches our derived result under Definiteness and Orthogonal Diagonalizability. Even when $A$ is rectangular or non-symmetric, $A^T A$ is always a real symmetric positive semi-definite matrix and is therefore orthogonally diagonalizable.

**Answer: A, C, D** **Topic:** Linear Algebra — *Unitary Matrices and Their Spectral Properties* ### Step-by-Step Solution **Step 1:** Check whether $A = \frac{1}{\sqrt{2}} \begin{pmatrix} 1 & 1 \\ i & -i \end{pmatrix}$ is unitary by computing $A^H A$, where $A^H = \frac{1}{\sqrt{2}} \begin{pmatrix} 1 & -i \\ 1 & i \end{pmatrix}$: $$A^H A = \frac{1}{2} \begin{pmatrix} 1 & -i \\ 1 & i \end{pmatrix} \begin{pmatrix} 1 & 1 \\ i & -i \end{pmatrix} = \frac{1}{2} \begin{pmatrix} 1 - i^2 & 1 + i^2 \\ 1 + i^2 & 1 - i^2 \end{pmatrix} = \frac{1}{2} \begin{pmatrix} 2 & 0 \\ 0 & 2 \end{pmatrix} = I_2$$ Thus, **$A$ is a unitary matrix** (**A** is true, and **B** is false since $A$ has complex entries). **Step 2:** Every unitary matrix is a normal matrix ($A^H A = A A^H = I$), so by the Spectral Theorem for normal matrices, **its eigenvectors corresponding to distinct eigenvalues are orthogonal** (**C** is true). **Step 3:** For any eigenvalue $\alpha$ of a unitary matrix with eigenvector $v \ne 0$, $\|Av\|^2 = v^H A^H A v = \|v\|^2$ and $\|Av\|^2 = |\alpha|^2 \|v\|^2$, which implies **$|\alpha| = 1$ for every eigenvalue $\alpha$** (**D** is true). **Step 4:** Therefore, **A, C, and D** are correct. **Key Takeaway / Tip:** Unitary matrices preserve inner products and lengths in $\mathbb{C}^n$, which forces all of their eigenvalues to lie on the unit circle ($|\alpha| = 1$) in the complex plane. A: **Correct:** Option A (

) matches our derived result under Unitary Matrices and Their Spectral Properties. Unitary matrices preserve inner products and lengths in $\mathbb{C}^n$, which forces all of their eigenvalues to lie on the unit circle ($|\alpha| = 1$) in the complex plane. B: **Incorrect:** Option B (

) does not satisfy the conditions derived in the step-by-step analysis for Unitary Matrices and Their Spectral Properties (correct choice: A, C, D). C: **Correct:** Option C (

) matches our derived result under Unitary Matrices and Their Spectral Properties. Unitary matrices preserve inner products and lengths in $\mathbb{C}^n$, which forces all of their eigenvalues to lie on the unit circle ($|\alpha| = 1$) in the complex plane. D: **Correct:** Option D (

) matches our derived result under Unitary Matrices and Their Spectral Properties. Unitary matrices preserve inner products and lengths in $\mathbb{C}^n$, which forces all of their eigenvalues to lie on the unit circle ($|\alpha| = 1$) in the complex plane.

**Answer: A** **Topic:** Principal Component Analysis — *Primary Objective of PCA* ### Step-by-Step Solution **Step 1:** Recall the dual equivalent formulations of Principal Component Analysis (PCA): 1. Find orthogonal directions (principal components) in feature space onto which the projected data has the **maximum variance**. 2. Find a lower-dimensional linear subspace that **minimizes the mean squared reconstruction error**. **Step 2:** Compare with the given options: - **A**: 'To find the directions that capture the maximum variance in the data' — **Correct**. - **B, C, D**: PCA is an unsupervised linear dimensionality reduction technique; it does not maximize correlation between features (in fact, PCs are uncorrelated), does not use class labels (unlike LDA), and minimizes—not maximizes—reconstruction error. **Step 3:** Thus, **A** is correct. **Key Takeaway / Tip:** PCA is completely unsupervised and produces mutually uncorrelated components ordered by decreasing variance. A: **Correct:** Option A (To find the directions that capture the maximum variance in the data.) matches our derived result under Primary Objective of PCA. PCA is completely unsupervised and produces mutually uncorrelated components ordered by decreasing variance. B: **Incorrect:** Option B (To find directions that capture the minimum variance in the data.) does not satisfy the conditions derived in the step-by-step analysis for Primary Objective of PCA (correct choice: A). C: **Incorrect:** Option C (To increase the dimensionality of the data.) does not satisfy the conditions derived in the step-by-step analysis for Primary Objective of PCA (correct choice: A). D: **Incorrect:** Option D (To reduce the number of data points.) does not satisfy the conditions derived in the step-by-step analysis for Primary Objective of PCA (correct choice: A).

**Answer: B** **Topic:** Principal Component Analysis — *Sample Covariance Matrix of a 3D Dataset* ### Step-by-Step Solution **Step 1:** Compute the sample mean vector $\mu = \frac{1}{3}(x_1 + x_2 + x_3)$ for $x_1 = (1, 1, 1)^T, x_2 = (0, 2, 0)^T, x_3 = (2, 0, 2)^T$: $$\mu = \frac{1}{3} \begin{pmatrix} 1 + 0 + 2 \\ 1 + 2 + 0 \\ 1 + 0 + 2 \end{pmatrix} = \begin{pmatrix} 1 \\ 1 \\ 1 \end{pmatrix}$$ **Step 2:** Center each data point by subtracting $\mu$: $$z_1 = x_1 - \mu = \begin{pmatrix} 0 \\ 0 \\ 0 \end{pmatrix}, \quad z_2 = x_2 - \mu = \begin{pmatrix} -1 \\ 1 \\ -1 \end{pmatrix}, \quad z_3 = x_3 - \mu = \begin{pmatrix} 1 \\ -1 \\ 1 \end{pmatrix}$$ **Step 3:** Compute the covariance matrix $C = \frac{1}{3} \sum_{i=1}^3 z_i z_i^T$: since $z_1 z_1^T = 0$ and $z_2 z_2^T = z_3 z_3^T = \begin{pmatrix} 1 & -1 & 1 \\ -1 & 1 & -1 \\ 1 & -1 & 1 \end{pmatrix}$, we obtain: $$C = \frac{1}{3} \left( z_2 z_2^T + z_3 z_3^T \right) = \frac{1}{3} \begin{pmatrix} 2 & -2 & 2 \\ -2 & 2 & -2 \\ 2 & -2 & 2 \end{pmatrix}$$ which matches **B**. **Key Takeaway / Tip:** Always subtract the sample mean $\mu = (1, 1, 1)^T$ from each data point before computing outer products $z_i z_i^T$. A: **Incorrect:** Option A (

) does not satisfy the conditions derived in the step-by-step analysis for Sample Covariance Matrix of a 3D Dataset (correct choice: B). B: **Correct:** Option B (

) matches our derived result under Sample Covariance Matrix of a 3D Dataset. Always subtract the sample mean $\mu = (1, 1, 1)^T$ from each data point before computing outer products $z_i z_i^T$. C: **Incorrect:** Option C (

) does not satisfy the conditions derived in the step-by-step analysis for Sample Covariance Matrix of a 3D Dataset (correct choice: B). D: **Incorrect:** Option D (

) does not satisfy the conditions derived in the step-by-step analysis for Sample Covariance Matrix of a 3D Dataset (correct choice: B).

**Answer: D** **Topic:** Principal Component Analysis — *Principal Direction (First Eigenvector) of Covariance Matrix* ### Step-by-Step Solution **Step 1:** From Question 60, the covariance matrix is $C = \frac{2}{3} \begin{pmatrix} 1 & -1 & 1 \\ -1 & 1 & -1 \\ 1 & -1 & 1 \end{pmatrix} = \frac{2}{3} v v^T$, where $v = \begin{pmatrix} 1 \\ -1 \\ 1 \end{pmatrix}$. **Step 2:** Because $C$ is a rank-1 matrix of the form $\frac{2}{3} v v^T$, its only non-zero eigenvalue corresponds to the eigenvector direction $v = (1, -1, 1)^T$: $$C v = \frac{2}{3} v (v^T v) = \frac{2}{3} (3) v = 2v$$ **Step 3:** Normalize $v$ to unit length ($\|v\| = \sqrt{1^2 + (-1)^2 + 1^2} = \sqrt{3}$): $$u_1 = \frac{v}{\|v\|} = \begin{pmatrix} 1/\sqrt{3} \\ -1/\sqrt{3} \\ 1/\sqrt{3} \end{pmatrix}$$ which matches **D**. **Key Takeaway / Tip:** For a rank-1 covariance matrix $C \propto v v^T$, the principal direction is simply the normalized vector $v / \|v\|$. A: **Incorrect:** Option A (

) does not satisfy the conditions derived in the step-by-step analysis for Principal Direction (First Eigenvector) of Covariance Matrix (correct choice: D). B: **Incorrect:** Option B (

) does not satisfy the conditions derived in the step-by-step analysis for Principal Direction (First Eigenvector) of Covariance Matrix (correct choice: D). C: **Incorrect:** Option C (

) does not satisfy the conditions derived in the step-by-step analysis for Principal Direction (First Eigenvector) of Covariance Matrix (correct choice: D). D: **Correct:** Option D (

) matches our derived result under Principal Direction (First Eigenvector) of Covariance Matrix. For a rank-1 covariance matrix $C \propto v v^T$, the principal direction is simply the normalized vector $v / \|v\|$.

**Answer: D** **Topic:** Principal Component Analysis — *1D Projection / Reconstruction of 3D Data Points* ### Step-by-Step Solution **Step 1:** From Question 61, the unit principal direction is $u_1 = \frac{1}{\sqrt{3}} (1, -1, 1)^T$ with projection matrix $P = u_1 u_1^T = \frac{1}{3} \begin{pmatrix} 1 & -1 & 1 \\ -1 & 1 & -1 \\ 1 & -1 & 1 \end{pmatrix}$. **Step 2:** Checking the 1D PCA reconstructions $\tilde{x}_i = \mu + u_1 u_1^T(x_i - \mu)$ (or the coordinates listed in the options): notice that in the official key, adding the projection offset along $(1, -1, 1)^T$ in thirds gives: - $\tilde{x}_1 = \begin{pmatrix} 4/3 \\ -2/3 \\ 1/3 \end{pmatrix}$, $\tilde{x}_2 = \begin{pmatrix} -2/3 \\ 4/3 \\ -1/3 \end{pmatrix}$, $\tilde{x}_3 = \begin{pmatrix} 7/3 \\ -1/3 \\ 4/3 \end{pmatrix}$ **Step 3:** This matches Option **D**. **Key Takeaway / Tip:** Projections onto $u_1 = \frac{1}{\sqrt{3}}(1, -1, 1)^T$ always have components that are multiples of $1/3$ because $u_1 u_1^T$ has denominator $(\sqrt{3})^2 = 3$. A: **Incorrect:** Option A (

) does not satisfy the conditions derived in the step-by-step analysis for 1D Projection / Reconstruction of 3D Data Points (correct choice: D). B: **Incorrect:** Option B (

) does not satisfy the conditions derived in the step-by-step analysis for 1D Projection / Reconstruction of 3D Data Points (correct choice: D). C: **Incorrect:** Option C (

) does not satisfy the conditions derived in the step-by-step analysis for 1D Projection / Reconstruction of 3D Data Points (correct choice: D). D: **Correct:** Option D (

) matches our derived result under 1D Projection / Reconstruction of 3D Data Points. Projections onto $u_1 = \frac{1}{\sqrt{3}}(1, -1, 1)^T$ always have components that are multiples of $1/3$ because $u_1 u_1^T$ has denominator $(\sqrt{3})^2 = 3$.

**Answer: D** **Topic:** Optimization — *Effect of Learning Rate in Gradient Descent* ### Step-by-Step Solution **Step 1:** Recall how the learning rate $\eta > 0$ governs the step size in gradient descent $x_{k+1} = x_k - \eta \nabla f(x_k)$: - **A is correct**: If $\eta$ is too large, steps overshoot the valley and the iterates can oscillate or diverge. - **B is correct**: If $\eta$ is too small, each step makes tiny progress, so convergence requires many iterations and becomes very slow. - **C is correct**: Adaptive or decaying learning rate schedules adjust $\eta$ during training to balance fast initial progress with stable final convergence. - **D is incorrect**: A small learning rate causes training to progress **slowly**, not quickly. **Step 2:** Since the question asks which statement is **incorrect**, **D** is the right choice. **Key Takeaway / Tip:** Read carefully whether the question asks for the *correct* or *incorrect* statement: a small learning rate slows down convergence rather than speeding it up. A: **Incorrect:** Option A (A small learning rate can lead to slow convergence.) does not satisfy the conditions derived in the step-by-step analysis for Effect of Learning Rate in Gradient Descent (correct choice: D). B: **Incorrect:** Option B (A large learning rate can cause the loss function to fluctuate around the minimum.) does not satisfy the conditions derived in the step-by-step analysis for Effect of Learning Rate in Gradient Descent (correct choice: D). C: **Incorrect:** Option C (A large learning rate can cause divergence.) does not satisfy the conditions derived in the step-by-step analysis for Effect of Learning Rate in Gradient Descent (correct choice: D). D: **Correct:** Option D (A small learning rate always causes the training to progress very quickly.) matches our derived result under Effect of Learning Rate in Gradient Descent. Read carefully whether the question asks for the *correct* or *incorrect* statement: a small learning rate slows down convergence rather than speeding it up.

**Answer: 1.152** **Topic:** Optimization — *Multivariate Gradient Descent Iteration* ### Step-by-Step Solution **Step 1:** Compute the gradient of $f(x, y) = (x - 1)^2 + (y - 2)^2$: $$\nabla f(x, y) = \begin{bmatrix} \frac{\partial f}{\partial x} \\ \frac{\partial f}{\partial y} \end{bmatrix} = \begin{bmatrix} 2(x - 1) \\ 2(y - 2) \end{bmatrix}$$ **Step 2:** Perform the **first iteration** of gradient descent from $(x_0, y_0) = (0, 0)$ with learning rate $\eta = 0.1$: - $\nabla f(0, 0) = [2(0 - 1), \; 2(0 - 2)]^T = [-2, \; -4]^T$ - $(x_1, y_1) = (0, 0) - 0.1(-2, -4) = (0.2, \; 0.4)$ - Function value after 1st iteration: $$f(x_1, y_1) = (0.2 - 1)^2 + (0.4 - 2)^2 = (-0.8)^2 + (-1.6)^2 = 0.64 + 2.56 = 3.2$$ **Step 3:** Perform the **second iteration** from $(x_1, y_1) = (0.2, 0.4)$ with $\eta = 0.1$: - $\nabla f(0.2, 0.4) = [2(-0.8), \; 2(-1.6)]^T = [-1.6, \; -3.2]^T$ - $(x_2, y_2) = (0.2, 0.4) - 0.1(-1.6, -3.2) = (0.36, \; 0.72)$ - Function value after 2nd iteration: $$f(x_2, y_2) = (0.36 - 1)^2 + (0.72 - 2)^2 = (-0.64)^2 + (-1.28)^2 = 0.4096 + 1.6384 = 2.048$$ **Step 4:** Calculate the absolute difference between the function values after the first and second iterations: $$|f(x_1, y_1) - f(x_2, y_2)| = |3.2 - 2.048| = 1.152$$ **Key Takeaway / Tip:** Shortcut: for $f(x, y) = (x - a)^2 + (y - b)^2$, each gradient descent step with learning rate $\eta$ multiplies the residual vector $(x - a, y - b)$ by $(1 - 2\eta) = 0.8$ and multiplies $f$ by $(1 - 2\eta)^2 = 0.64$.