Machine Learning Techniques · End Term · Jan 2026 · FN

**1. Understand the dataset and the maximum-margin weight vector \(w^*\):** We have two points in \(\mathbb{R}^2\): - Positive point: \(x_1 = (1, 0)^T\) with label \(y_1 = +1\) - Negative point: \(x_2 = (-1, 0)^T\) with label \(y_2 = -1\) (or \(0\) for logistic regression) By symmetry, the separating hyperplane with the largest margin passes through the origin along the \(y\)-axis (\(x^{(1)} = 0\)), and its normal vector points along the positive \(x\)-axis: \(w^* = (c, 0)^T\) with \(c > 0\) (specifically \(w^* = (1, 0)^T\) for hard-margin SVM). Thus, all three classifiers output a weight vector proportional to \(w^*\) (same direction and decision boundary \(x^{(1)} = 0\)) if and only if their weight vector is a positive multiple of \((1, 0)^T\). **2. Check each of the three classifiers:** - **Hard-margin SVM:** Minimizes \(\frac{1}{2}\|w\|^2\) subject to \(w_1(1) \ge 1\) and \(-w_1(-1) \ge 1\), giving \(w_{\text{SVM}} = (1, 0)^T = w^*\). - **Perceptron (1 iteration from \(w^{(0)} = (0, 0)^T\) with \(\eta = 1\)):** - On \(x_1 = (1, 0)^T\), \(w^{(0)T} x_1 = 0\) (misclassified), so \(w \leftarrow (0,0)^T + 1 \cdot (1,0)^T = (1, 0)^T\). - On \(x_2 = (-1, 0)^T\), \(y_2 (w^T x_2) = (-1)(-1) = +1 > 0\) (correctly classified, no update). - After 1 iteration, \(w^{(1)}_{\text{perc}} = (1, 0)^T = w^*\) (or if batch update is used: \((1,0)^T + (-1)(-1,0)^T = (2,0)^T\), which defines the exact same classifier!). - **Logistic Regression (1 iteration of gradient ascent from \(w^{(0)} = (0, 0)^T\) with \(\eta = 1\)):** - At \(w^{(0)} = (0, 0)^T\), \(\sigma(w^{(0)T} x_i) = \sigma(0) = 0.5\) for both points. - The gradient of log-likelihood is \(\sum_i (y_i - \sigma(w^T x_i)) x_i = (1 - 0.5)(1, 0)^T + (0 - 0.5)(-1, 0)^T = 0.5(1, 0)^T + 0.5(1, 0)^T = (1, 0)^T\). - After 1 step with \(\eta = 1\), \(w^{(1)}_{\text{LR}} = (0, 0)^T + 1 \cdot (1, 0)^T = (1, 0)^T = w^*\)! **3. Conclude:** All three classifiers output \(w^* = (1, 0)^T\)! **Answer: A — All three classifiers output w^*.** A: **Correct:** SVM gives \(w^* = (1, 0)^T\), one pass of Perceptron from \((0,0)\) gives \((1, 0)^T\), and one step of gradient ascent on Logistic Regression from \((0,0)\) with \(\eta = 1\) also gives \(0.5(1,0)^T - 0.5(-1,0)^T = (1, 0)^T = w^*\). B: **Incorrect:** Both Perceptron and Logistic Regression also reach \(w^* = (1, 0)^T\) after one iteration. C: **Incorrect:** Logistic regression with \(\eta = 1\) starting from \((0, 0)\) has gradient \((1 - 0.5)(1,0)^T + (0 - 0.5)(-1,0)^T = (1, 0)^T\), so it also outputs \(w^*\). D: **Incorrect:** Hard-margin SVM directly finds the maximum-margin weight vector \(w^* = (1, 0)^T\) as well.

**1. Understand the condition for a linear separator through the origin:** We are given the dataset \(\mathcal{D}\) in \(\mathbb{R}^2\): - Positive class (\(y = +1\)): \(x_1 = [1, 1]^T\) and \(x_2 = [2, 1]^T\) - Negative class (\(y = -1\)): \(x_3 = [1, -1]^T\) and \(x_4 = [2, -1]^T\) A weight vector \(w = [w_1, w_2]^T\) correctly separates the two classes if \(y_i (w^T x_i) > 0\) for all four points (i.e., \(w^T x > 0\) for \(y = +1\) and \(w^T x < 0\) for \(y = -1\)). **2. Test each candidate weight vector \(w\):** - **Option A: \(w = [0, 1]^T\)** - \(w^T [1, 1]^T = 1 > 0\) and \(w^T [2, 1]^T = 1 > 0\) (both positive points get \(> 0\)). - \(w^T [1, -1]^T = -1 < 0\) and \(w^T [2, -1]^T = -1 < 0\) (both negative points get \(< 0\)). - Thus, \(w = [0, 1]^T\) **correctly separates** the classes! - **Option B: \(w = [1, 0]^T\)** - \(w^T [1, -1]^T = 1 > 0\), which misclassifies the negative point \([1, -1]^T\) as positive. - **Option C: \(w = [1, 5]^T\)** - For positive points: \(1(1) + 5(1) = 6 > 0\) and \(1(2) + 5(1) = 7 > 0\). - For negative points: \(1(1) + 5(-1) = -4 < 0\) and \(1(2) + 5(-1) = -3 < 0\). - Thus, \(w = [1, 5]^T\) **correctly separates** the classes! - **Option D: \(w = [1, 1]^T\)** - For negative point \([1, -1]^T\): \(w^T [1, -1]^T = 1 - 1 = 0\) (lies on the boundary), and for \([2, -1]^T\): \(w^T [2, -1]^T = 2 - 1 = +1 > 0\) (misclassified). **Answer: A, C — w = [0, 1]^T and w = [1, 5]^T.** A: **Correct:** With \(w = [0, 1]^T\), \(w^T x = x_2\), which is \(+1 > 0\) for both positive points and \(-1 < 0\) for both negative points. B: **Incorrect:** With \(w = [1, 0]^T\), \(w^T x = x_1 > 0\) for all four points, so both negative points are misclassified. C: **Correct:** With \(w = [1, 5]^T\), the positive points give \(6, 7 > 0\) and the negative points give \(-4, -3 < 0\), separating both classes. D: **Incorrect:** With \(w = [1, 1]^T\), the negative point \([2, -1]^T\) gives \(w^T x = 2 - 1 = 1 > 0\) and \([1, -1]^T\) gives \(0\), failing to separate the classes.

**1. Understand the effect of exact multicollinearity (\(f_4 = 2f_1 + 3f_2\)):** Because the fourth feature is an exact linear combination of the first two features (\(f_4 - 2f_1 - 3f_2 = 0\)), the columns of the design matrix \(X\) are linearly dependent. This means \(X^T X\) is singular (not invertible) and has a non-trivial null space containing \(v = [-2, -3, 0, 1]^T\). **2. Analyze each statement:** - **Statement A:** Unregularized Ordinary Least Squares (OLS) directly minimizes the training Mean Squared Error (MSE) over all \(w \in \mathbb{R}^4\) without any penalty. Any other weight vector (including the ridge regression solution \(w_{\text{ridge}}\)) cannot achieve a strictly smaller value of the unpenalized training MSE than the global minimum of the training MSE itself! Thus, \(\text{MSE}_{\text{train}}(w_{\text{OLS}}) \le \text{MSE}_{\text{train}}(w_{\text{ridge}})\). (Correct!) - **Statement B:** Because \(X v = 0\) for \(v = [-2, -3, 0, 1]^T\), if \(w^*\) is an OLS solution, then \(w^* + c v\) gives the exact same predictions \(X(w^* + cv) = Xw^*\) and the exact same minimum training MSE for any scalar \(c \in \mathbb{R}\). Thus, the unregularized OLS solution is not unique. (Correct!) - **Statement C:** Ridge regression minimizes \(\|y - Xw\|^2 + \lambda \|w\|_2^2\). Rather than setting a collinear feature's weight to zero, the L2 penalty distributes weight across correlated features (Specifically, \(w_{\text{ridge}}\) is orthogonal to the null space of \(X\), so \(w_4 = 2w_1 + 3w_2\), which is non-zero in general). (Incorrect.) - **Statement D:** Lasso regression uses an L1 penalty \(\lambda \|w\|_1\), whose diamond-shaped constraint region has sharp corners along the coordinate axes, allowing it to set redundant or weak feature weights to **exactly zero**. (Correct!) **Answer: A, B, D — Unregularized linear regression achieves training MSE <= ridge regression, its solution may not be unique, and Lasso can force feature weights to exactly zero.** A: **Correct:** Unregularized linear regression minimizes training MSE directly; adding a ridge penalty \(\lambda \|w\|_2^2\) can only keep the training MSE the same or increase it. B: **Correct:** Because \(f_4 = 2f_1 + 3f_2\), \(X\) has linearly dependent columns, so \(X^T X\) is singular and infinitely many weight vectors achieve the minimum training MSE. C: **Incorrect:** L2 regularization shrinks weights smoothly and shares weight among collinear features rather than setting coefficients to zero. D: **Correct:** The non-differentiable corners of the L1 norm penalty \(\|w\|_1\) in Lasso induce sparsity by driving redundant coefficients to exactly zero.

**1. Understand the behavior of the Perceptron algorithm on linearly separable data:** When a binary classification dataset is linearly separable, there are infinitely many separating hyperplanes (here, two distinct separating weight vectors \(w_1\) and \(w_2\) are given). **2. Evaluate each statement:** - **Statement A:** The Perceptron stops as soon as it finds *any* hyperplane that classifies all training points correctly. It is not guaranteed to converge specifically to \(w_1\) (it could converge to \(w_2\) or another separator). (Incorrect.) - **Statement B:** Because Perceptron only updates when a mistake occurs on the current point and stops as soon as zero mistakes are made in a pass, the final weight vector depends on the initial weight vector \(w^{(0)}\) and the order in which training examples are presented. (Correct!) - **Statement C:** By the **Perceptron Convergence Theorem** (Novikoff), whenever the training dataset is linearly separable with a positive margin, the Perceptron algorithm is guaranteed to make at most \((R/\gamma)^2\) updates and converge to *some* separating hyperplane in a finite number of steps. (Correct!) - **Statement D:** Finding the maximum-margin separator is a property of hard-margin **Support Vector Machines (SVMs)** (and gradient descent on logistic/exponential loss asymptotically), not the standard Perceptron algorithm. (Incorrect.) **Answer: B, C — The final weight vector depends on initialization and presentation order, and the Perceptron is guaranteed to converge to some separating hyperplane in a finite number of updates.** A: **Incorrect:** Many separating hyperplanes exist, and Perceptron stops at whichever valid separator it reaches first, not necessarily \(w_1\). B: **Correct:** Changing the initial weights or shuffling the order of training points changes which mistakes occur and thus alters the final separating hyperplane. C: **Correct:** By the Perceptron Convergence Theorem, linear separability guarantees convergence to a separating hyperplane in a finite number of updates. D: **Incorrect:** The Perceptron stops as soon as all points are classified on the correct side of the boundary, regardless of how small the resulting margin is; SVMs maximize the margin.

**1. Identify the given training data and optimal Lagrange multipliers:** We are given 4 training points in \(\mathbb{R}^2\): - \(x_1 = (1, 2),\; y_1 = +1\) with \(\alpha_1 = 2\) - \(x_2 = (2, 2),\; y_2 = +1\) with \(\alpha_2 = 0\) - \(x_3 = (2, -1),\; y_3 = -1\) with \(\alpha_3 = 2\) - \(x_4 = (1, -2),\; y_4 = -1\) with \(\alpha_4 = 0\) **2. Compute the optimal weight vector \(w^*\):** In the dual formulation of a hard-margin SVM (without bias, as indicated by \(\hat{y} = \text{sign}((w^*)^T x_{\text{test}})\)), the optimal weight vector is: \[w^* = \sum_{i=1}^{4} \alpha_i y_i x_i\] Substituting the non-zero multipliers (\(\alpha_1 = 2\) and \(\alpha_3 = 2\)): \[w^* = 2(+1)\begin{bmatrix} 1 \\ 2 \end{bmatrix} + 0 + 2(-1)\begin{bmatrix} 2 \\ -1 \end{bmatrix} + 0 = \begin{bmatrix} 2 \\ 4 \end{bmatrix} - \begin{bmatrix} 4 \\ -2 \end{bmatrix} = \begin{bmatrix} -2 \\ 6 \end{bmatrix}\] **3. Predict the label for \(x_{\text{test}} = (1, 1)\):** Compute the inner product \((w^*)^T x_{\text{test}}\): \[(w^*)^T x_{\text{test}} = (-2)(1) + (6)(1) = 4 > 0\] Taking the sign gives \(\hat{y} = \text{sign}(4) = +1 = 1\). **Answer: 1.**

**1. Find the linear score threshold corresponding to \(P(y = +1 \mid x) > 0.7\):** The logistic regression model predicts \(P(y = +1 \mid x) = \frac{1}{1 + e^{-w^T x}}\). For a point to be classified as \(+1\), we require: \[\frac{1}{1 + e^{-w^T x}} > 0.7 = \frac{7}{10}\] Taking reciprocals and solving for \(w^T x\): \[1 + e^{-w^T x} < \frac{10}{7} \implies e^{-w^T x} < \frac{3}{7} \implies e^{w^T x} > \frac{7}{3} \approx 2.333\] Taking the natural logarithm gives the threshold on \(w^T x\): \[w^T x > \ln\left(\frac{7}{3}\right) \approx 0.8473\] **2. Check each of the 5 given datapoints:** We are given the values of \(w^T x\) for the 5 points: - \(x_1: w^T x_1 = -1.2 < 0.8473\) (Class \(-1\)) - \(x_2: w^T x_2 = 0.0 < 0.8473\) (Class \(-1\)) - \(x_3: w^T x_3 = 2.1 > 0.8473\) (Class \(+1\), since \(\frac{1}{1 + e^{-2.1}} \approx 0.8909 > 0.7\)) - \(x_4: w^T x_4 = -0.4 < 0.8473\) (Class \(-1\)) - \(x_5: w^T x_5 = 0.8 < 0.8473\) (Class \(-1\), since \(\frac{1}{1 + e^{-0.8}} \approx 0.68997 < 0.7\)) **3. Count how many datapoints are classified as \(+1\):** Only \(1\) datapoint (\(x_3\), with \(w^T x_3 = 2.1\)) exceeds the threshold \(0.8473\). **Answer: 1.**

**1. Recall the Beta-Binomial conjugate update rule:** Suppose the probability of heads \(\theta\) has a prior distribution \(\text{Beta}(\alpha, \beta)\). When we flip the coin \(N\) times and observe \(H\) heads and \(T = N - H\) tails, the posterior distribution of \(\theta\) is: \[\theta \mid D \sim \text{Beta}(\alpha + H,\; \beta + T)\] **2. Substitute the given prior and observed counts:** We are given: - Prior parameters: \(\alpha = 10\), \(\beta = 40\) - Observations from \(N = 100\) flips: \(H = 60\) heads and \(T = 100 - 60 = 40\) tails Updating the parameters gives the posterior distribution: \[\alpha_{\text{post}} = 10 + 60 = 70, \quad \beta_{\text{post}} = 40 + 40 = 80\] **3. Calculate the posterior mean:** Using the formula for the mean of a Beta distribution \(E[\theta] = \frac{\alpha_{\text{post}}}{\alpha_{\text{post}} + \beta_{\text{post}}}\): \[E[\theta \mid D] = \frac{70}{70 + 80} = \frac{70}{150} = \frac{7}{15} \approx 0.46667 \approx 0.467\] **Answer: 0.467.**

**1. Understand which points have non-zero slack variables (\(\xi_i > 0\)) in a soft-margin SVM:** In a soft-margin SVM, the margin boundaries are \(w^T x + b = +1\) (for the positive class, shown in green) and \(w^T x + b = -1\) (for the negative class, shown in red). A training point \((x_i, y_i)\) incurs a penalty (\(\xi_i = \max(0, 1 - y_i(w^T x_i + b)) > 0\)) if and only if it violates its class margin—that is, if it lies **strictly inside the margin band** (between the two dotted lines) or **on the wrong side** of its margin boundary. **2. Count the margin-violating points from the diagram:** Looking at the soft-margin SVM plot: - Inside the margin band (between \(w^T x + b = -1\) and \(w^T x + b = +1\)) or across the boundary into the opposite region, there are: - Red (negative class) points that lie above the lower dotted line \(w^T x + b = -1\) (including points inside the margin and on the positive side of the boundary). - Green (positive class) points that lie below the upper dotted line \(w^T x + b = +1\) (including points inside the margin and on the negative side of the boundary). Counting all such points that strictly violate their respective margin boundary \(y_i(w^T x_i + b) < 1\) gives exactly \(5\) datapoints. **Answer: 5.**

**1. Recall the KKT complementary slackness condition for \(\alpha_i = 0\):** In the dual formulation of a soft-margin SVM, each training point \(i\) satisfies: \[\alpha_i \big(y_i(w^T x_i + b) - 1 + \xi_i\big) = 0\] If a datapoint lies **strictly outside the margin** on the correct side—meaning \(y_i(w^T x_i + b) > 1\) (and therefore \(\xi_i = 0\))—then the term \(y_i(w^T x_i + b) - 1 + \xi_i\) is strictly positive. By complementary slackness, its dual multiplier **must** be \(\alpha_i = 0\) (it is not a support vector). **2. Count the non-support vectors from the diagram:** Looking at the SVM diagram: - In the upper-left region strictly above the positive margin line \(w^T x + b = +1\), there are \(3\) green (positive class) points. - In the lower-right region strictly below the negative margin line \(w^T x + b = -1\), there are \(3\) red (negative class) points. All other points lie either directly on the dotted margin lines (\(y_i(w^T x_i + b) = 1\)) or inside/across the margin (\(\xi_i > 0 \implies \alpha_i = C > 0\)). **3. Sum the points that definitely have \(\alpha_i = 0\):** \[3 + 3 = 6\] **Answer: 6.**

**1. Read the coordinates of the test point and its nearest neighbors from the 2D diagram:** From the given plot, the test point \(x_{\text{test}}\) is located on the horizontal line \(y = 1\) at \(x = 2\), i.e., \(x_{\text{test}} = (2, 1)\). Along the same horizontal line \(y = 1\), we see the closest data points: - \(x_1 = (0, 1)\) (Red, negative class \(-1\)): Euclidean distance \(= |2 - 0| = 2\). - \(x_2 = (1, 1)\) (Red, negative class \(-1\)): Euclidean distance \(= |2 - 1| = 1\). - \(x_3 = (4, 1)\) (Green, positive class \(+1\)): Euclidean distance \(= |4 - 2| = 2\). All other green points (at \(y = 2, 4, 5, 6\) with \(x \ge 5\)) are at distance \(\ge \sqrt{(5-2)^2 + (2-1)^2} = \sqrt{10} > 3\). **2. Apply the \(K\)-Nearest Neighbors rule with \(K = 3\):** The \(K = 3\) closest neighbors to \(x_{\text{test}} = (2, 1)\) are \(x_2 = (1, 1)\) at distance \(1\), \(x_1 = (0, 1)\) at distance \(2\), and \(x_3 = (4, 1)\) at distance \(2\). Their class labels are: - \(x_1\): \(-1\) (Red) - \(x_2\): \(-1\) (Red) - \(x_3\): \(+1\) (Green) **3. Take the majority vote:** Among the 3 nearest neighbors, \(2\) belong to class \(-1\) and \(1\) belongs to class \(+1\). By majority voting, the predicted class of \(x_{\text{test}}\) is \(-1\). **Answer: B — -1.** A: **Incorrect:** Only 1 of the 3 nearest neighbors (\(x_3\)) belongs to the positive class \(+1\). B: **Correct:** The 3 nearest neighbors to \(x_{\text{test}}\) are \(x_2\) (class \(-1\)), \(x_1\) (class \(-1\)), and \(x_3\) (class \(+1\)); majority voting (2 vs. 1) predicts \(-1\). C: **Incorrect:** Since \(K = 3\) is odd and the labels are \(+1\) or \(-1\), a tie is impossible (the vote is 2 to 1). D: **Incorrect:** All distances are clearly defined from the plot coordinates, uniquely determining the 3 nearest neighbors.

**1. Analyze how \(K\)-NN classifies \(x_{\text{test}}\) with \(K = 3\):** As found in the previous question, the \(3\) nearest neighbors of \(x_{\text{test}}\) are \(x_1, x_2\) from the negative class (\(-1\), red) and \(x_3\) from the positive class (\(+1\), green). **2. Evaluate each statement:** - **Statement A:** False, because out of the 3 nearest neighbors, 2 are negative and only 1 is positive. - **Statement B:** True, because 2 out of the 3 nearest neighbors belong to the negative class (\(-1\)), so negative class points dominate the local neighborhood of \(x_{\text{test}}\). - **Statement C:** False, because the negative points \(x_1, x_2\) to the left are closer than the remaining positive points to the upper right. - **Statement D:** True, because standard unweighted \(K\)-NN classification assigns the label by taking the majority vote among the \(K\) nearest neighbors. **Answer: B, D — The nearest neighbors are dominated by negative class points, and the classification is determined by majority voting among the nearest neighbors.** A: **Incorrect:** Only 1 of the 3 nearest neighbors is positive, while 2 are negative. B: **Correct:** Two of the three closest points (\(x_1\) and \(x_2\)) belong to the negative class \(-1\), dominating the \(K=3\) neighborhood. C: **Incorrect:** The negative points on the left are closer to \(x_{\text{test}}\) than almost all positive points. D: **Correct:** In \(K\)-NN classification, the predicted class is chosen by majority vote across the \(K\) nearest neighbors.

**1. Compare the parent node and child splits in Tree 1 and Tree 2:** Both Tree 1 and Tree 2 start with the exact same root (parent) dataset of \(500\) examples: \(300\) from class \(+1\) and \(200\) from class \(-1\). Therefore, both trees have the exact same parent entropy \(H(\text{root})\). Recall that Information Gain is: \[\text{IG} = H(\text{root}) - H(\text{children})\] Thus, whichever tree has the **lower weighted child entropy** \(H(\text{children})\) (i.e., purer child nodes) has the **higher information gain**. **2. Examine the child node distributions:** - **In Tree 1:** - Left child (`Y`): \(+1: 100,\; -1: 200\) (total \(300\), proportions \(\frac{1}{3}\) and \(\frac{2}{3}\), so \(H_{\text{left}, 1} = H(1/3, 2/3) \approx 0.9183\)). - Right child (`N`): Since the root had \(300\) of \(+1\) and \(200\) of \(-1\), the remaining examples in the right child are \(+1: 200,\; -1: 0\) (total \(200\), **completely pure**, so \(H_{\text{right}, 1} = 0\)). - Weighted child entropy of Tree 1: \(\frac{300}{500}(0.9183) + \frac{200}{500}(0) \approx 0.551\). - **In Tree 2:** - Left child (`Y`): \(+1: 150,\; -1: 200\) (total \(350\), proportions \(\frac{3}{7}\) and \(\frac{4}{7}\), which is closer to \(50/50\) and thus *more impure* than \(1/3, 2/3\): \(H_{\text{left}, 2} = H(3/7, 4/7) \approx 0.9852\)). - Right child (`N`): Remaining \(150\) examples are \(+1: 150,\; -1: 0\) (total \(150\), pure, \(H_{\text{right}, 2} = 0\)). - Weighted child entropy of Tree 2: \(\frac{350}{500}(0.9852) + 0 \approx 0.690\). **3. Conclude:** Tree 1 puts more samples into the completely pure right leaf (\(200\) vs. \(150\)) and leaves a purer left node (\(1:2\) ratio vs. \(1.5:2\) ratio). Thus, Tree 1 has lower conditional entropy and **higher information gain** than Tree 2. **Answer: A — Tree 1 has higher information gain than Tree 2.** A: **Correct:** Tree 1 has a smaller and purer impure child (\(100\) vs \(200\), weighted entropy \(\approx 0.551\)) compared to Tree 2 (\(150\) vs \(200\), weighted entropy \(\approx 0.690\)), yielding higher information gain. B: **Incorrect:** Tree 2's left child is both larger (\(350\) vs \(300\)) and closer to a 50-50 split (\(150\) vs \(200\)), giving higher child entropy and lower information gain. C: **Incorrect:** The two splits produce different class proportions in the child nodes, so their information gains are not equal. D: **Incorrect:** Information gain of a split depends only on the class counts in the parent and immediate child nodes, which are fully determined.

**1. Compare Bagging and Boosting in ensemble learning:** - **Bagging (Bootstrap Aggregating):** Trains multiple base models (typically deep, low-bias, high-variance trees) independently in parallel on bootstrap samples and averages their predictions. Averaging independent predictions cancels out random fluctuations, primarily **reducing variance**. - **Boosting (e.g., AdaBoost, Gradient Boosting):** Trains weak base models (typically shallow, high-bias, low-variance decision stumps/trees) sequentially, where each new model focuses on correcting the residual errors (or misclassified examples) of the previous ensemble. Combining weak learners sequentially makes the overall model more expressive, primarily **reducing bias**. **Answer: A — Bagging reduces variance, Boosting reduces bias.** A: **Correct:** Bagging averages parallel bootstrap models to reduce variance, whereas Boosting sequentially corrects errors of weak learners to reduce bias. B: **Incorrect:** This reverses the roles of Bagging and Boosting. C: **Incorrect:** Boosting primarily reduces bias by sequentially fitting the residuals of high-bias weak learners. D: **Incorrect:** Bagging does not reduce bias; its primary mechanism is variance reduction through model averaging.

**1. Understand the geometry of Hard-Margin SVM on two points:** We are given two datapoints in \(\mathbb{R}^2\): - Positive point: \(x_1 = (-2, -2)\) with label \(y_1 = +1\) - Negative point: \(x_2 = (2, 2)\) with label \(y_2 = -1\) In a hard-margin SVM with two points (one from each class), both points are support vectors, and the maximum-margin decision boundary is the **perpendicular bisector** of the line segment joining \(x_1\) and \(x_2\). **2. Find the midpoint and normal vector:** - **Midpoint \(m\):** \[m = \frac{x_1 + x_2}{2} = \frac{(-2, -2) + (2, 2)}{2} = (0, 0)\] So the decision boundary passes through the origin \((0, 0)\), meaning \(b = 0\). - **Direction (normal) vector \(w\):** The vector pointing from \(x_2\) to \(x_1\) is \(x_1 - x_2 = (-4, -4)\), which is parallel to \((1, 1)\). Thus, the hyperplane normal to \(x_1 - x_2\) passing through \((0, 0)\) has equation: \[x + y = 0\] **3. Verify using the SVM optimization problem:** With \(w = (-1/4, -1/4)\) and \(b = 0\), we have \(w^T x_1 = (-1/4)(-2) + (-1/4)(-2) = +1\) and \(w^T x_2 = (-1/4)(2) + (-1/4)(2) = -1\). The decision boundary \(w^T x + b = 0\) is \(-\frac{1}{4}x - \frac{1}{4}y = 0 \iff x + y = 0\). **Answer: B — x + y = 0.** A: **Incorrect:** The line \(x + 2y = 0\) has normal vector \((1, 2)\), which is not perpendicular to the displacement vector \(x_2 - x_1 = (4, 4)\). B: **Correct:** The perpendicular bisector of \((-2, -2)\) and \((2, 2)\) passes through the midpoint \((0, 0)\) with normal vector \((1, 1)\), giving \(x + y = 0\). C: **Incorrect:** Both points \((-2, -2)\) and \((2, 2)\) lie *on* the line \(x - y = 0\), so \(x - y = 0\) passes directly through the data points instead of separating them. D: **Incorrect:** The equation \(y - x = 0\) is the same line as \(x - y = 0\), which passes through both points rather than separating them.

**1. Understand Principal Component Analysis (PCA) when all components are retained:** We have a dataset \(x_1, x_2, \dots, x_n\) where each \(x_i \in \mathbb{R}^5\) has \(d = 5\) features. PCA computes \(5\) orthogonal principal axes (eigenvectors of the \(5 \times 5\) covariance matrix) and projects each \(5\)-dimensional point onto the chosen \(k\) principal components. **2. Analyze the effect of retaining all \(k = 5\) principal components:** - Dimensionality reduction occurs in PCA only when we retain \(k < d\) principal components (e.g., keeping \(k = 2\) or \(3\) components out of \(5\)). - When all \(k = 5\) principal components are retained, each transformed data point \(z_i \in \mathbb{R}^5\) still has \(5\) features. This is simply an orthogonal rotation (change of basis) of \(\mathbb{R}^5\) that decorrelates the features without reducing the dimensionality of the dataset. **Answer: B — PCA has not reduced the dimensionality of the dataset because all principal components are retained.** A: **Incorrect:** Decorrelating features via an orthogonal rotation from \(\mathbb{R}^5\) to \(\mathbb{R}^5\) changes the coordinate basis but keeps the dimension equal to \(5\). B: **Correct:** Retaining all \(5\) principal components maps each \(x_i \in \mathbb{R}^5\) to a \(5\)-dimensional vector, so the dimensionality remains \(5\). C: **Incorrect:** PCA reduces dimensionality only when the number of retained components \(k\) is strictly less than the original dimension \(d\). D: **Incorrect:** PCA is one of the most widely used linear dimensionality reduction techniques whenever \(k < d\) components are retained.

**1. Identify the neural network architecture:** We are given a fully connected feed-forward neural network (ignoring biases) with the following layers: - Input layer: \(n_0 = 5\) neurons - Hidden layer 1: \(n_1 = 10\) neurons - Hidden layer 2: \(n_2 = 10\) neurons - Hidden layer 3: \(n_3 = 10\) neurons - Output layer: \(n_4 = 1\) neuron **2. Calculate the number of weight parameters between each pair of adjacent layers:** Without bias terms, a fully connected layer from \(n_{l-1}\) units to \(n_l\) units has \(n_{l-1} \times n_l\) weights: - Input to Hidden layer 1: \(5 \times 10 = 50\) - Hidden layer 1 to Hidden layer 2: \(10 \times 10 = 100\) - Hidden layer 2 to Hidden layer 3: \(10 \times 10 = 100\) - Hidden layer 3 to Output layer: \(10 \times 1 = 10\) **3. Sum all weight parameters:** \[\text{Total parameters} = 50 + 100 + 100 + 10 = 260\] **Answer: 260.**