Week 08 · Midterm — English

Русский

Part A — Test questions

Task 1

Which belongs to supervised learning?

1. k-means clustering
2. PCA
3. Logistic regression
4. Apriori frequent itemset mining

Show correct answer

C. Logistic regression

Task 2

Which standardization is correct for scalar \(x\) given mean \(\mu\) and standard deviation \(\sigma>0\)?

1. \((x-\sigma)/\mu\)
2. \(x/(\mu+\sigma)\)
3. \((x-\mu)/\sigma\)
4. \(x-\mu\sigma\)

Show correct answer

C. \((x-\mu)/\sigma\)

Task 3

One-hot encoding is typically preferred over label encoding for:

1. Ordinal features with clear order
2. Continuous variables
3. Binary variables already in \(\{0,1\}\)
4. Nominal categorical features

Show correct answer

D. Nominal categorical features

Task 4

Which model is most sensitive to feature scaling?

1. kNN
2. Decision trees
3. Naive Bayes (Gaussian)
4. Random Forest

Show correct answer

A. kNN

Task 5

You compare several models using a training set, a validation set, and a test set. Which procedure keeps the test set independent of model selection?

1. Choose the model with the highest test accuracy.
2. Train every model on the combined training and test sets.
3. Select the model using validation results, then evaluate the selected model on the test set.
4. Adjust hyperparameters repeatedly until test accuracy improves.

Show correct answer

C. Validation results guide model selection; the test set is reserved for evaluating the selected model.

Task 6

In \(k\)-fold CV, each example appears in the validation set:

1. Exactly once
2. Multiple times
3. Never
4. Only if it’s misclassified

Show correct answer

A. Exactly once

Task 7

For large \(n\), a bootstrap sample is drawn by making \(n\) independent uniform draws with replacement from \(n\) items. The expected fraction of distinct original items included is approximately:

1. \(1- e^{-1} \approx 0.632\)
2. \(0.5\)
3. \(0.75\)
4. \(1.0\)

Show correct answer

A. \(1- e^{-1} \approx 0.632\)

Task 8

Gini impurity for class probs \((p,1-p)\) equals:

1. \(-p\log p -(1-p)\log(1-p)\)
2. \(2p(1-p)\)
3. \(1-p\)
4. \(p^2+(1-p)^2\)

Show correct answer

B. \(2p(1-p)\)

Task 9

Which is true?

1. Bagging primarily reduces bias
2. Increasing model complexity always reduces variance
3. Early stopping increases variance
4. Bagging primarily reduces variance

Show correct answer

D. Bagging primarily reduces variance

Task 10

Out-of-bag (OOB) estimate in Random Forests:

1. Uses samples not included in a tree’s bootstrap draw to estimate generalization error
2. Requires an external validation set
3. Is identical to \(k\)-fold CV
4. Can’t be used for model selection

Show correct answer

A. Uses samples not included in a tree’s bootstrap draw to estimate generalization error

Task 11

Boosting differs from bagging primarily by:

1. Using bootstrap sampling
2. Using only linear weak learners
3. Sequentially reweighing/misfit focusing
4. Random feature subspacing only

Show correct answer

C. Sequentially reweighing/misfit focusing

Task 12

The key idea behind a Support Vector Machine (SVM) is to:

1. Minimize the training error by fitting all points exactly
2. Combine many weak classifiers to reduce bias
3. Find the decision boundary that maximizes the margin between classes
4. Randomly sample features to avoid overfitting

Show correct answer

C. Find the decision boundary that maximizes the margin between classes

Task 13

The kernel trick allows:

1. Implicit feature mapping via kernel evaluations
2. Faster gradient descent on raw features
3. Exact Bayes optimal classification
4. Avoiding regularization entirely

Show correct answer

A. Implicit feature mapping via kernel evaluations

Task 14

A binary classifier predicts a fixed probability \(p(y=1\mid\mathbf{x})\) for each input. It predicts class 1 when this probability is at least a threshold \(t\). If \(t\) increases from 0.5 to 0.8, what must happen to the number of inputs predicted as class 1?

1. It must increase.
2. It cannot increase.
3. It must decrease to zero.
4. It must stay unchanged.

Show correct answer

B. Every probability at least 0.8 is also at least 0.5. The new positive predictions are a subset of the old ones; their number can decrease or stay unchanged.

Task 15

Binary cross-entropy is generally preferable to MSE for probabilistic binary classification because:

1. It penalizes errors linearly
2. It is invariant to class imbalance
3. It aligns with Bernoulli likelihood and provides better gradients for confidently incorrect predictions near 0/1
4. It removes need for regularization

Show correct answer

C. It aligns with Bernoulli likelihood and provides better gradients for confidently incorrect predictions near 0/1

Task 16

Too large learning rate in GD most likely causes:

1. Faster guaranteed convergence
2. Better generalization
3. Divergence/oscillation of loss
4. Exact line search

Show correct answer

C. Divergence/oscillation of loss

Task 17

\(L_1\) vs \(L_2\):

1. Both encourage sparsity equally
2. \(L_2\) sets many weights exactly to zero
3. \(L_1\) increases variance
4. \(L_1\) encourages sparsity; \(L_2\) shrinks weights without sparsity

Show correct answer

D. \(L_1\) encourages sparsity; \(L_2\) shrinks weights without sparsity

Task 18

The main purpose of applying dropout during training is to:

1. Prevent co-adaptation of neurons and reduce overfitting
2. Increase model capacity by adding more neurons
3. Ensure every neuron is used in each forward pass
4. Improve convergence speed on the training set

Show correct answer

A. Prevent co-adaptation of neurons and reduce overfitting

Task 19

Which activation is not bounded?

1. Sigmoid
2. Tanh
3. Hard-sigmoid
4. ReLU

Show correct answer

D. ReLU

Task 20

Backprop relies fundamentally on:

1. The chain rule of calculus
2. Monte Carlo integration
3. Bayes theorem
4. Central limit theorem

Show correct answer

A. The chain rule of calculus

Task 21

Given TP=30, FP=10, FN=20, TN=40, recall equals:

1. \(30/(30+20)=0.6\)
2. \(30/(30+10)=0.75\)
3. \(40/(40+10)=0.8\)
4. \((30+40)/100=0.7\)

Show correct answer

A. \(30/(30+20)=0.6\)

Task 22

A regression model has prediction errors \(\hat{y}^{(1)}-y^{(1)}=-1\) and \(\hat{y}^{(2)}-y^{(2)}=3\). Using \[ \mathrm{MSE}=\frac1n\sum_{i=1}^n(\hat{y}^{(i)}-y^{(i)})^2, \] what is its mean squared error on these two samples?

1. 2
2. 4
3. 5
4. 10

Show correct answer

C. \(\mathrm{MSE}=((-1)^2+3^2)/2=5\).

Task 23

Compared to MSE, MAE is typically:

1. More robust to outliers
2. Less robust to outliers
3. Equivalent under all distributions
4. Differentiable everywhere with same gradient

Show correct answer

A. More robust to outliers

Task 24

kNN can model:

1. Highly non-linear decision boundaries
2. Only linear boundaries
3. Only quadratic boundaries
4. Only piecewise-linear with fixed number of pieces

Show correct answer

A. Highly non-linear decision boundaries

Part B — Basic practical tasks

Task 25

Consider a logistic regression binary classifier \(f(\mathbf{x};\mathbf{w})\) with weights \(\mathbf{w}=[w_0,w_1,w_2]\) and input \(\mathbf{x}^{(i)}=[x^{(i)}_1,x^{(i)}_2]\). The linear score is \[ z^{(i)} = w_0 + w_1 x^{(i)}_1 + w_2 x^{(i)}_2, \] and the hypothesis is the sigmoid \[ h^{(i)} = \sigma(z^{(i)}) = \frac{1}{1+\exp(-z^{(i)})}, \] which should be close to the label \(y^{(i)}\in\{0,1\}\).

Given \(h^{(i)} = 0.5\), derive the decision boundary equation in the form \[ x_2 = x_2(x_1). \]

Show correct answer

Since \(\sigma(0)=0.5\), the boundary satisfies \[ w_0+w_1x_1+w_2x_2=0. \] If \(w_2\ne0\), \[ x_2(x_1)=-\frac{w_0+w_1x_1}{w_2}. \] If \(w_2=0\) and \(w_1\ne0\), the boundary is the vertical line \(x_1=-w_0/w_1\). If both are zero, every input is on the boundary when \(w_0=0\); otherwise there is no boundary.

Task 26

You are implementing a custom PyTorch dataset for image classification. It should store image–label pairs, apply an optional transform, and return one sample at a time. Complete the code marked with 'Enter your code here'.

from torch.utils.data import Dataset
from torchvision import transforms

# Example transforms (given):
basic_transform = transforms.Compose([
    transforms.Resize((64, 64)),
    transforms.ToTensor()
])

class CustomImageDataset(Dataset):
    def __init__(self, images, labels, transform=None):
        # Store dataset attributes
        self.images = images
        # Enter your code here
                                # Store labels
        # Enter your code here
                                # Store transform (optional)
    def __len__(self):
        # Enter your code here
                                # Return number of samples, e.g. using len()

    def __getitem__(self, idx):
        img = self.images[idx]
        label = self.labels[idx]

        # Apply transformation if available
        if self.transform:
            # Enter your code here
                                # Apply transform to img
        return img, label

# Example usage:
# dataset = CustomImageDataset(image_list, label_list, transform=basic_transform)
# img, label = dataset[0]

Show correct answer

self.labels = labels
self.transform = transform
return len(self.images)
img = self.transform(img)

Insert these four lines in the corresponding marked positions.

Task 27

For univariate linear regression \(h_w(x)=w_0+w_1 x\) with MSE on a single sample \((x,y)\), write the partial derivatives with respect to \(w_0\) and \(w_1\) and one SGD step with learning rate \(\eta\).

Show correct answer

Let \(r=w_0+w_1x-y\) and \(L=r^2\). Then \[ \frac{\partial L}{\partial w_0}=2r,\qquad \frac{\partial L}{\partial w_1}=2rx. \] One SGD step is \[ w_0\leftarrow w_0-2\eta r,\qquad w_1\leftarrow w_1-2\eta rx. \] Both updates use the residual computed before either parameter is changed.

Part C — Advanced practical tasks

Task 28

You are training a simple linear regression model \(y = wx + b\) using PyTorch tensors. Complete the missing parts in the training loop.

import torch

# Data
x = torch.tensor([[1.0], [2.0], [3.0]])
y = torch.tensor([[2.0], [4.0], [6.0]])

# Initialize parameters
w = torch.randn(1, requires_grad=True)
b = torch.randn(1, requires_grad=True)

lr = 0.1
for epoch in range(100):
    # Forward pass
    # Enter your code here
                            # Compute y_pred
    # Compute loss (MSE)
    # Enter your code here
                            # Compute mean squared error
    # Backpropagation
    loss.backward()

    # Update weights
    with torch.no_grad():
        # Enter your code here
                            # Gradient descent step for w
                            # Gradient descent step for b
        w.grad.zero_()
        b.grad.zero_()

Show correct answer

y_pred = w * x + b
loss = ((y_pred - y)**2).mean()

Inside the existing torch.no_grad() block:

w -= lr * w.grad
b -= lr * b.grad

Keep the existing gradient reset lines after the updates.

Task 29

You are given a partially implemented neural network model in PyTorch. It consists of an input layer, one hidden layer with ReLU activation, a Batch Normalization layer, a Dropout layer for regularization, and an output layer (no activation). Complete the forward method by filling the lines marked with 'Enter your code here'. Use the given layer names correctly.

import torch
import torch.nn as nn

class RegularizedFeedForwardModel(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, dropout_p=0.5):
        super().__init__()

        # Define layers
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.bn1 = nn.BatchNorm1d(hidden_size)
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(p=dropout_p)
        self.fc2 = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        # Forward pass through the first fully connected layer
        # Enter your code here
                                  # Input to hidden layer
        # Apply batch normalization
        # Enter your code here
                                  # Normalize activations
        # Apply ReLU activation
        # Enter your code here
                                  # Apply activation function
        # Apply dropout
        # Enter your code here
                                  # Randomly drop some neurons
        # Forward pass through output layer
        # Enter your code here
                                  # Hidden to output layer
        return x

Show correct answer

x = self.fc1(x)
x = self.bn1(x)
x = self.relu(x)
x = self.dropout(x)
x = self.fc2(x)

Insert these lines in order in the forward method. The output layer has no activation.

Part D — Advanced theoretical task

Task 30

For \(n\) labeled samples with \(K\) classes, write \(p_i(y)=p_{\mathbf{w}}(y\mid\mathbf{x}^{(i)})\) for the model’s predicted class probabilities and \(q_i(y)\) for fixed target probabilities. Both distributions sum to one, \(p_i(y)>0\), and only \(p_i\) depends on the parameters \(\mathbf{w}\). Initially, the targets are one-hot: \(q_i(y^{(i)})=1\) and all other entries are zero.

The mean KL divergence is \[ J(\mathbf{w})=\frac1n\sum_{i=1}^n\sum_{y=1}^K q_i(y)\log\frac{q_i(y)}{p_i(y)}. \] Terms with zero target probability contribute zero. Assuming labels are conditionally independent given their inputs and \(\mathbf{w}\), their likelihood is \[ P(\mathbf{w})=\prod_{i=1}^n p_i(y^{(i)}). \]

Task. Construct an additive loss that avoids logarithms of target probabilities and has the same minimizers as \(J\). Determine whether minimizing this loss is equivalent to maximizing \(P\), and justify your conclusion. Would each of these equivalences still hold if the fixed targets assigned positive probability to several classes?

Use the supplied formulas to support your reasoning; naming a familiar loss is not sufficient. Several derivations are possible; one valid solution is sufficient.

Show correct answer

One possible solution. Define \[ L(\mathbf{w})=-\frac1n\sum_{i=1}^n\sum_{y=1}^K q_i(y)\log p_i(y). \] Expanding the log ratio gives \(J=L+C\), where \[ C=\frac1n\sum_{i=1}^n\sum_{y:q_i(y)>0}q_i(y)\log q_i(y) \] is independent of \(\mathbf{w}\). Thus \(J\) and \(L\) have the same minimizers. This derived loss is cross-entropy.

For one-hot targets, \(C=0\) and \[ L(\mathbf{w})=-\frac1n\sum_{i=1}^n\log p_i(y^{(i)}) =-\frac1n\log P(\mathbf{w}). \] Since the logarithm is increasing, minimizing \(L\) is equivalent to maximizing \(P\).

For fixed soft targets, \(J=L+C\) still holds. However, \(L\) generally includes several class terms per sample, so it is no longer \(-\log P/n\) for the supplied observed-label likelihood. The first equivalence survives; the second does not hold in general.

Other mathematically justified derivations are equally valid.

---

Back to practice quizzes