Week 5: Classification Losses

Binary Classification Setup#

In binary classification, labels are $y \in \{0, 1\}$. The model outputs a real-valued score $z = w^T x$ (logit), which is squashed through the sigmoid function to produce a probability:

$\sigma(z) = \frac{1}{1 + e^{-z}}$

The sigmoid maps $\mathbb{R} \to (0, 1)$, with $\sigma(0) = 0.5$, $\sigma(z) \to 1$ as $z \to \infty$, and $\sigma(z) \to 0$ as $z \to -\infty$. Its derivative is clean:

$\sigma'(z) = \sigma(z)(1 - \sigma(z))$

The logistic loss (also called binary cross-entropy) for a single example is:

$\ell(\hat y, y) = -y \log \hat y - (1-y) \log (1-\hat y) \quad \text{where } \hat y = \sigma(w^T x)$

When $y=1$, the loss is $-\log \hat y$ — heavily penalises $\hat y \approx 0$. When $y=0$, the loss is $-\log(1-\hat y)$ — heavily penalises $\hat y \approx 1$.

Logistic Regression Gradient#

For one example, the gradient with respect to $w$ is:

$\nabla_w \ell = (\sigma(w^T x) - y) \, x$

The derivation uses the chain rule and the sigmoid derivative:

$\frac{\partial \ell}{\partial w} = \left(\frac{-y}{\sigma} + \frac{1-y}{1-\sigma}\right) \cdot \sigma(1-\sigma) \cdot x = (\sigma - y) x$

The result is remarkably simple: the gradient is the feature vector weighted by the prediction error. This is the classification analogue of the least squares gradient $(w^T x - y)x$.

For the full dataset, the ERM objective and its gradient are:

$J(w) = -\frac{1}{n} \sum_{i=1}^n \left[y_i \log \sigma(w^T x_i) + (1-y_i) \log (1 - \sigma(w^T x_i))\right]$

$\nabla J(w) = \frac{1}{n} \sum_{i=1}^n (\sigma(w^T x_i) - y_i) \, x_i$

Worked numeric example:

Take $w = (1, -1)$, $x = (2, 1)$, $y = 1$. Then $z = w^T x = 1 \cdot 2 + (-1) \cdot 1 = 1$, $\hat y = \sigma(1) = 1/(1+e^{-1}) \approx 0.731$. The gradient is $(\hat y - y)x = (0.731 - 1)(2, 1) = (-0.538, -0.269)$.

The loss is $-\log(0.731) \approx 0.313$. If the model had been perfectly confident ($\hat y \approx 1$), the loss would be near 0.

Softmax and Multi-Class#

For $K$ classes, the model outputs $K$ logits $z_1, \ldots, z_K$ (typically $z = W x$ where $W \in \mathbb{R}^{K \times d}$). The softmax function converts these to a probability distribution:

$\hat y_k = \text{softmax}(z)_k = \frac{e^{z_k}}{\sum_{j=1}^K e^{z_j}}$

Softmax outputs sum to 1 and are all positive. The cross-entropy loss for true class $y \in \{1, \ldots, K\}$ is:

$\ell = -\log \hat y_y = -z_y + \log \sum_{j=1}^K e^{z_j}$

The gradient with respect to the logits has a clean form:

$\frac{\partial \ell}{\partial z_k} = \hat y_k - \mathbf{1}[k = y]$

where $\mathbf{1}[k = y]$ is 1 if $k$ is the true class, 0 otherwise. So the gradient vector is $\hat y - e_y$ where $e_y$ is the one-hot encoding of the true label. By the chain rule, $\nabla_W \ell = (\hat y - e_y) x^T$.

Again, the gradient is the probability-error-weighted feature vector — exactly the same structural form as the binary case.

Convexity of Logistic Loss#

For a single example, the logistic loss $\ell(w) = -y \log \sigma(w^T x) - (1-y)\log(1-\sigma(w^T x))$ is convex in $w$. The Hessian is:

$\nabla^2 \ell(w) = \sigma(w^T x)(1 - \sigma(w^T x)) \cdot x x^T$

The term $\sigma(1-\sigma) > 0$ (the sigmoid output is always strictly between 0 and 1), and $x x^T$ is PSD. Therefore $\nabla^2 \ell(w) \succeq 0$ for all $w$ — the function is convex.

For the full dataset, the Hessian is:

$\nabla^2 J(w) = \frac{1}{n} X^T D X \quad \text{where } D = \text{diag}(\sigma(w^T x_i)(1 - \sigma(w^T x_i)))$

Since $D$ has positive entries on the diagonal, $D \succ 0$, making $X^T D X$ PSD. The logistic regression objective is convex.

However, logistic regression is not strongly convex unless $X$ has full column rank and the data is non-separable. When the data is linearly separable, $\|w\| \to \infty$ can drive the loss arbitrarily close to 0 (no finite minimiser — this motivates regularisation).

Loss and Likelihood#

Minimising cross-entropy (logistic loss) is equivalent to maximum likelihood estimation (MLEMaximum Likelihood Estimation) under a Bernoulli model. The probability of label $y$ given features $x$ and parameters $w$ is:

$P(y \mid x; w) = \sigma(w^T x)^y (1 - \sigma(w^T x))^{1-y}$

The log-likelihood of the full dataset is $\sum_i \log P(y_i \mid x_i; w) = -\sum_i \ell_i(w)$. Maximising the log-likelihood = minimising the cross-entropy loss. This connection — loss functions as negative log-likelihoods — is fundamental to modern ML and will be explored fully in the Probability & Statistics course.

Browser lab#

Plot the logistic loss surface in 2D (weight space) for a small synthetic dataset, and run GD to find the decision boundary.

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(42)
n = 100
X_pos = rng.normal(loc=[2, 2], scale=1.0, size=(n//2, 2))
X_neg = rng.normal(loc=[-2, -2], scale=1.0, size=(n//2, 2))
X = np.vstack([X_pos, X_neg])
y = np.hstack([np.ones(n//2), np.zeros(n//2)])

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def loss_and_grad(w, X, y):
    logits = X @ w
    preds = sigmoid(logits)
    eps = 1e-15
    loss = -np.mean(y * np.log(preds + eps) + (1 - y) * np.log(1 - preds + eps))
    grad = X.T @ (preds - y) / len(y)
    return loss, grad

w = np.zeros(2)
eta = 1.0
losses = []
for _ in range(100):
    loss, grad = loss_and_grad(w, X, y)
    losses.append(loss)
    w -= eta * grad

print(f"Final w: {w}")
print(f"Final loss: {loss:.6f}")
print(f"Decision boundary: x2 = {-w[0]/w[1]:.3f} x1")

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(11, 4.5))
ax1.scatter(X_pos[:, 0], X_pos[:, 1], c="tab:blue", alpha=0.6, label="y=1")
ax1.scatter(X_neg[:, 0], X_neg[:, 1], c="tab:red", alpha=0.6, label="y=0")
xx = np.linspace(-5, 5, 100)
ax1.plot(xx, -w[0]/w[1] * xx, "k--", label="decision boundary")
ax1.set_xlim(-5, 5); ax1.set_ylim(-5, 5)
ax1.set_xlabel("x1"); ax1.set_ylabel("x2")
ax1.set_title("Data and Learned Decision Boundary")
ax1.legend()
ax2.plot(losses)
ax2.set_xlabel("Iteration"); ax2.set_ylabel("Logistic Loss")
ax2.set_title("Loss vs Iteration")
ax2.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()