Generalized Linear Models

A unifying framework — link function plus exponential-family distribution — that includes linear, logistic, and Poisson regression as special cases.

GLMs keep the linear score but swap the link and response distribution to match the data

Linear regression

Response: continuous
Support: any real value
Link: identity
Noise: Normal

Logistic regression

Response: binary
Support: 0 to 1 probability
Link: logit
Noise: Bernoulli

Poisson regression

Response: counts
Support: zero or positive
Link: log
Noise: Poisson

Same backbone: features become a linear score. The link turns that score into a legal mean for the response.

Definition

A Generalized Linear Model (GLM) is a recipe with three ingredients:

A linear predictor $\eta = X\beta$ — the same weighted sum of features used in ordinary linear regression.
A link function $g$ that connects $\eta$ to the mean $\mu$ of the response: $g(\mu) = \eta$ .
A choice of distribution (from the exponential family) for how the response varies around that mean.

Ordinary linear regression, logistic regression, and Poisson regression are not three unrelated techniques — they're the same model with three different choices of link function and distribution.

Spotting the pattern

Linear regression: predict a continuous value $\Rightarrow$ identity link ( $g(\mu) = \mu$ ), Normal distribution.

Logistic regression: predict a probability $\Rightarrow$ logit link, Bernoulli distribution.

Both fit $\eta = X\beta$ first — they only differ in how $\eta$ gets turned into a prediction and how the error around that prediction is modeled.

Try it

Why can't ordinary linear regression be used directly to predict a probability (a value that must stay between 0 and 1)?

Solution

Linear regression's predictions $X\beta$ can be any real number — nothing stops them from going below 0 or above 1, which makes no sense as a probability. The logit link function specifically maps the unrestricted real line to the (0, 1) interval, which is exactly why logistic regression uses it instead of the identity link.

Related concepts

Needs first

Linear Regression Logistic Regression Probability

INLA Model Evaluation Regularization

View in full concept graph →