The Central Limit Theorem: Formulation, Intuition, and Statistical Inference

Why sums of independent random variables converge to the Gaussian distribution, how the √n scaling law governs sample means, and how the CLT drives statistical inference.

Universal Convergence to the Gaussian

Why does the Gaussian bell curve $\phi(z) = \frac{1}{\sqrt{2\pi}}e^{-z^2/2}$ govern phenomena as disparate as thermal noise in silicon, measurement errors in astronomy, biological heights, and poll margins of error?

The mathematical answer is the Central Limit Theorem (CLT): whenever an observable quantity is the additive aggregate of many small, independent random fluctuations with finite variance, its standardized distribution converges toward a normal distribution—regardless of the shape of the individual fluctuations.

First observed for binomial coin flips by Abraham de Moivre (1733), generalized by Pierre-Simon Laplace (1812), and proven in full generality by Aleksandr Lyapunov (1901), Jarl Lindeberg (1920), and Paul Lévy (1925), the CLT is the pillar connecting probability theory to statistical inference.


Formal Statement (Lindeberg–Lévy CLT)

Let $X_1, X_2, \dots, X_n$ be independent and identically distributed (i.i.d.) random variables with finite mean and positive, finite variance:

$$\mu = \mathbb{E}[X_i], \qquad 0 < \sigma^2 = \text{Var}(X_i) < \infty.$$

Define the partial sum $S_n = \sum_{i=1}^n X_i$ and the sample mean $\bar{X}_n = \frac{1}{n}S_n$. Because expectation is linear and variance is additive over independent variables:

$$\mathbb{E}[S_n] = n\mu, \qquad \text{Var}(S_n) = n\sigma^2, \qquad \mathbb{E}[\bar{X}_n] = \mu, \qquad \text{Var}(\bar{X}_n) = \frac{\sigma^2}{n}.$$

Standardizing $S_n$ (or equivalently $\bar{X}_n$) by subtracting its mean and dividing by its standard deviation defines the dimensionless pivot:

$$Z_n = \frac{S_n - n\mu}{\sigma\sqrt{n}} = \frac{\bar{X}_n - \mu}{\sigma / \sqrt{n}}.$$

Theorem (Central Limit Theorem)

As $n \to \infty$, the standardized sum $Z_n$ converges in distribution to the standard normal random variable $Z \sim \mathcal{N}(0, 1)$:

$$Z_n \xrightarrow{d} \mathcal{N}(0, 1), \qquad \text{i.e.,} \qquad \lim_{n \to \infty} \mathbb{P}(Z_n \le z) = \Phi(z) = \int_{-\infty}^{z} \frac{1}{\sqrt{2\pi}}e^{-u^2/2},du \quad \forall z \in \mathbb{R}.$$


Why $\sqrt{n}$ Is the Exact Scaling Window

It is instructive to compare what happens to $S_n - n\mu$ under different normalizations as $n \to \infty$:

  1. Dividing by $n$ (The Law of Large Numbers Scale): $$\frac{S_n - n\mu}{n} = \bar{X}_n - \mu \xrightarrow{p} 0.$$ At scale $1/n$, the variance is $\sigma^2 / n \to 0$: all randomness collapses to a single deterministic spike at $0$.
  2. Dividing by $1$ (Unnormalized Sum): The variance $n\sigma^2 \to \infty$: the distribution spreads out flat over the entire real line.
  3. Dividing by $\sqrt{n}$ (The CLT Fluctuation Scale): $$\text{Var}!\left(\frac{S_n - n\mu}{\sqrt{n}}\right) = \sigma^2.$$ Dividing by $\sqrt{n}$ acts as a microscope zoomed to the exact scale where the variance remains constant ($\sigma^2$) for all $n$, revealing the universal Gaussian shape of the random fluctuations around the mean.

Convolution Smoothing in Action

1. Sums of Discrete Uniform Variables (Dice)

Let $X_i \in {1, 2, 3, 4, 5, 6}$ be the outcome of a single fair die ($\mu = 3.5$, $\sigma^2 = 35/12 \approx 2.917$). The probability mass function (PMF) of $X_1$ is completely flat.

Adding a second independent die $S_2 = X_1 + X_2$ computes the discrete convolution of two flat rectangular distributions, which is a triangle peaking at $7$. Adding a third die $S_3$ convolves the triangle with a rectangle, producing a piecewise-quadratic bell curve (B-spline smoothing). By $n = 10$ dice, the standardized PMF is visually indistinguishable from the continuous Gaussian density $\mathcal{N}(0, 1)$.

2. De Moivre–Laplace Binomial Approximation

Let $X_1, \dots, X_n \sim \text{Bernoulli}(p)$ be independent coin flips with success probability $p \in (0, 1)$. Their sum $S_n \sim \text{Binomial}(n, p)$ has mean $np$ and variance $np(1-p)$. By the CLT:

$$\frac{S_n - np}{\sqrt{np(1-p)}} \xrightarrow{d} \mathcal{N}(0, 1), \qquad \text{so} \qquad S_n \approx \mathcal{N}!\left(np,, np(1-p)\right).$$

(When approximating a discrete integer sum $S_n$ with a continuous normal integral for moderate $n$, applying the $\pm 0.5$ continuity correction $\mathbb{P}(a \le S_n \le b) \approx \Phi!\left(\frac{b + 0.5 - np}{\sqrt{np(1-p)}}\right) - \Phi!\left(\frac{a - 0.5 - np}{\sqrt{np(1-p)}}\right)$ substantially improves accuracy).


Statistical Inference and Confidence Intervals

In empirical data analysis and Monte Carlo simulation, we observe a sample $X_1, \dots, X_n$ from an unknown distribution $F$ and want to estimate its true mean $\mu = \mathbb{E}[X]$ with a rigorous error bar.

Even though we do not know $F$, the CLT guarantees that for large $n$, the pivot $\frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}}$ is approximately $\mathcal{N}(0, 1)$. Furthermore, if the true standard deviation $\sigma$ is unknown, replacing $\sigma$ with the consistent sample standard deviation

$$s_n = \sqrt{\frac{1}{n-1}\sum_{i=1}^n (X_i - \bar{X}_n)^2}$$

preserves asymptotic normality by Slutsky’s Theorem (since $s_n \xrightarrow{p} \sigma$):

$$\frac{\bar{X}_n - \mu}{s_n / \sqrt{n}} \xrightarrow{d} \mathcal{N}(0, 1).$$

Asymptotic $(1 - \alpha)$ Confidence Interval

Let $z_{\alpha/2} = \Phi^{-1}(1 - \alpha/2)$ denote the upper $\alpha/2$ quantile of $\mathcal{N}(0, 1)$ (for a $95%$ confidence level, $\alpha = 0.05$ and $z_{0.025} \approx 1.96$). Then:

$$\lim_{n \to \infty} \mathbb{P}!\left(\mu \in \left[\bar{X}n - z{\alpha/2}\frac{s_n}{\sqrt{n}}, ;; \bar{X}n + z{\alpha/2}\frac{s_n}{\sqrt{n}}\right]\right) = 1 - \alpha.$$

Notice the $\sqrt{n}$ law of precision: the width of the confidence interval is proportional to $1/\sqrt{n}$. To reduce estimation uncertainty by a factor of $10$, a Monte Carlo simulation or clinical trial must collect $100\times$ as many samples.