Why Characteristic Functions?
Let $X_1, X_2, \dots, X_n$ be independent and identically distributed (i.i.d.) random variables with mean $\mu = \mathbb{E}[X_i]$ and finite variance $0 < \sigma^2 = \text{Var}(X_i) < \infty$. Why does the standardized sum
$$Z_n = \frac{1}{\sqrt{n}}\sum_{k=1}^n \frac{X_k - \mu}{\sigma}$$
converge in distribution to $\mathcal{N}(0, 1)$ regardless of the distribution of $X_k$?
In the spatial domain, the probability distribution of a sum of $n$ independent variables is the $n$-fold convolution $F * F * \cdots * F$, which is analytically intractable for general distributions. However, in Fourier space, convolution turns into ordinary multiplication.
While one might be tempted to use the Moment Generating Function $M_X(t) = \mathbb{E}[e^{tX}]$, $M_X(t)$ is only finite in a neighborhood of $t = 0$ if $X$ has sub-exponential tails (for instance, the log-normal and Student-$t$ distributions have finite variance and obey the CLT, yet their MGFs diverge for all $t > 0$). Replacing the real exponent $tX$ with the purely imaginary exponent $itX$ resolves this completely.
Fourier Analysis of Probability Measures
Definition
The characteristic function of a real-valued random variable $X$ with cumulative distribution function $F_X$ is the Fourier–Stieltjes transform $\phi_X : \mathbb{R} \to \mathbb{C}$ defined by:
$$\phi_X(t) = \mathbb{E}!\left[e^{itX}\right] = \int_{-\infty}^{\infty} e^{itx},dF_X(x).$$
Because $\left|e^{itx}\right| = |\cos(tx) + i\sin(tx)| = 1$ for all real $t$ and $x$, the integral converges absolutely for every random variable: $\phi_X(0) = 1$ and $|\phi_X(t)| \le 1$.
Four Foundational Properties
- Affine Transformation: For any $a, b \in \mathbb{R}$, $\phi_{aX + b}(t) = e^{itb},\phi_X(at)$.
- Independence Multiplicativity: If $X$ and $Y$ are independent, then $\phi_{X+Y}(t) = \mathbb{E}[e^{itX}e^{itY}] = \phi_X(t),\phi_Y(t)$.
- Moments and Taylor Expansion: If $\mathbb{E}[|X|^k] < \infty$, then $\phi_X$ is $k$-times continuously differentiable on $\mathbb{R}$ with $\phi_X^{(m)}(0) = i^m \mathbb{E}[X^m]$ for $1 \le m \le k$. By Peano’s form of Taylor’s theorem around $t = 0$: $$\phi_X(t) = \sum_{m=0}^k \frac{(it)^m}{m!}\mathbb{E}[X^m] + o(|t|^k) \quad \text{as } t \to 0.$$
- The Gaussian Fixed Point: If $Z \sim \mathcal{N}(0, 1)$, completing the square in the complex integral (or solving the differential equation $\phi_Z’(t) = -t\phi_Z(t)$ with $\phi_Z(0)=1$) yields: $$\phi_Z(t) = e^{-t^2 / 2}.$$
Theorem (LĂ©vy’s Continuity Theorem)
Let ${Z_n}{n=1}^\infty$ be a sequence of random variables with characteristic functions ${\phi{Z_n}}{n=1}^\infty$. If $\lim{n \to \infty} \phi_{Z_n}(t) = \phi(t)$ pointwise for every $t \in \mathbb{R}$, and $\phi(t)$ is continuous at $t = 0$, then $\phi$ is the characteristic function of a valid probability distribution $Z$, and $Z_n \xrightarrow{d} Z$.
Rigorous Proof of the Lindeberg–Lévy CLT
Proof. Define the standardized summands $Y_k = \frac{X_k - \mu}{\sigma}$. Then $Y_1, \dots, Y_n$ are i.i.d. with:
$$\mathbb{E}[Y_k] = 0, \qquad \mathbb{E}[Y_k^2] = \text{Var}(Y_k) = 1, \qquad Z_n = \frac{1}{\sqrt{n}}\sum_{k=1}^n Y_k.$$
Let $\phi_Y(t) = \mathbb{E}[e^{itY_1}]$ denote the common characteristic function of each $Y_k$.
- Second-Order Taylor Expansion: Since $\mathbb{E}[Y_1] = 0$ and $\mathbb{E}[Y_1^2] = 1 < \infty$, expanding $\phi_Y$ around $0$ gives: $$\phi_Y(u) = 1 + iu,\mathbb{E}[Y_1] + \frac{(iu)^2}{2}\mathbb{E}[Y_1^2] + o(u^2) = 1 - \frac{u^2}{2} + o(u^2) \quad \text{as } u \to 0.$$
- Characteristic Function of $Z_n$: By independence and the scaling property ($a = 1/\sqrt{n}$): $$\phi_{Z_n}(t) = \prod_{k=1}^n \phi_{Y_k}!\left(\frac{t}{\sqrt{n}}\right) = \left[\phi_Y!\left(\frac{t}{\sqrt{n}}\right)\right]^n.$$
- Pointwise Limit: Fix any $t \in \mathbb{R}$. As $n \to \infty$, $u_n = t/\sqrt{n} \to 0$. Substituting the Taylor expansion at $u = t/\sqrt{n}$ yields: $$\phi_{Z_n}(t) = \left[1 - \frac{t^2}{2n} + o!\left(\frac{t^2}{n}\right)\right]^n = \left[1 + \frac{-t^2/2 + n \cdot o(t^2/n)}{n}\right]^n.$$ Because $n \cdot o(t^2/n) \to 0$ as $n \to \infty$ for fixed $t$, the complex exponential limit $\lim_{n \to \infty}(1 + c_n/n)^n = e^c$ (when $c_n \to c$) gives: $$\lim_{n \to \infty} \phi_{Z_n}(t) = e^{-t^2 / 2} \qquad \forall t \in \mathbb{R}.$$
- Conclusion: Since $t \mapsto e^{-t^2/2}$ is continuous at $t = 0$ and is the characteristic function of $Z \sim \mathcal{N}(0, 1)$, LĂ©vy’s Continuity Theorem implies $Z_n \xrightarrow{d} \mathcal{N}(0, 1)$. $\square$
When the CLT Breaks: Heavy Tails and Cauchy Stability
Look at step 1 of the proof: everything hinges on $\phi_Y(u)$ having a quadratic local expansion $1 - \frac{1}{2}u^2 + o(u^2)$ near the origin, which requires $\mathbb{E}[Y^2] < \infty$. What happens when the variance is infinite?
Consider the standard Cauchy distribution with density $f(x) = \frac{1}{\pi(1 + x^2)}$ on $\mathbb{R}$. Because $\int_{-\infty}^\infty \frac{|x|}{\pi(1+x^2)},dx = \infty$, neither its mean nor its variance exists. By residue calculus, its characteristic function is:
$$\phi_X(t) = e^{-|t|},$$
which has a sharp non-differentiable cusp at $t = 0$ rather than a parabolic peak!
Now compute the characteristic function of the sample mean $\bar{X}n = \frac{1}{n}\sum{k=1}^n X_k$ of $n$ i.i.d. Cauchy variables:
$$\phi_{\bar{X}_n}(t) = \left[\phi_X!\left(\frac{t}{n}\right)\right]^n = \left(e^{-|t|/n}\right)^n = e^{-|t|} = \phi_X(t).$$
The sample mean of $n = 10^9$ independent Cauchy measurements has the exact same distribution as a single measurement ($n = 1$)! Extreme tail spikes of magnitude $O(n)$ occur with probability $\Theta(1/n)$, completely overwhelming the averaging of typical values.
More generally, the Gnedenko–Kolmogorov Generalized CLT shows that if $X_i$ has power-law tails $\mathbb{P}(|X| > x) \sim c x^{-\alpha}$ with stability index $\alpha \in (0, 2)$, then $\phi_X(t) = 1 - c’|t|^\alpha + o(|t|^\alpha)$, so the sum must be normalized by $n^{1/\alpha}$ (rather than $n^{1/2}$) and converges to a symmetric $\alpha$-stable LĂ©vy distribution with characteristic function $e^{-c’|t|^\alpha}$.
Non-Asymptotic Error Bounds: The Berry–Esseen Theorem
Even when $\sigma^2 < \infty$, convergence in distribution is an asymptotic statement: how large is the error $|F_{Z_n}(z) - \Phi(z)|$ at finite $n$?
If we control the third absolute central moment $\rho = \mathbb{E}[|X_1 - \mu|^3] < \infty$, we can replace the qualitative Peano remainder $o(u^2)$ in Taylor’s expansion with a quantitative $O(|u|^3)$ bound and invert via Esseen’s smoothing inequality:
Theorem (Berry–Esseen, 1941–1942)
If $X_1, \dots, X_n$ are i.i.d. with mean $\mu$, variance $\sigma^2 > 0$, and third absolute central moment $\rho = \mathbb{E}[|X_1 - \mu|^3] < \infty$, then for every $n \ge 1$:
$$\sup_{z \in \mathbb{R}} \left|\mathbb{P}(Z_n \le z) - \Phi(z)\right| \le \frac{C,\rho}{\sigma^3\sqrt{n}},$$
where $C < 0.4748$ (Shevtsova, 2011) is a universal constant.
The dimensionless factor $\rho / \sigma^3 \ge 1$ measures the asymmetry and tail heaviness of $X_1$:
- For symmetric $\text{Bernoulli}(0.5)$ trials, $\rho / \sigma^3 = 1$, and convergence is rapid.
- For highly skewed $\text{Bernoulli}(p)$ trials with tiny $p \ll 1$, $\rho / \sigma^3 \approx 1/\sqrt{p}$, warning us that $n$ must satisfy $n p \gg 1$ before the Gaussian approximation becomes trustworthy.
References
- Billingsley, P. (1995). Probability and Measure (3rd ed.). Wiley.
- Durrett, R. (2019). Probability: Theory and Examples (5th ed.). Cambridge University Press.
- Feller, W. (1971). An Introduction to Probability Theory and Its Applications, Vol. 2 (2nd ed.). Wiley.
- Gnedenko, B. V., & Kolmogorov, A. N. (1968). Limit Distributions for Sums of Independent Random Variables. Addison-Wesley.
- Shevtsova, I. (2011). On the absolute constants in the Berry–Esseen type inequalities for identically distributed summands. arXiv:1111.6554.