<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Analysis on LCS.Developer</title><link>https://www.lcs-dev.org/tags/analysis/</link><description>Recent content in Analysis on LCS.Developer</description><generator>Hugo -- gohugo.io</generator><language>en-us</language><copyright>LCS.Developer</copyright><lastBuildDate>Tue, 29 Apr 2025 20:00:00 +0000</lastBuildDate><atom:link href="https://www.lcs-dev.org/tags/analysis/index.xml" rel="self" type="application/rss+xml"/><item><title>Characteristic Functions and the Analytical Proof of the CLT</title><link>https://www.lcs-dev.org/posts/characteristic-functions-and-the-analytical-proof-of-the-clt/</link><pubDate>Tue, 29 Apr 2025 20:00:00 +0000</pubDate><guid>https://www.lcs-dev.org/posts/characteristic-functions-and-the-analytical-proof-of-the-clt/</guid><description>&lt;h2 id="why-characteristic-functions"&gt;Why Characteristic Functions?
&lt;/h2&gt;&lt;p&gt;Let $X_1, X_2, \dots, X_n$ be independent and identically distributed (i.i.d.) random variables with mean $\mu = \mathbb{E}[X_i]$ and finite variance $0 &amp;lt; \sigma^2 = \text{Var}(X_i) &amp;lt; \infty$. Why does the standardized sum&lt;/p&gt;
&lt;p&gt;$$Z_n = \frac{1}{\sqrt{n}}\sum_{k=1}^n \frac{X_k - \mu}{\sigma}$$&lt;/p&gt;
&lt;p&gt;converge in distribution to $\mathcal{N}(0, 1)$ regardless of the distribution of $X_k$?&lt;/p&gt;
&lt;p&gt;In the spatial domain, the probability distribution of a sum of $n$ independent variables is the &lt;strong&gt;$n$-fold convolution&lt;/strong&gt; $F * F * \cdots * F$, which is analytically intractable for general distributions. However, in Fourier space, &lt;strong&gt;convolution turns into ordinary multiplication&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;While one might be tempted to use the Moment Generating Function $M_X(t) = \mathbb{E}[e^{tX}]$, $M_X(t)$ is only finite in a neighborhood of $t = 0$ if $X$ has sub-exponential tails (for instance, the log-normal and Student-$t$ distributions have finite variance and obey the CLT, yet their MGFs diverge for all $t &amp;gt; 0$). Replacing the real exponent $tX$ with the purely imaginary exponent $itX$ resolves this completely.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="fourier-analysis-of-probability-measures"&gt;Fourier Analysis of Probability Measures
&lt;/h2&gt;&lt;h3 id="definition"&gt;Definition
&lt;/h3&gt;&lt;p&gt;The &lt;strong&gt;characteristic function&lt;/strong&gt; of a real-valued random variable $X$ with cumulative distribution function $F_X$ is the Fourier–Stieltjes transform $\phi_X : \mathbb{R} \to \mathbb{C}$ defined by:&lt;/p&gt;
&lt;p&gt;$$\phi_X(t) = \mathbb{E}!\left[e^{itX}\right] = \int_{-\infty}^{\infty} e^{itx},dF_X(x).$$&lt;/p&gt;
&lt;p&gt;Because $\left|e^{itx}\right| = |\cos(tx) + i\sin(tx)| = 1$ for all real $t$ and $x$, the integral converges absolutely for &lt;strong&gt;every&lt;/strong&gt; random variable: $\phi_X(0) = 1$ and $|\phi_X(t)| \le 1$.&lt;/p&gt;
&lt;h3 id="four-foundational-properties"&gt;Four Foundational Properties
&lt;/h3&gt;&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Affine Transformation&lt;/strong&gt;: For any $a, b \in \mathbb{R}$, $\phi_{aX + b}(t) = e^{itb},\phi_X(at)$.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Independence Multiplicativity&lt;/strong&gt;: If $X$ and $Y$ are independent, then $\phi_{X+Y}(t) = \mathbb{E}[e^{itX}e^{itY}] = \phi_X(t),\phi_Y(t)$.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Moments and Taylor Expansion&lt;/strong&gt;: If $\mathbb{E}[|X|^k] &amp;lt; \infty$, then $\phi_X$ is $k$-times continuously differentiable on $\mathbb{R}$ with $\phi_X^{(m)}(0) = i^m \mathbb{E}[X^m]$ for $1 \le m \le k$. By Peano&amp;rsquo;s form of Taylor&amp;rsquo;s theorem around $t = 0$:
$$\phi_X(t) = \sum_{m=0}^k \frac{(it)^m}{m!}\mathbb{E}[X^m] + o(|t|^k) \quad \text{as } t \to 0.$$&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The Gaussian Fixed Point&lt;/strong&gt;: If $Z \sim \mathcal{N}(0, 1)$, completing the square in the complex integral (or solving the differential equation $\phi_Z&amp;rsquo;(t) = -t\phi_Z(t)$ with $\phi_Z(0)=1$) yields:
$$\phi_Z(t) = e^{-t^2 / 2}.$$&lt;/li&gt;
&lt;/ol&gt;
&lt;h3 id="theorem-lévys-continuity-theorem"&gt;Theorem (Lévy&amp;rsquo;s Continuity Theorem)
&lt;/h3&gt;&lt;p&gt;Let ${Z_n}&lt;em&gt;{n=1}^\infty$ be a sequence of random variables with characteristic functions ${\phi&lt;/em&gt;{Z_n}}&lt;em&gt;{n=1}^\infty$. If $\lim&lt;/em&gt;{n \to \infty} \phi_{Z_n}(t) = \phi(t)$ pointwise for every $t \in \mathbb{R}$, and $\phi(t)$ is continuous at $t = 0$, then $\phi$ is the characteristic function of a valid probability distribution $Z$, and $Z_n \xrightarrow{d} Z$.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="rigorous-proof-of-the-lindeberglévy-clt"&gt;Rigorous Proof of the Lindeberg–Lévy CLT
&lt;/h2&gt;&lt;p&gt;&lt;strong&gt;Proof.&lt;/strong&gt; Define the standardized summands $Y_k = \frac{X_k - \mu}{\sigma}$. Then $Y_1, \dots, Y_n$ are i.i.d. with:&lt;/p&gt;
&lt;p&gt;$$\mathbb{E}[Y_k] = 0, \qquad \mathbb{E}[Y_k^2] = \text{Var}(Y_k) = 1, \qquad Z_n = \frac{1}{\sqrt{n}}\sum_{k=1}^n Y_k.$$&lt;/p&gt;
&lt;p&gt;Let $\phi_Y(t) = \mathbb{E}[e^{itY_1}]$ denote the common characteristic function of each $Y_k$.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Second-Order Taylor Expansion&lt;/strong&gt;: Since $\mathbb{E}[Y_1] = 0$ and $\mathbb{E}[Y_1^2] = 1 &amp;lt; \infty$, expanding $\phi_Y$ around $0$ gives:
$$\phi_Y(u) = 1 + iu,\mathbb{E}[Y_1] + \frac{(iu)^2}{2}\mathbb{E}[Y_1^2] + o(u^2) = 1 - \frac{u^2}{2} + o(u^2) \quad \text{as } u \to 0.$$&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Characteristic Function of $Z_n$&lt;/strong&gt;: By independence and the scaling property ($a = 1/\sqrt{n}$):
$$\phi_{Z_n}(t) = \prod_{k=1}^n \phi_{Y_k}!\left(\frac{t}{\sqrt{n}}\right) = \left[\phi_Y!\left(\frac{t}{\sqrt{n}}\right)\right]^n.$$&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Pointwise Limit&lt;/strong&gt;: Fix any $t \in \mathbb{R}$. As $n \to \infty$, $u_n = t/\sqrt{n} \to 0$. Substituting the Taylor expansion at $u = t/\sqrt{n}$ yields:
$$\phi_{Z_n}(t) = \left[1 - \frac{t^2}{2n} + o!\left(\frac{t^2}{n}\right)\right]^n = \left[1 + \frac{-t^2/2 + n \cdot o(t^2/n)}{n}\right]^n.$$
Because $n \cdot o(t^2/n) \to 0$ as $n \to \infty$ for fixed $t$, the complex exponential limit $\lim_{n \to \infty}(1 + c_n/n)^n = e^c$ (when $c_n \to c$) gives:
$$\lim_{n \to \infty} \phi_{Z_n}(t) = e^{-t^2 / 2} \qquad \forall t \in \mathbb{R}.$$&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Conclusion&lt;/strong&gt;: Since $t \mapsto e^{-t^2/2}$ is continuous at $t = 0$ and is the characteristic function of $Z \sim \mathcal{N}(0, 1)$, Lévy&amp;rsquo;s Continuity Theorem implies $Z_n \xrightarrow{d} \mathcal{N}(0, 1)$. $\square$&lt;/li&gt;
&lt;/ol&gt;
&lt;hr&gt;
&lt;h2 id="when-the-clt-breaks-heavy-tails-and-cauchy-stability"&gt;When the CLT Breaks: Heavy Tails and Cauchy Stability
&lt;/h2&gt;&lt;p&gt;Look at step 1 of the proof: everything hinges on $\phi_Y(u)$ having a quadratic local expansion $1 - \frac{1}{2}u^2 + o(u^2)$ near the origin, which requires $\mathbb{E}[Y^2] &amp;lt; \infty$. What happens when the variance is infinite?&lt;/p&gt;
&lt;p&gt;Consider the standard &lt;strong&gt;Cauchy distribution&lt;/strong&gt; with density $f(x) = \frac{1}{\pi(1 + x^2)}$ on $\mathbb{R}$. Because $\int_{-\infty}^\infty \frac{|x|}{\pi(1+x^2)},dx = \infty$, neither its mean nor its variance exists. By residue calculus, its characteristic function is:&lt;/p&gt;
&lt;p&gt;$$\phi_X(t) = e^{-|t|},$$&lt;/p&gt;
&lt;p&gt;which has a sharp &lt;strong&gt;non-differentiable cusp&lt;/strong&gt; at $t = 0$ rather than a parabolic peak!&lt;/p&gt;
&lt;p&gt;Now compute the characteristic function of the sample mean $\bar{X}&lt;em&gt;n = \frac{1}{n}\sum&lt;/em&gt;{k=1}^n X_k$ of $n$ i.i.d. Cauchy variables:&lt;/p&gt;
&lt;p&gt;$$\phi_{\bar{X}_n}(t) = \left[\phi_X!\left(\frac{t}{n}\right)\right]^n = \left(e^{-|t|/n}\right)^n = e^{-|t|} = \phi_X(t).$$&lt;/p&gt;
&lt;p&gt;The sample mean of $n = 10^9$ independent Cauchy measurements has the &lt;strong&gt;exact same distribution as a single measurement ($n = 1$)&lt;/strong&gt;! Extreme tail spikes of magnitude $O(n)$ occur with probability $\Theta(1/n)$, completely overwhelming the averaging of typical values.&lt;/p&gt;
&lt;p&gt;More generally, the &lt;strong&gt;Gnedenko–Kolmogorov Generalized CLT&lt;/strong&gt; shows that if $X_i$ has power-law tails $\mathbb{P}(|X| &amp;gt; x) \sim c x^{-\alpha}$ with stability index $\alpha \in (0, 2)$, then $\phi_X(t) = 1 - c&amp;rsquo;|t|^\alpha + o(|t|^\alpha)$, so the sum must be normalized by $n^{1/\alpha}$ (rather than $n^{1/2}$) and converges to a symmetric &lt;strong&gt;$\alpha$-stable Lévy distribution&lt;/strong&gt; with characteristic function $e^{-c&amp;rsquo;|t|^\alpha}$.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="non-asymptotic-error-bounds-the-berryesseen-theorem"&gt;Non-Asymptotic Error Bounds: The Berry–Esseen Theorem
&lt;/h2&gt;&lt;p&gt;Even when $\sigma^2 &amp;lt; \infty$, convergence in distribution is an asymptotic statement: how large is the error $|F_{Z_n}(z) - \Phi(z)|$ at finite $n$?&lt;/p&gt;
&lt;p&gt;If we control the &lt;strong&gt;third absolute central moment&lt;/strong&gt; $\rho = \mathbb{E}[|X_1 - \mu|^3] &amp;lt; \infty$, we can replace the qualitative Peano remainder $o(u^2)$ in Taylor&amp;rsquo;s expansion with a quantitative $O(|u|^3)$ bound and invert via Esseen&amp;rsquo;s smoothing inequality:&lt;/p&gt;
&lt;h3 id="theorem-berryesseen-19411942"&gt;Theorem (Berry–Esseen, 1941–1942)
&lt;/h3&gt;&lt;p&gt;If $X_1, \dots, X_n$ are i.i.d. with mean $\mu$, variance $\sigma^2 &amp;gt; 0$, and third absolute central moment $\rho = \mathbb{E}[|X_1 - \mu|^3] &amp;lt; \infty$, then for every $n \ge 1$:&lt;/p&gt;
&lt;p&gt;$$\sup_{z \in \mathbb{R}} \left|\mathbb{P}(Z_n \le z) - \Phi(z)\right| \le \frac{C,\rho}{\sigma^3\sqrt{n}},$$&lt;/p&gt;
&lt;p&gt;where $C &amp;lt; 0.4748$ (Shevtsova, 2011) is a universal constant.&lt;/p&gt;
&lt;p&gt;The dimensionless factor $\rho / \sigma^3 \ge 1$ measures the asymmetry and tail heaviness of $X_1$:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;For symmetric $\text{Bernoulli}(0.5)$ trials, $\rho / \sigma^3 = 1$, and convergence is rapid.&lt;/li&gt;
&lt;li&gt;For highly skewed $\text{Bernoulli}(p)$ trials with tiny $p \ll 1$, $\rho / \sigma^3 \approx 1/\sqrt{p}$, warning us that $n$ must satisfy $n p \gg 1$ before the Gaussian approximation becomes trustworthy.&lt;/li&gt;
&lt;/ul&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References
&lt;/h2&gt;&lt;ol&gt;
&lt;li&gt;Billingsley, P. (1995). &lt;em&gt;Probability and Measure&lt;/em&gt; (3rd ed.). Wiley.&lt;/li&gt;
&lt;li&gt;Durrett, R. (2019). &lt;em&gt;Probability: Theory and Examples&lt;/em&gt; (5th ed.). Cambridge University Press.&lt;/li&gt;
&lt;li&gt;Feller, W. (1971). &lt;em&gt;An Introduction to Probability Theory and Its Applications, Vol. 2&lt;/em&gt; (2nd ed.). Wiley.&lt;/li&gt;
&lt;li&gt;Gnedenko, B. V., &amp;amp; Kolmogorov, A. N. (1968). &lt;em&gt;Limit Distributions for Sums of Independent Random Variables&lt;/em&gt;. Addison-Wesley.&lt;/li&gt;
&lt;li&gt;Shevtsova, I. (2011). On the absolute constants in the Berry–Esseen type inequalities for identically distributed summands. &lt;em&gt;arXiv:1111.6554&lt;/em&gt;.&lt;/li&gt;
&lt;/ol&gt;</description></item></channel></rss>