<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Theoretical-CS on LCS.Developer</title><link>https://www.lcs-dev.org/tags/theoretical-cs/</link><description>Recent content in Theoretical-CS on LCS.Developer</description><generator>Hugo -- gohugo.io</generator><language>en-us</language><copyright>LCS.Developer</copyright><lastBuildDate>Tue, 18 Mar 2025 20:00:00 +0000</lastBuildDate><atom:link href="https://www.lcs-dev.org/tags/theoretical-cs/index.xml" rel="self" type="application/rss+xml"/><item><title>Concentration Inequalities: From Markov and Chebyshev to Chernoff Bounds</title><link>https://www.lcs-dev.org/posts/concentration-inequalities---from-markov-and-chebyshev-to-chernoff-bounds/</link><pubDate>Tue, 18 Mar 2025 20:00:00 +0000</pubDate><guid>https://www.lcs-dev.org/posts/concentration-inequalities---from-markov-and-chebyshev-to-chernoff-bounds/</guid><description>&lt;h2 id="beyond-asymptotic-convergence"&gt;Beyond Asymptotic Convergence
&lt;/h2&gt;&lt;p&gt;Let $X_1, X_2, \dots, X_n$ be independent and identically distributed (i.i.d.) random variables with finite mean $\mu = \mathbb{E}[X_i]$. The &lt;strong&gt;Weak Law of Large Numbers (WLLN)&lt;/strong&gt; guarantees that the sample mean $\bar{X}&lt;em&gt;n = \frac{1}{n}\sum&lt;/em&gt;{i=1}^n X_i$ converges in probability to $\mu$:&lt;/p&gt;
&lt;p&gt;$$\forall \varepsilon &amp;gt; 0, \qquad \lim_{n \to \infty} \mathbb{P}!\left(|\bar{X}_n - \mu| \ge \varepsilon\right) = 0.$$&lt;/p&gt;
&lt;p&gt;In theoretical computer science—whether we are analyzing Randomized QuickSort, universal hashing, load balancing, or Monte Carlo error amplification—asymptotic convergence as $n \to \infty$ is not enough. An algorithm executes a &lt;strong&gt;finite&lt;/strong&gt; number of steps $n$, and we need an explicit, non-asymptotic upper bound on the tail probability $\mathbb{P}(|\bar{X}_n - \mu| \ge t)$.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Concentration inequalities&lt;/strong&gt; provide these finite-sample guarantees. They form a strict hierarchy: the higher the moments we control, the faster the tail probability decays.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="1-markovs-inequality-first-moment"&gt;1. Markov&amp;rsquo;s Inequality (First Moment)
&lt;/h2&gt;&lt;p&gt;Suppose all we know about a random variable $X$ is that it is non-negative ($X \ge 0$) and has finite expectation $\mathbb{E}[X]$.&lt;/p&gt;
&lt;h3 id="theorem-markovs-inequality"&gt;Theorem (Markov&amp;rsquo;s Inequality)
&lt;/h3&gt;&lt;p&gt;For any non-negative random variable $X \ge 0$ and any threshold $a &amp;gt; 0$:&lt;/p&gt;
&lt;p&gt;$$\mathbb{P}(X \ge a) \le \frac{\mathbb{E}[X]}{a}.$$&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Proof.&lt;/strong&gt; Let $\mathbb{I}&lt;em&gt;{{X \ge a}}$ be the indicator variable equal to $1$ if $X \ge a$ and $0$ otherwise. Because $X \ge 0$, the deterministic inequality $a \cdot \mathbb{I}&lt;/em&gt;{{X \ge a}} \le X$ holds at every point of the sample space. Taking expectations on both sides yields:&lt;/p&gt;
&lt;p&gt;$$a \cdot \mathbb{E}!\left[\mathbb{I}_{{X \ge a}}\right] = a \cdot \mathbb{P}(X \ge a) \le \mathbb{E}[X]. \quad \square$$&lt;/p&gt;
&lt;h3 id="algorithmic-application-las-vegas-to-monte-carlo"&gt;Algorithmic Application: Las Vegas to Monte Carlo
&lt;/h3&gt;&lt;p&gt;A &lt;strong&gt;Las Vegas&lt;/strong&gt; randomized algorithm always returns the correct answer, but its running time $T$ is a random variable with expected value $\mathbb{E}[T] = \mu$. By Markov&amp;rsquo;s inequality, stopping the algorithm if it has not finished after $2\mu$ steps aborts with probability at most $\mathbb{P}(T \ge 2\mu) \le 1/2$. Repeating $k$ independent runs of length $2\mu$ reduces the failure probability from $1/2$ to $2^{-k}$ while bounding worst-case runtime by $2k\mu$.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="2-chebyshevs-inequality-second-moment"&gt;2. Chebyshev&amp;rsquo;s Inequality (Second Moment)
&lt;/h2&gt;&lt;p&gt;Markov&amp;rsquo;s inequality decays only as $O(1/a)$ and yields no improvement when applied directly to a sum $S_n = \sum_{i=1}^n X_i$, because $\mathbb{E}[S_n]$ and the threshold $n a$ both scale linearly with $n$. To capture cancellation between fluctuations above and below the mean, we move to the &lt;strong&gt;second central moment&lt;/strong&gt;: the variance $\sigma^2 = \text{Var}(X) = \mathbb{E}[(X - \mu)^2]$.&lt;/p&gt;
&lt;h3 id="theorem-chebyshevs-inequality"&gt;Theorem (Chebyshev&amp;rsquo;s Inequality)
&lt;/h3&gt;&lt;p&gt;For any random variable $X$ with finite mean $\mu$ and variance $\sigma^2$, and any $t &amp;gt; 0$:&lt;/p&gt;
&lt;p&gt;$$\mathbb{P}(|X - \mu| \ge t) \le \frac{\text{Var}(X)}{t^2}.$$&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Proof.&lt;/strong&gt; Since $|X - \mu| \ge t \iff (X - \mu)^2 \ge t^2$ and $(X - \mu)^2 \ge 0$, applying Markov&amp;rsquo;s inequality to the non-negative variable $Y = (X - \mu)^2$ with threshold $a = t^2$ gives:&lt;/p&gt;
&lt;p&gt;$$\mathbb{P}(|X - \mu| \ge t) = \mathbb{P}!\left((X - \mu)^2 \ge t^2\right) \le \frac{\mathbb{E}[(X - \mu)^2]}{t^2} = \frac{\text{Var}(X)}{t^2}. \quad \square$$&lt;/p&gt;
&lt;h3 id="pairwise-independence-and-the-weak-law-of-large-numbers"&gt;Pairwise Independence and the Weak Law of Large Numbers
&lt;/h3&gt;&lt;p&gt;If $X_1, \dots, X_n$ are &lt;strong&gt;pairwise independent&lt;/strong&gt; with mean $\mu$ and variance $\sigma^2$, their covariance terms vanish: $\text{Var}(\bar{X}&lt;em&gt;n) = \frac{1}{n^2}\sum&lt;/em&gt;{i=1}^n \text{Var}(X_i) = \frac{\sigma^2}{n}$. Chebyshev&amp;rsquo;s inequality then gives:&lt;/p&gt;
&lt;p&gt;$$\mathbb{P}(|\bar{X}_n - \mu| \ge \varepsilon) \le \frac{\sigma^2}{n\varepsilon^2}.$$&lt;/p&gt;
&lt;p&gt;This immediately proves the Weak Law of Large Numbers and shows that &lt;strong&gt;pairwise independence alone&lt;/strong&gt; guarantees $O(1/n)$ polynomial tail decay—a fact heavily exploited in derandomization and 2-universal hash families where full mutual independence is too costly to construct.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="3-chernoff-bounds-exponential-moments"&gt;3. Chernoff Bounds (Exponential Moments)
&lt;/h2&gt;&lt;p&gt;When $X_1, \dots, X_n$ are &lt;strong&gt;mutually independent&lt;/strong&gt; and bounded, $O(1/n)$ polynomial decay is still vastly pessimistic. Because full independence implies that the expectation of a product equals the product of expectations, we can apply Markov&amp;rsquo;s inequality to the &lt;strong&gt;Moment Generating Function (MGF)&lt;/strong&gt; $\mathbb{E}[e^{\lambda S_n}]$—a technique due to Sergei Bernstein and Herman Chernoff.&lt;/p&gt;
&lt;p&gt;For any parameter $\lambda &amp;gt; 0$, since $x \mapsto e^{\lambda x}$ is strictly increasing:&lt;/p&gt;
&lt;p&gt;$$\mathbb{P}(S_n \ge a) = \mathbb{P}!\left(e^{\lambda S_n} \ge e^{\lambda a}\right) \le e^{-\lambda a},\mathbb{E}!\left[\prod_{i=1}^n e^{\lambda X_i}\right] = e^{-\lambda a}\prod_{i=1}^n \mathbb{E}!\left[e^{\lambda X_i}\right].$$&lt;/p&gt;
&lt;h3 id="theorem-multiplicative-chernoff-bounds-for-poisson-trials"&gt;Theorem (Multiplicative Chernoff Bounds for Poisson Trials)
&lt;/h3&gt;&lt;p&gt;Let $X_1, \dots, X_n$ be mutually independent Bernoulli trials with $\mathbb{P}(X_i = 1) = p_i$ and $\mathbb{P}(X_i = 0) = 1 - p_i$. Let $S_n = \sum_{i=1}^n X_i$ and $\mu = \mathbb{E}[S_n] = \sum_{i=1}^n p_i$. For any $\delta \in (0, 1)$:&lt;/p&gt;
&lt;p&gt;$$\mathbb{P}!\left(S_n \ge (1 + \delta)\mu\right) \le \left(\frac{e^\delta}{(1+\delta)^{1+\delta}}\right)^\mu \le \exp!\left(-\frac{\delta^2 \mu}{3}\right),$$&lt;/p&gt;
&lt;p&gt;$$\mathbb{P}!\left(S_n \le (1 - \delta)\mu\right) \le \left(\frac{e^{-\delta}}{(1-\delta)^{1-\delta}}\right)^\mu \le \exp!\left(-\frac{\delta^2 \mu}{2}\right).$$&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Proof (Upper Tail).&lt;/strong&gt; For each Bernoulli variable $X_i$, using the inequality $1 + x \le e^x$:&lt;/p&gt;
&lt;p&gt;$$\mathbb{E}!\left[e^{\lambda X_i}\right] = (1 - p_i) + p_i e^\lambda = 1 + p_i(e^\lambda - 1) \le \exp!\left(p_i(e^\lambda - 1)\right).$$&lt;/p&gt;
&lt;p&gt;Multiplying across all $n$ independent variables gives $\mathbb{E}[e^{\lambda S_n}] \le \exp(\mu(e^\lambda - 1))$. Substituting $a = (1+\delta)\mu$ into the exponential Markov bound yields:&lt;/p&gt;
&lt;p&gt;$$\mathbb{P}!\left(S_n \ge (1+\delta)\mu\right) \le \exp!\left(\mu(e^\lambda - 1) - \lambda(1+\delta)\mu\right).$$&lt;/p&gt;
&lt;p&gt;To minimize the exponent with respect to $\lambda &amp;gt; 0$, we differentiate $\mu(e^\lambda - 1 - \lambda(1+\delta))$ and set the derivative to zero, obtaining the optimal tilting parameter $\lambda^* = \ln(1 + \delta) &amp;gt; 0$. Substituting $\lambda^* = \ln(1 + \delta)$ gives:&lt;/p&gt;
&lt;p&gt;$$\mathbb{P}!\left(S_n \ge (1+\delta)\mu\right) \le \left(\frac{e^\delta}{(1+\delta)^{1+\delta}}\right)^\mu.$$&lt;/p&gt;
&lt;p&gt;Finally, the elementary Taylor bound $\delta - (1+\delta)\ln(1+\delta) \le -\frac{\delta^2}{2 + 2\delta/3} \le -\frac{\delta^2}{3}$ for $\delta \in (0, 1)$ yields $\exp(-\delta^2\mu / 3)$. $\square$&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="quantitative-comparison"&gt;Quantitative Comparison
&lt;/h2&gt;&lt;p&gt;Consider $n = 1000$ independent fair coin flips ($\mu = 500$, $\sigma^2 = 250$), and suppose we want to bound the probability of observing at least $750$ heads ($\delta = 0.5$):&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style="text-align: left"&gt;Inequality&lt;/th&gt;
&lt;th style="text-align: left"&gt;Moments Used&lt;/th&gt;
&lt;th style="text-align: left"&gt;Independence Needed&lt;/th&gt;
&lt;th style="text-align: left"&gt;Upper Bound on $\mathbb{P}(S_{1000} \ge 750)$&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style="text-align: left"&gt;&lt;strong&gt;Markov&lt;/strong&gt;&lt;/td&gt;
&lt;td style="text-align: left"&gt;$\mathbb{E}[S_n]$&lt;/td&gt;
&lt;td style="text-align: left"&gt;None&lt;/td&gt;
&lt;td style="text-align: left"&gt;$\le \frac{500}{750} \approx 0.6667$&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align: left"&gt;&lt;strong&gt;Chebyshev&lt;/strong&gt;&lt;/td&gt;
&lt;td style="text-align: left"&gt;$\mathbb{E}[S_n], \text{Var}(S_n)$&lt;/td&gt;
&lt;td style="text-align: left"&gt;Pairwise ($2$-wise)&lt;/td&gt;
&lt;td style="text-align: left"&gt;$\le \frac{250}{250^2} = 0.0040$&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align: left"&gt;&lt;strong&gt;Chernoff&lt;/strong&gt;&lt;/td&gt;
&lt;td style="text-align: left"&gt;$\mathbb{E}[e^{\lambda S_n}]$&lt;/td&gt;
&lt;td style="text-align: left"&gt;Mutual ($n$-wise)&lt;/td&gt;
&lt;td style="text-align: left"&gt;$\le e^{-500(0.25)/3} \approx 7.89 \times 10^{-19}$&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Moving from second moments to exponential moments tightens the bound by sixteen orders of magnitude—turning a weak constant probability into certainty at algorithmic scale.&lt;/p&gt;</description></item></channel></rss>