UNIT V: Probability: Inequalities and Limit Theorems

Chebychev's Inequality and its Applications
Convergence in Probability
Convergence in Distribution
Weak Law of Large Numbers (WLLN)
Strong Law of Large Numbers (SLLN)
Bernoulli's Law of Large Numbers
Central Limit Theorem (CLT)

Chebychev's Inequality and its Applications

Chebychev's inequality is a fundamental concept in probability theory that provides a lower bound on the probability that a random variable deviates from its mean by a certain amount. It is powerful because it applies to any probability distribution for which the mean and variance are finite, without needing to know the exact distribution.

Conceptual Explanation

Imagine you have a random variable X. Chebychev's inequality tells us that the probability of X being far away from its expected value (mean) is relatively small. Specifically, it gives an upper bound for the probability that the absolute difference between X and its mean (μ) is greater than or equal to some positive value (ε).

Chebychev's Inequality Statement:

For a random variable X with finite expected value E[X] = μ and finite non-zero variance Var(X) = σ², for any positive real number ε (epsilon), the probability that X deviates from its mean by at least ε is:

P(|X - μ| ≥ ε) ≤ σ² / ε²

Alternatively, if we let ε = kσ for some k > 0 (where k is the number of standard deviations), the inequality can be written as:

P(|X - μ| ≥ kσ) ≤ 1 / k²

Key Observations

It applies to any random variable with a finite mean and variance.
It provides an upper bound for the probability of deviation, meaning the actual probability might be much smaller.
The larger the deviation (ε or k), the smaller the upper bound on the probability, which makes intuitive sense.
The larger the variance (σ²), the looser the bound, meaning more variability allows for greater deviation.

Practical Examples and Applications

Quality Control: Suppose a machine produces bolts with a mean length of 5 cm and a standard deviation of 0.1 cm. We want to estimate the probability that a randomly selected bolt has a length outside the range [4.8 cm, 5.2 cm].
- μ = 5 cm, σ = 0.1 cm.
- We are interested in |X - 5| ≥ 0.2 cm (since 5.2 - 5 = 0.2 and 5 - 4.8 = 0.2). So, ε = 0.2.
- Using Chebychev's inequality: P(|X - 5| ≥ 0.2) ≤ (0.1)² / (0.2)² = 0.01 / 0.04 = 0.25.
- This means there is at most a 25% chance that a bolt's length will be outside the range [4.8 cm, 5.2 cm].
Polling and Surveys: In a political poll, we might know the average support for a candidate (μ) and the variability (σ). Chebychev's inequality can be used to set a bound on the probability that the actual support deviates significantly from the poll's average, even if we don't assume a normal distribution for the polling errors.
Risk Management: In finance, it can be used to estimate the maximum probability of a portfolio's return deviating by a certain amount from its expected return, providing a conservative estimate of risk.

Convergence in Probability

Convergence in probability is one of the modes of convergence for a sequence of random variables. It describes how a sequence of random variables (X_n) behaves as n approaches infinity.

Definition of Convergence in Probability:

A sequence of random variables X₁, X₂, ..., X_n, ... is said to converge in probability to a random variable X if, for every ε > 0, the following holds:

lim_n→∞ P(|X_n - X| ≥ ε) = 0

This is often written as X_n →^P X.

Explanation

In simple terms, convergence in probability means that as n gets very large, the probability that X_n is "far away" from X becomes extremely small and eventually approaches zero. It implies that the values of the random variables in the sequence are increasingly likely to be close to the limiting random variable X.

Convergence in Distribution

Convergence in distribution (also known as convergence in law) describes the convergence of the cumulative distribution functions (CDFs) of a sequence of random variables.

Definition of Convergence in Distribution:

A sequence of random variables X₁, X₂, ..., X_n, ... is said to converge in distribution to a random variable X if, for every point x where the cumulative distribution function F_X(x) of X is continuous, the following holds:

lim_n→∞ F_{X_n}(x) = F_X(x)

This is often written as X_n →^D X or X_n →^L X.

Explanation

This type of convergence means that the probability distribution of X_n becomes more and more similar to the probability distribution of X as n increases. It focuses on the behavior of the distribution functions rather than the random variables themselves. Convergence in probability implies convergence in distribution, but the reverse is not always true.

Weak Law of Large Numbers (WLLN)

The Weak Law of Large Numbers (WLLN) is a fundamental theorem in probability theory that describes how the average of a sequence of random variables converges to the expected value of those variables as the number of variables increases.

Conceptual Explanation

The WLLN states that if you take a large number of independent and identically distributed (i.i.d.) random variables, their sample mean will converge to their common expected value. This convergence is in probability, meaning that as you collect more and more data, the probability that your sample average is significantly different from the true population average becomes very small.

Weak Law of Large Numbers Statement:

Let X₁, X₂, ..., X_n be a sequence of independent and identically distributed (i.i.d.) random variables, each with a finite mean E[X_i] = μ and a finite variance Var(X_i) = σ².

Let S_n = (1/n) ∑_i=1ⁿ X_i be the sample mean. Then, the sample mean S_n converges in probability to the population mean μ:

lim_n→∞ P(|S_n - μ| ≥ ε) = 0 for every ε > 0

Or simply, S_n →^P μ.

Conditions for WLLN

The random variables must be independent.
The random variables must be identically distributed (i.i.d.).
Each random variable must have a finite mean (μ).
Each random variable must have a finite variance (σ²). While some versions exist with only finite mean, the most common form assumes finite variance, especially when proved using Chebychev's inequality.

Connection to Chebychev's Inequality

The WLLN can be proven using Chebychev's inequality. For the sample mean S_n:

E[S_n] = μ
Var(S_n) = Var((1/n) ∑ X_i) = (1/n²) ∑ Var(X_i) = (1/n²) nσ² = σ² / n

Applying Chebychev's inequality to S_n:

P(|S_n - μ| ≥ ε) ≤ Var(S_n) / ε² = (σ² / n) / ε² = σ² / (nε²)

As n → ∞, σ² / (nε²) → 0. Therefore, P(|S_n - μ| ≥ ε) → 0, which is the definition of convergence in probability. This demonstrates the WLLN.

Real-world Application

When you flip a fair coin many times, the proportion of heads will get closer and closer to 0.5. The WLLN mathematically justifies this observation: the sample mean (proportion of heads) converges to the true probability (0.5) in probability.

Strong Law of Large Numbers (SLLN)

The Strong Law of Large Numbers (SLLN) is a more powerful result than the WLLN. It states that the sample mean converges to the expected value not just in probability, but "almost surely".

Strong Law of Large Numbers (SLLN) Statement:

Let X₁, X₂, ..., X_n be a sequence of independent and identically distributed (i.i.d.) random variables, each with a finite mean E[X_i] = μ.

Let S_n = (1/n) ∑_i=1ⁿ X_i be the sample mean. Then, the sample mean S_n converges almost surely to the population mean μ:

P(lim_n→∞ S_n = μ) = 1

This is often written as S_n →^a.s. μ or S_n →^P μ (a.s. implies convergence in probability).

Distinction from WLLN

The term "almost surely" means that the event of the sample mean converging to the true mean will happen with probability 1. While WLLN says that the probability of the sample mean deviating decreases to zero, SLLN states that the sequence of sample means will eventually stay arbitrarily close to the true mean forever, except possibly on a set of outcomes with probability zero.

Stronger Implication: If a sequence converges almost surely, it also converges in probability. However, convergence in probability does not necessarily imply almost sure convergence.

Conditions: SLLN generally requires only a finite mean, though some versions might have stricter conditions (e.g., E[|X|] < ∞ for some common versions).

Bernoulli's Law of Large Numbers

Bernoulli's Law of Large Numbers is a specific case of the Weak Law of Large Numbers, particularly applied to Bernoulli trials.

Conceptual Explanation

It states that if you repeat a Bernoulli experiment (an experiment with only two possible outcomes, "success" or "failure") many times, the proportion of successes observed in your trials will approach the true probability of success for a single trial.

Bernoulli's Law of Large Numbers Statement:

Let X₁, X₂, ..., X_n be a sequence of independent Bernoulli random variables, each with the same probability of success p. That is, P(X_i = 1) = p and P(X_i = 0) = 1 - p.

Let S_n = (1/n) ∑_i=1ⁿ X_i be the sample proportion of successes. Then, for any ε > 0:

lim_n→∞ P(|S_n - p| ≥ ε) = 0

Or simply, S_n →^P p.

Relationship to WLLN

This is a direct application of the WLLN where:

The random variables X_i are i.i.d. Bernoulli(p).
The mean E[X_i] = p.
The variance Var(X_i) = p(1-p), which is finite.

Thus, the WLLN directly implies Bernoulli's Law of Large Numbers for the sample proportion.

Practical Example

If you repeatedly flip a fair coin (where the probability of heads, p, is 0.5), as the number of flips increases, the proportion of heads you observe will converge to 0.5. This law underpins the idea that empirical observation can lead to an understanding of underlying probabilities.

Central Limit Theorem (CLT)

The Central Limit Theorem (CLT) is one of the most powerful and widely used theorems in statistics and probability theory. It states that the distribution of the sample mean of a large number of independent and identically distributed random variables approaches a normal distribution, regardless of the original distribution of the individual variables.

Central Limit Theorem (CLT) Definition:

Let X₁, X₂, ..., X_n be a sequence of independent and identically distributed (i.i.d.) random variables, each with a finite mean E[X_i] = μ and a finite non-zero variance Var(X_i) = σ².

Let S_n = ∑_i=1ⁿ X_i be the sum of these random variables. Then, as n approaches infinity, the distribution of the standardized sum Z_n approaches the standard normal distribution N(0, 1).

Z_n = (S_n - nμ) / (σ√n) →^D N(0, 1)

Alternatively, in terms of the sample mean (X̄_n = S_n/n):

Z_n = (X̄_n - μ) / (σ/√n) →^D N(0, 1)

Key Aspects and Conditions

Independence and Identical Distribution (i.i.d.): The random variables must be independent and come from the same distribution.
Finite Mean (μ) and Variance (σ²): Both the mean and variance of the individual random variables must be finite.
Large n: The theorem holds as n tends to infinity. In practice, for many distributions, n ≥ 30 is often considered large enough for the approximation to be reasonable.
Approximation to Normal Distribution: The distribution of the sum (or average) of the random variables becomes increasingly bell-shaped (normal) as n increases, even if the original population distribution is not normal.

Implications and Applications

The CLT is why the normal distribution is so prevalent in statistical inference. Even if individual data points don't follow a normal distribution, the means of samples taken from that data will tend to be normally distributed.
It allows us to use normal distribution-based statistical methods (like constructing confidence intervals or performing hypothesis tests for means) even when the underlying population distribution is unknown or non-normal, provided the sample size is sufficiently large.
It explains why many natural phenomena (e.g., heights, measurement errors) exhibit normal distributions, as they are often the result of many small, independent random effects summing up.