UNIT-II: Expectation, Variance, Moments, and Generating Functions

Expectation of Random Variables and its Properties

1.1. Definition of Expectation

The expectation (or expected value or mean) of a random variable is the weighted average of all possible values that the random variable can take on. The weights are the probabilities associated with each value. It represents the central tendency of the distribution.

For a Discrete Random Variable X

If X is a discrete random variable with probability mass function (PMF) P(X=x), its expectation is:

E[X] = ∑_x x P(X=x)

The sum is over all possible values x that X can take.

For a Continuous Random Variable X

If X is a continuous random variable with probability density function (PDF) f(x), its expectation is:

E[X] = ∫_-∞^∞ x f(x) dx

Expectation of a Function of a Random Variable

For a function g(X) of a discrete random variable X:

E[g(X)] = ∑_x g(x) P(X=x)

For a function g(X) of a continuous random variable X:

E[g(X)] = ∫_-∞^∞ g(x) f(x) dx

1.2. Properties of Expectation

Expectation is a linear operator and possesses several important properties:

• Expectation of a Constant: The expected value of a constant 'c' is the constant itself.

E[c] = c

• Expectation of a Constant times a Random Variable: If 'c' is a constant and X is a random variable.

E[cX] = c E[X]

• Expectation of a Sum/Difference of Random Variables: For any two random variables X and Y (dependent or independent).

E[X + Y] = E[X] + E[Y]

E[X - Y] = E[X] - E[Y]

• Expectation of a Linear Combination: For constants a, b and random variables X, Y.

E[aX + bY] = a E[X] + b E[Y]

• Expectation of a Product of Independent Random Variables: If X and Y are independent random variables.

E[XY] = E[X] E[Y]

Important Note: This property does NOT generally hold if X and Y are dependent.

Variance of Random Variables and its Properties

2.1. Definition of Variance

Variance measures the dispersion or spread of a random variable's values around its expected value (mean). A high variance indicates that data points are generally very far from the mean, while a low variance indicates that data points are clustered closely around the mean.

The variance of a random variable X, denoted as Var(X) or σ²_X, is defined as the expected value of the squared deviation from the mean:

Var(X) = E[(X - E[X])²]

An alternative and often more convenient formula for calculation is:

Var(X) = E[X²] - (E[X])²

Where E[X²] is the expected value of X squared, and (E[X])² is the square of the expected value of X.

The standard deviation, σ_X, is the positive square root of the variance: σ_X = √Var(X).

2.2. Properties of Variance

• Variance of a Constant: The variance of a constant 'c' is zero, as a constant has no spread.

Var(c) = 0

• Variance of a Constant times a Random Variable: If 'c' is a constant and X is a random variable.

Var(cX) = c² Var(X)

• Variance of a Sum/Difference of Independent Random Variables: If X and Y are independent random variables.

Var(X + Y) = Var(X) + Var(Y)

Var(X - Y) = Var(X) + Var(Y)

Important Note: For dependent variables, covariance terms are introduced. Since covariance is not explicitly in the syllabus, we stick to the independent case here.

• Variance of a Linear Combination: For constants a, b and independent random variables X, Y.

Var(aX + bY) = a² Var(X) + b² Var(Y)

• Adding a Constant Does Not Change Variance: Adding a constant to a random variable shifts its distribution but does not change its spread.

Var(X + c) = Var(X)

Moments of Random Variables

Moments are a set of statistical parameters that characterize the shape and features of a probability distribution. They provide information about the central tendency, dispersion, skewness, and kurtosis of the distribution.

3.1. Raw Moments (Moments about the Origin)

The k^th raw moment (or moment about the origin) of a random variable X, denoted as μ'_k, is defined as:

μ'_k = E[X^k]

• For k=1: μ'₁ = E[X] (the mean of the distribution).
• For k=2: μ'₂ = E[X²].
• For k=3: μ'₃ = E[X³].
• And so on.

These moments exist if E[|X|^k] is finite.

3.2. Central Moments (Moments about the Mean)

Central moments describe the shape of the distribution by removing the effect of its location (mean).

The k^th central moment (or moment about the mean) of a random variable X, denoted as μ_k, is defined as:

μ_k = E[(X - E[X])^k]

• For k=1: μ₁ = E[X - E[X]] = E[X] - E[X] = 0. The first central moment is always zero.
• For k=2: μ₂ = E[(X - E[X])²] = Var(X). The second central moment is the variance.
• For k=3: μ₃ is related to the skewness of the distribution.
• For k=4: μ₄ is related to the kurtosis of the distribution.

3.3. Relationship Between Raw and Central Moments

Central moments can be expressed in terms of raw moments:

• μ₁ = 0
• μ₂ = μ'₂ - (μ'₁)² (This is Var(X) = E[X²] - (E[X])²)
• μ₃ = μ'₃ - 3μ'₂μ'₁ + 2(μ'₁)³
• μ₄ = μ'₄ - 4μ'₃μ'₁ + 6μ'₂(μ'₁)² - 3(μ'₁)⁴

Conversely, raw moments can be expressed in terms of central moments and the mean (μ = μ'₁):

• μ'₁ = μ
• μ'₂ = μ₂ + μ²
• μ'₃ = μ₃ + 3μμ₂ + μ³
• μ'₄ = μ₄ + 4μμ₃ + 6μ²μ₂ + μ⁴

Cumulants of Random Variables

Cumulants are another set of statistical parameters related to moments, which also characterize the shape of a probability distribution. They often have simpler additive properties for sums of independent random variables compared to moments.

4.1. Definition of Cumulants

Cumulants, denoted by κ_k (kappa), are typically defined through the Cumulant Generating Function (CGF).

The k^th cumulant, κ_k, is the coefficient of (t^k / k!) in the Taylor series expansion of the Cumulant Generating Function, K(t) = log(M_X(t)).

4.2. Relationship Between Cumulants and Moments

The first few cumulants are related to the central moments (and raw moments) as follows:

• κ₁ = μ'₁ = E[X] (The first cumulant is the mean)
• κ₂ = μ₂ = Var(X) (The second cumulant is the variance)
• κ₃ = μ₃ (The third cumulant is the third central moment, related to skewness)
• κ₄ = μ₄ - 3μ²₂ (The fourth cumulant is related to kurtosis)

Key Observation: For a normal distribution, all cumulants higher than the second (κ₃, κ₄, ...) are zero. This makes cumulants particularly useful in characterizing non-normality.

Property: Cumulants are additive for independent random variables. If X₁, ..., X_n are independent random variables, then the k^th cumulant of their sum S = X₁ + ... + X_n is the sum of their individual k^th cumulants: κ_k(S) = ∑_i=1ⁿ κ_k(X_i).

Moment Generating Function (MGF)

The Moment Generating Function (MGF) is a powerful tool in probability theory for characterizing probability distributions. It can generate all the moments of a random variable, provided it exists.

5.1. Definition of MGF

The Moment Generating Function (MGF) of a random variable X, denoted as M_X(t), is defined as the expected value of e^tX, for all real values of 't' for which the expectation exists:

M_X(t) = E[e^tX]

For a Discrete Random Variable X

M_X(t) = ∑_x e^tx P(X=x)

For a Continuous Random Variable X

M_X(t) = ∫_-∞^∞ e^tx f(x) dx

The MGF might not exist for all random variables. If it exists in some open interval around t=0, then all moments exist.

5.2. Properties of MGF

• Generating Moments: The most significant property of MGF is its ability to generate moments. The k^th raw moment can be obtained by taking the k^th derivative of the MGF with respect to 't' and then setting t=0.

μ'_k = E[X^k] = d^k/dt^k [M_X(t)] |_t=0

For example:

• E[X] = M'_X(0) (First derivative at t=0)
• E[X²] = M''_X(0) (Second derivative at t=0)
• Uniqueness Property: If two random variables have the same MGF, then they have the same probability distribution. This property is crucial for identifying distributions.
• MGF of a Linear Transformation: If Y = aX + b, where a and b are constants.

M_Y(t) = E[e^t(aX+b)] = E[e^atX e^bt] = e^bt E[e^atX] = e^bt M_X(at)

• MGF of a Sum of Independent Random Variables: If X₁, X₂, ..., X_n are independent random variables, and S = X₁ + X₂ + ... + X_n, then the MGF of their sum is the product of their individual MGFs.

M_S(t) = M_X1(t) × M_X2(t) × ... × M_Xn(t)

• MGF at t=0: M_X(0) = E[e^0X] = E[1] = 1.

Cumulants Generating Function (CGF)

The Cumulant Generating Function (CGF) is closely related to the MGF and is particularly useful because cumulants exhibit additive properties for independent random variables, similar to how MGFs multiply for sums of independent random variables.

6.1. Definition of CGF

The Cumulant Generating Function (CGF) of a random variable X, denoted as K_X(t), is defined as the natural logarithm of its Moment Generating Function (MGF):

K_X(t) = log(M_X(t))

The CGF exists whenever the MGF exists and is positive around t=0.

6.2. Properties of CGF

• Generating Cumulants: Similar to MGF for moments, the k^th cumulant can be obtained by taking the k^th derivative of the CGF with respect to 't' and then setting t=0.

κ_k = d^k/dt^k [K_X(t)] |_t=0

For example:

• κ₁ = K'_X(0) = E[X] (Mean)
• κ₂ = K''_X(0) = Var(X) (Variance)
• κ₃ = K'''_X(0) (Third central moment)
• κ₄ = K''''_X(0) (Related to kurtosis)
• CGF of a Sum of Independent Random Variables: If X₁, X₂, ..., X_n are independent random variables, and S = X₁ + X₂ + ... + X_n, then the CGF of their sum is the sum of their individual CGFs.

K_S(t) = K_X1(t) + K_X2(t) + ... + K_Xn(t)

This additive property is a major advantage of CGF over MGF when dealing with sums of independent random variables.

• Relationship with MGF: Directly defined by K_X(t) = log(M_X(t)), which implies M_X(t) = e^K_X(t).