Unit 4: The Chi-Square Distribution: Theory and Applications

Introduction to Exact Sampling Distributions

In inferential statistics, an exact sampling distribution refers to the probability distribution of a statistic when the sample size is finite and the population distribution is precisely known. Unlike asymptotic distributions, which are approximations for large sample sizes, exact sampling distributions provide precise probabilities regardless of sample size (provided the underlying assumptions are met). The Chi-Square (χ²) distribution is one such fundamental exact sampling distribution.

The Chi-Square (χ²) Distribution

Definition

The Chi-Square (χ²) distribution with n degrees of freedom (d.f.) is defined as the distribution of the sum of the squares of n independent standard normal random variables.

If Z₁, Z₂, ..., Z_n are independent and identically distributed standard normal random variables (i.e., Z_i ~ N(0, 1)), then the random variable X defined as:

X = Z₁² + Z₂² + ... + Z_n² = ∑_i=1ⁿ Z_i²

follows a Chi-Square distribution with n degrees of freedom, denoted as X ~ χ²(n).

Probability Density Function (p.d.f.)

Derivation of p.d.f. using Moment Generating Function (m.g.f.)

The p.d.f. of the χ² distribution can be derived using the moment generating function (m.g.f.).

1. M.g.f. of the Square of a Standard Normal Variable:
   Let Z ~ N(0, 1). The random variable Y = Z² has a p.d.f. given by:
   f_Y(y) = (1 / (Γ(1/2) 2^1/2)) y^(1/2)-1 e^-y/2 for y > 0.
   This is a Gamma distribution with shape parameter α = 1/2 and scale parameter β = 2. Its m.g.f. is:
   

   M_Y(t) = (1 - 2t)^-1/2 for t < 1/2.

2. M.g.f. of the Sum of Independent Variables:
   If X = ∑_i=1ⁿ Z_i², and each Z_i² is independently χ²(1) (or Gamma(1/2, 2)), then the m.g.f. of X is the product of the individual m.g.f.s:
   

   M_X(t) = E[e^tX] = E[e^{t ∑_i=1n Z_i2}] = ∏_i=1ⁿ E[e^{t Z_i2}]
   M_X(t) = ∏_i=1ⁿ (1 - 2t)^-1/2 = (1 - 2t)^-n/2 for t < 1/2.

3. Identifying the Distribution from the M.g.f.:
   The m.g.f. M_X(t) = (1 - 2t)^-n/2 is the m.g.f. of a Gamma distribution with shape parameter α = n/2 and scale parameter β = 2.
4. Probability Density Function of χ²(n):
   Therefore, the p.d.f. of a Chi-Square distribution with n degrees of freedom is the p.d.f. of a Gamma(α = n/2, β = 2) distribution:
   

   f(x; n) = (1 / (Γ(n/2) 2^n/2)) x^(n/2)-1 e^-x/2 for x > 0, and 0 otherwise.

   Here, Γ(n/2) is the Gamma function evaluated at n/2. The parameter n is the degrees of freedom (d.f.).

Properties of the Chi-Square (χ²) Distribution

Let X ~ χ²(n).

• Mean:
  

  E[X] = n

  The mean of a χ² distribution is equal to its degrees of freedom.

• Variance:
  

  Var[X] = 2n

  The variance of a χ² distribution is twice its degrees of freedom.

• Moment Generating Function (m.g.f.):
  

  M_X(t) = (1 - 2t)^-n/2 for t < 1/2.

  As derived earlier, the m.g.f. is a crucial tool for understanding and deriving properties of the distribution.

• Cumulant Generating Function (c.g.f.):
  The c.g.f. K_X(t) is the natural logarithm of the m.g.f.
  

  K_X(t) = ln(M_X(t)) = ln((1 - 2t)^-n/2) = -(n/2) ln(1 - 2t) for t < 1/2.

  Cumulants, derived from the c.g.f., are useful for describing the shape of a distribution. The first cumulant is the mean, the second is the variance, the third is related to skewness, and the fourth to kurtosis.

• Mode:
  

  The mode of the χ² distribution depends on the degrees of freedom (n):

  • If n > 2, the mode is at n - 2.
  • If n = 2, the p.d.f. is maximized at x = 0 (the mode is 0).
  • If n = 1, the p.d.f. approaches infinity as x approaches 0, so there is no unique mode in the conventional sense, or some consider 0 as the mode. However, often only n > 2 is considered for a 'peak'.
• Additive Property:
  

  If X₁ ~ χ²(n₁) and X₂ ~ χ²(n₂) are independent Chi-Square random variables, then their sum X₁ + X₂ follows a Chi-Square distribution with (n₁ + n₂) degrees of freedom.
  X₁ + X₂ ~ χ²(n₁ + n₂)

  This property is very important in applications where sums of independent squared normal variates are encountered.

Sum and Quotient Theorems

Understanding sums and quotients of random variables is fundamental in probability theory, as these operations often lead to new, important distributions. The syllabus mentions examples like X, X+Y, Y/X, and X/(X+Y).

Sum of Random Variables (e.g., X + Y)

When two independent random variables, X and Y, are added, the distribution of their sum Z = X + Y can often be found using convolution or by combining their moment generating functions (if they exist).

• For Chi-Square variables, if X ~ χ²(n₁) and Y ~ χ²(n₂) are independent, then Z = X + Y ~ χ²(n₁ + n₂). This is precisely the additive property discussed above, highlighting its significance.

Quotient of Random Variables (e.g., Y / X and X / (X + Y))

When two independent random variables, X and Y, are divided or combined in a ratio, this also leads to new distributions. These are often more complex than sums but are crucial in many statistical tests.

• If X ~ χ²(n₁) and Y ~ χ²(n₂) are independent, then the ratio (X/n₁) / (Y/n₂) follows an F-distribution with (n₁, n₂) degrees of freedom. This is a fundamental distribution for comparing variances.
• If X ~ χ²(n₁) and Y ~ χ²(n₂) are independent, then the ratio X / (X + Y) follows a Beta distribution of the first kind with parameters α = n₁/2 and β = n₂/2.

These examples illustrate how specific combinations of χ² variables give rise to other common sampling distributions used in hypothesis testing.

Limiting Form of χ² Distribution

As the degrees of freedom (n) increase, the shape of the χ² distribution changes. For large values of n, the χ² distribution approaches a normal distribution.

Specifically, if X ~ χ²(n), then for large n, X is approximately normally distributed with mean n and variance 2n.

X ≈ N(n, 2n)

A better approximation involves standardizing the variable. For large n, the expression √(2X) - √(2n - 1) approaches a standard normal distribution, N(0, 1).

This limiting property is important because it allows us to use normal distribution approximations for χ² tests when the degrees of freedom are sufficiently large, simplifying calculations in some cases.

Applications of the Chi-Square (χ²) Distribution

Tests of Significance Based on χ² Distribution

The χ² distribution is widely used in hypothesis testing, particularly for analyzing categorical data.

• Chi-Square Test for Goodness of Fit:
  

  This test assesses whether an observed frequency distribution significantly differs from an expected frequency distribution. It is used to test if a sample comes from a population with a specific hypothesized distribution (e.g., uniform, normal, Poisson).

  Hypothesis:
  H₀: The observed frequencies fit the expected distribution.
  H₁: The observed frequencies do not fit the expected distribution.

  Test Statistic:
  χ² = ∑ [(O_i - E_i)² / E_i]

  Where O_i is the observed frequency in category i, and E_i is the expected frequency in category i. The degrees of freedom are k - 1 - m, where k is the number of categories and m is the number of parameters estimated from the sample to calculate E_i.

  Example: Testing if a die is fair by comparing observed roll frequencies to expected uniform frequencies.

• Chi-Square Test for Independence (Association):
  

  This test determines if there is a statistically significant association between two categorical variables in a contingency table. It tests whether the two variables are independent of each other.

  Hypothesis:
  H₀: The two categorical variables are independent.
  H₁: The two categorical variables are dependent (associated).

  Test Statistic:
  χ² = ∑ ∑ [(O_ij - E_ij)² / E_ij]

  Where O_ij is the observed frequency in cell (i, j) of the contingency table, and E_ij is the expected frequency in cell (i, j) assuming independence. The degrees of freedom are (r - 1)(c - 1), where r is the number of rows and c is the number of columns.

  Example: Testing if there is an association between gender and preferred type of beverage.

• Chi-Square Test for Homogeneity:
  

  This test is similar to the test for independence but is used when comparing the distribution of a single categorical variable across two or more independent populations. It tests if the proportions of categories are the same across different populations.

  Hypothesis:
  H₀: The distributions of the categorical variable are homogeneous across the populations.
  H₁: The distributions are not homogeneous.

  Test Statistic:
  The test statistic and degrees of freedom are calculated in the same way as the test for independence: χ² = ∑ ∑ [(O_ij - E_ij)² / E_ij] with d.f. = (r - 1)(c - 1).

  Example: Comparing the proportion of voters favoring a candidate across different regions.

Confidence Intervals Based on χ² Distribution

The χ² distribution is also used to construct confidence intervals for the population variance (σ²) of a normally distributed population.

If we have a random sample of size n from a normal population with variance σ², then the sample variance S² is related to the χ² distribution through the statistic:

((n - 1) S²) / σ² ~ χ²(n - 1)

A (1 - α)100% confidence interval for σ² is given by:

[ ((n - 1) S²) / χ²_{α/2, n-1} , ((n - 1) S²) / χ²_{1-α/2, n-1} ]

Where χ²_{α/2, n-1} and χ²_{1-α/2, n-1} are the critical values from the χ² distribution with (n - 1) degrees of freedom, corresponding to the lower and upper tails of the distribution, respectively.

This confidence interval provides a range of plausible values for the true population variance based on the observed sample variance. It is particularly useful in quality control and process capability studies where the variability of a process is a key concern.