Unit 5: Statistical Estimation and Hypothesis Testing: Comprehensive Study Guide

1. Introduction to Statistical Inference

Statistical inference is the process of using data analysis to deduce properties of an underlying probability distribution. It allows researchers to draw conclusions about a large population based on a smaller, representative sample. This process is divided into two primary areas: Estimation and Hypothesis Testing.

Key Terms: Parameter, Statistic, Estimator, and Estimate

To understand statistical inference, we must first distinguish between populations and samples, and the values that describe them.

• Parameter: A numerical characteristic of an entire population. It is a fixed, constant value, but it is typically unknown because measuring an entire population is often impossible or impractical. Examples include the population mean (μ) and the population standard deviation (σ).
• Statistic: A numerical characteristic of a sample. It is calculated directly from the observed sample data and is a variable because its value changes from one sample to another. Examples include the sample mean (X̄) and the sample standard deviation (s).
• Estimator: A rule, formula, or function used to estimate an unknown population parameter based on sample data. For example, the sample mean formula is an estimator for the population mean.
• Estimate: The specific numerical value obtained by applying an estimator to a concrete set of sample data. For instance, if a specific sample yields a mean of 15.4, then 15.4 is the estimate.

Sampling Distribution of a Statistic

If we take multiple random samples of the same size from a population and calculate the same statistic (e.g., the sample mean) for each sample, we will get different values. The probability distribution of these sample statistics is called the Sampling Distribution.

The Sampling Distribution of a statistic is the theoretical probability distribution of all possible values that the statistic can take when computed from random samples of a specified size (n) drawn from a given population.

For example, if we repeatedly sample 50 students from a university and calculate their average height, the collection of all those sample means forms the Sampling Distribution of the Sample Mean. According to the Central Limit Theorem, if the sample size is sufficiently large (typically n ≥ 30), the sampling distribution of the sample mean will be approximately normally distributed, regardless of the shape of the population distribution.

Concept of Bias and Standard Error

When using sample statistics to draw conclusions about population parameters, we must evaluate the quality of our estimators using two key concepts: bias and standard error.

Bias

Bias is the difference between the expected value of an estimator and the true value of the population parameter it is trying to estimate.

Bias = E(θ̂) - θ

Where θ̂ is the estimator and θ is the true population parameter.

• An estimator is unbiased if its expected value equals the true parameter value (E(θ̂) = θ). This means that on average, the estimator hits the target. For example, the sample mean (X̄) is an unbiased estimator of the population mean (μ).
• An estimator is biased if its expected value systematically overestimates or underestimates the parameter (E(θ̂) ≠ θ). For example, if we calculate sample variance by dividing by n instead of (n - 1), we get a biased estimator that tends to underestimate the population variance.

Standard Error

The standard deviation of a sampling distribution is called the Standard Error (SE). It measures the precision of the estimator by showing how much the statistic varies from sample to sample. A smaller standard error indicates that sample statistics are clustered closely around the true parameter, representing high precision.

Standard Error of the Mean (SE):
SE = σ / √n

Where σ is the population standard deviation and n is the sample size. If the population standard deviation is unknown, we estimate it using the sample standard deviation (s):

Estimated Standard Error (SE):
SE ≈ s / √n

Observation: As the sample size (n) increases, the standard error decreases. This means larger samples provide more precise estimates.

2. Statistical Estimation

Estimation is the process of using sample data to make a reasonable guess about an unknown population parameter. There are two primary types of estimation: Point Estimation and Interval Estimation.

Point Estimation

A point estimation provides a single numerical value as the best guess for the unknown population parameter.

• Example: Stating that the average height of all university students is exactly 170 cm based on a sample mean of 170 cm.
• Advantage: Simple, straightforward, and easy to communicate.
• Disadvantage: It does not give any information about the precision or reliability of the estimate. The probability that a point estimate is exactly equal to the true population parameter is virtually zero for continuous variables.

Interval Estimation

An interval estimation specifies a range of values within which the population parameter is expected to lie with a certain level of confidence. This range is known as a Confidence Interval (CI).

Confidence Interval Formula:
Point Estimate ± (Critical Value × Standard Error)

For estimating a population mean μ with a known standard deviation σ, the formula is:

CI = X̄ ± Z_α/2 × (σ / √n)

Where Z_α/2 is the critical value from the standard normal distribution corresponding to the desired confidence level (1 - α).

• Confidence Level (1 - α): The probability that the interval estimation procedure will capture the true population parameter over repeated sampling. Common confidence levels are 95% (Z = 1.96) and 99% (Z = 2.576).
• Margin of Error: The term subtracted from and added to the point estimate (Critical Value × Standard Error). It defines the width of the confidence interval.
• Interpretation: A 95% confidence interval means that if we take 100 random samples and construct an interval from each, approximately 95 of those intervals will contain the true population parameter. It does not mean there is a 95% probability that the parameter lies in that specific interval once calculated.

3. Hypothesis Testing

Hypothesis testing is a formal statistical procedure used to make decisions about a population based on sample data. It determines whether there is enough evidence to support a specific claim.

Null and Alternative Hypotheses

Every hypothesis test involves formulating two opposing hypotheses:

• Null Hypothesis (H₀): This represents the status quo, no effect, or no difference. It is assumed to be true unless there is strong evidence against it. We always test the null hypothesis directly.
• Alternative Hypothesis (H₁ or H_a): This represents the claim we want to prove or gather evidence for. It asserts that there is an effect, a change, or a difference.

For example, if testing whether a new drug reduces blood pressure:

• H₀: μ_new = μ_old (The new drug has no different effect.)
• H₁: μ_new < μ_old (The new drug reduces blood pressure.)

Type I and Type II Errors

Because we make decisions about a whole population using only a sample, there is always a risk of making an incorrect decision. These errors are classified into two types:

• Type I Error (False Positive): Occurs when we reject a true null hypothesis (H₀). The probability of committing a Type I error is denoted by alpha (α), which is also known as the significance level of the test.
• Type II Error (False Negative): Occurs when we fail to reject a false null hypothesis (H₀). The probability of committing a Type II error is denoted by beta (β). The power of a test is defined as (1 - β), representing the probability of correctly rejecting a false null hypothesis.

Important Note: There is an inverse relationship between Type I and Type II errors. If you decrease the significance level (α) to reduce the risk of a Type I error, you automatically increase the probability of a Type II error (β), assuming the sample size remains constant. The only way to decrease both error probabilities simultaneously is to increase the sample size (n).

One-Tailed vs. Two-Tailed Hypothesis Testing

The directionality of the alternative hypothesis determines whether a test is one-tailed or two-tailed.

• One-Tailed Test: Used when the alternative hypothesis specifies a direction of change (either an increase or a decrease). The rejection region is entirely in one tail of the sampling distribution.
  • Left-tailed: H₁: μ < μ₀
  • Right-tailed: H₁: μ > μ₀
• Two-Tailed Test: Used when the alternative hypothesis does not specify a direction, but merely states that the parameter is different from the null value. The rejection region is split equally between the two tails of the sampling distribution.
  • Formula: H₁: μ ≠ μ₀ (which means μ < μ₀ or μ > μ₀)

4. Hypothesis Testing Using Z and t Tests

To perform hypothesis tests about a population mean, statisticians typically use either a Z-test or a t-test. Choosing between them depends on whether the population variance is known and the size of the sample.

The Z-Test

The Z-test is used when the population variance (σ²) is known, or when the sample size is large (n ≥ 30) and the sample standard deviation is used to estimate the population standard deviation.

Z-Test Statistic Formula:
Z = (X̄ - μ₀) / (σ / √n)

Where:

• X̄ is the sample mean.
• μ₀ is the hypothesized population mean under H₀.
• σ is the population standard deviation.
• n is the sample size.

Assumptions for a Z-Test:

1. The data must be obtained through random sampling.
2. The population standard deviation (σ) is known, OR the sample size (n) is large (n ≥ 30) so that the sample standard deviation (s) serves as a reliable approximation for σ.
3. The underlying distribution is normal, or the sample size is large enough to rely on the Central Limit Theorem.

The t-Test

The t-test (specifically Student's t-test) is used when the population standard deviation (σ) is unknown and the sample size is small (n < 30).

t-Test Statistic Formula:
t = (X̄ - μ₀) / (s / √n)

Where:

• s is the sample standard deviation.
• The test statistic follows a t-distribution with degrees of freedom (df) = n - 1.

Assumptions for a t-Test:

1. The population standard deviation (σ) is unknown.
2. The sample is randomly selected.
3. The sample size is small (n < 30), requiring the population to be approximately normally distributed.

Comparing Z-Distribution and t-Distribution:

• The t-distribution is symmetric and bell-shaped, much like the standard normal Z-distribution, but it has heavier tails. This means it allows for more variability, reflecting the extra uncertainty introduced by estimating the unknown σ with s.
• As the degrees of freedom (df) increase (i.e., as the sample size increases), the t-distribution approaches the standard normal distribution. For n ≥ 30, the differences between t and Z values become negligible.

Summary of Steps in Hypothesis Testing

1. State the null hypothesis (H₀) and alternative hypothesis (H₁).
2. Choose the appropriate significance level (α), commonly 0.05, 0.01, or 0.10.
3. Identify the test statistic (Z or t) depending on whether σ is known and the sample size.
4. Determine the decision rule (find critical values or calculate the p-value).
5. Calculate the test statistic value from the sample data.
6. Make a statistical decision: Reject H₀ if the calculated statistic falls in the rejection region (or if p-value ≤ α); otherwise, fail to reject H₀.
7. State the final conclusion in the context of the real-world problem.