Unit 4: Correlation and Regression Analysis Study Guide

1. Introduction to Correlation

Correlation analysis is a statistical method used to evaluate the strength and direction of the relationship between two quantitative variables. It helps in understanding how changes in one variable are associated with changes in another variable.

Correlation between Two Variables

When two variables vary together, they are said to be correlated. For instance, an increase in household income is often accompanied by an increase in consumer spending. There are two primary classifications of this relationship:

• Positive Correlation: When both variables move in the same direction (e.g., as one increases, the other also increases).
• Negative Correlation: When the variables move in opposite directions (e.g., as one increases, the other decreases).

Degrees of Correlation

The degree of correlation refers to the intensity or strength of the relationship between the two variables. This degree is measured by a correlation coefficient, which ranges strictly between -1 and +1.

2. Scatter Diagrams

A scatter diagram (or scatter plot) is a visual representation of bivariate data. It is constructed by plotting values of the independent variable on the horizontal X-axis and the dependent variable on the vertical Y-axis. The resulting pattern of points reveals the nature of the correlation.

• Perfect Positive: Points lie exactly on a straight line rising from the lower-left corner to the upper-right corner.
• Perfect Negative: Points lie exactly on a straight line falling from the upper-left corner to the lower-right corner.
• Strong Positive / Negative: Points are highly concentrated around an imaginary upward or downward sloping line.
• Weak Positive / Negative: Points are widely dispersed but show a general upward or downward trend.
• No Correlation: Points are scattered randomly across the plane in a circular or chaotic manner, showing no linear pattern.

3. Pearson’s Simple Correlation Coefficient

Pearson’s Product-Moment Correlation Coefficient (denoted by r) is a parametric measure of the linear association between two continuous variables. It assumes that the data is normally distributed and a linear relationship exists between the variables.

Formula

The simple correlation coefficient is calculated using the following algebraic formula:

r = [ N ∑XY - (∑X)(∑Y) ] / √[ {N ∑X² - (∑X)²} × {N ∑Y² - (∑Y)²} ]

Where:

• N: The total number of paired observations.
• X, Y: The individual values of the two variables.
• ∑: The summation symbol representing the sum over all observations.

An alternative representation using covariance and standard deviations (σ) is:

r = Cov(X, Y) / (σ_x × σ_y)

4. Spearman’s Rank Correlation Coefficient

When the data is qualitative (such as ranking candidates in a beauty contest, evaluating intelligence, or honesty) or when the normality assumption of Pearson's coefficient is violated, Spearman’s Rank Correlation Coefficient (denoted by R or ρ) is used. It is a non-parametric measure based on ranks rather than exact values.

Formula for Untied Ranks

When no two ranks are identical, the formula is:

R = 1 - [ (6 ∑d²) / (n(n² - 1)) ]

Where:

• d: The difference between the ranks assigned to the two variables for each individual (d = R₁ - R₂).
• n: The total number of pairs of observations.

Formula for Tied Ranks

When two or more observations have the same value, they are assigned the average of the ranks they would have otherwise occupied. A correction factor must be added for each set of tied values:

R = 1 - [ 6 { ∑d² + ∑(m³ - m)/12 } / (n(n² - 1)) ]

Where m is the number of observations tied for a particular rank.

5. Two-Variable Linear Regression

While correlation measures the strength of association, regression analysis models the functional relationship between the variables, allowing one to predict the value of a dependent variable based on an independent variable.

Regression Lines

In a simple two-variable system, there are two regression lines because either variable can be modeled as dependent on the other:

1. Regression Line of Y on X: Used to estimate Y for given values of X. The line is represented as:

   Y - Ȳ = b_yx(X - X̄)

2. Regression Line of X on Y: Used to estimate X for given values of Y. The line is represented as:

   X - X̄ = b_xy(Y - Ȳ)

Where X̄ and Ȳ represent the arithmetic means of the respective variables.

Regression Coefficients

The coefficients b_yx and b_xy represent the slopes of the regression lines. They measure the change in the dependent variable per unit change in the independent variable.

• Regression Coefficient of Y on X (b_yx):

  b_yx = r × (σ_y / σ_x) = [ N ∑XY - (∑X)(∑Y) ] / [ N ∑X² - (∑X)² ]

• Regression Coefficient of X on Y (b_xy):

  b_xy = r × (σ_x / σ_y) = [ N ∑XY - (∑X)(∑Y) ] / [ N ∑Y² - (∑Y)² ]

6. Properties of Correlation and Regression Coefficients

Understanding the properties of correlation and regression coefficients is critical for computational verification and statistical interpretation.

• Property 1: Range of Correlation Coefficient: The value of the correlation coefficient r must lie between -1 and +1.

  -1 ≤ r ≤ +1

• Property 2: Relationship with Regression Coefficients: The correlation coefficient is the geometric mean of the two regression coefficients.

  r = ±√(b_yx × b_xy)

  The sign of r is determined by the signs of the regression coefficients.
• Property 3: Sign Consistency: Both regression coefficients (b_yx and b_xy) and the correlation coefficient (r) must always share the same algebraic sign. If one is negative, all must be negative.
• Property 4: Mutual Bounds: If one regression coefficient is greater than 1 in absolute value, the other must be less than 1 in absolute value.

  If |b_yx| > 1, then |b_xy| < 1

• Property 5: Arithmetic Mean Comparison: The arithmetic mean of the two regression coefficients is greater than or equal to the correlation coefficient.

  (b_yx + b_xy) / 2 ≥ r

• Property 6: Change of Origin and Scale:
  • The correlation coefficient (r) is independent of both change of origin and change of scale.
  • The regression coefficients (b_yx and b_xy) are independent of change of origin, but they are dependent on change of scale.

Summary Comparison