Errors and Post-hoc Tests

Online lesson from biological data analysis modules

What will you learn?

In this lesson you will learn to:

  1. Explain the types of errors inherent to hypothesis testing
  2. Describe the problem of multiple comparisons
  3. Explain why post-hoc tests are necessary

A Classic Scam

(Video 1 min 23 sec)

A Classic Scam Revealed!

(Video 1 min)

Multiple Testing Scam

(Video 55 sec)

Multiple Testing Scam Revealed?

(Video 1 min 31 sec)

Making the wrong decision

(Video 3 mins 30 sec)

Post-hoc Testing

(Video 6 mins 2 sec)

Key Points (Errors)

  • Hypothesis tests have two unavoidable errors
    1. False negatives (Type II errors)
    2. False positives (Type I errors)
  • These two errors cannot be avoided, but their rates can be controlled
  • The rate at which false positives occur is the critical significance level (commonly 5%)
  • It is important that the false positive error rate is known a priori

Key Points (Multiple Comparisons)

  • Performing multiple tests increases the overall rate of false positives
  • Critical significance thresholds must be modified when performing multiple tests to keep overall false positive rates close to 5%
  • Post-hoc tests modify significance thresholds to keep the error rate of false positives close to 5%
  • Bonferroni correction and Tukey's HSD are two common post-hoc methodologies