PSYC3001

Summary of Course Content

 What have we learnt?
1. The two‐group randomised experiment. Review of statistical inference on a comparison between two means: hypothesis tests and confidence intervals. Levels of inference: confidence
interval inference, directional inference, inequality inference. Inferential errors ‐ Type I, Type II and Type III errors, non‐coverage errors. Practical equivalence inference.
2. The problem of multiple comparisons with more than two groups. Monte Carlo sampling experiments. Logical and statistical dependence among comparisons. Error rate units. Per‐
comparison error rates and familywise error rates for individual t‐test and CI procedures when J > 2.
3. Controlling the familywise error rate for test of the maximal comparison. The Tukey (Honestly Significant Difference) multiple comparison procedure (MCP) based on the range of means.
Properties of the Tukey simultaneous test procedure (STP) and simultaneous confidence interval procedure (SCI).
4. Single‐factor fixed‐effects ANOVA model. Effect parameters, effect size and levels of inference. Partition of variation and degrees of freedom. The standard ANOVA‐model analysis.
Assumptions. Central and non‐central F distribution. Heterogeneity inference.
5. Contrasts on effect parameters and means. Simple and complex contrasts. Contrast statistics. The sampling distribution of the sample value of a single planned contrast. CI and directional
inference on a single planned contrast – unstandardised and standardised effect size. Scale of contrast coefficients.
6. Controlling the familywise error rate with the F STP. The maximal contrast. The Scheffé SCI procedure. Coherence and consonance. Carrying out an F‐based analysis with PSY.
Unstandardised and standardised CIs.
7. Planned vs post hoc analyses. Alternatives to the F STP for planned contrast analyses. The Bonferroni‐t procedure. Using PSY to carry out Bonferroni t analyses.
8. Coherent vs incoherent MCPs. Comparison of simultaneous MCPs that control FWER ‐ Scheffé, Bonferroni and Tukey procedures. Examples of sequential MCPs that do not control
FWER ‐ ‘protected’ t‐test procedures.
9. Orthogonal contrasts. Properties. Controlling the per‐contrast error rate (PCER) in analyses of planned orthogonal contrasts.
10. Trend contrasts – ANOVA model analysis of single factor experiments with a quantitative IV. Inference on planned linear and quadratic trend contrasts controlling PCER.
11. The 2 x 2 factorial design. Parameters of two‐factor ANOVA model. Sources of variation. Factorial effect contrasts. The cell means model.
12. Analysis of J × K factorial between‐subjects designs. Heterogeneity inference. F STPs for main effect and interaction contrasts. Scheffé SCIs.
13. Bonferroni t procedures for analyses based on planned main effect and interaction contrasts for between‐subjects factorial designs.
14. Planned and post hoc coherent analyses of J × K factorial designs allowing for inferences on simple effects. The A simple‐effects model and the A(B) family of contrasts. The all‐factorial‐
contrasts family.
15. Within subjects designs. Planned analyses of within subjects contrasts. The multivariate model vs univariate model for single‐factor within‐subjects designs. Assumptions.
16. Two‐factor mixed designs (one between subjects factor, one within subjects factor). Planned analyses of main and interaction contrasts, based on the two‐factor model. The MANOVA
(multivariate ANOVA) vs univariate (ANOVA) model for mixed factorial designs. Planned analyses of B × (W) factorial designs allowing for inferences on simple effect contrasts.
17. Two‐factor within‐Ss designs. Planned analyses of main and interaction contrasts based on two‐factor MANOVA model. Planned analyses allowing for inferences on simple effect contrasts
What have we really learnt?

How to make confident statistical inferences

• Test outcomes
• Confidence intervals

For two types of designs

• Single Factor
• Factorial
 1. Single Factor Designs

Between Within

Tukey Post-hoc Planned Planned

(Scheffé) (Bonferroni) (Bonferroni)
 1. Single Factor Designs

Between Within

Tukey Post-hoc Planned Planned

(Scheffé) (Bonferroni) (Bonferroni)
 Single Factor Between
Tukey Scheffé Bonferroni
Valid for? Pairwise comparisons Post-hoc contrast Planned contrast analysis
analysis (and planned)

Power/flexibility Best power/precision Best flexibility More powerful than Sch when
for all pairwise k ≤ J-1
comparisons
May be more powerful
for planned than Bonf
if k > J-1
 ν𝟐𝟐 = J(n-1)

Single Factor Between

k = # of contrasts
J = # of groups
α = .05
ν1 = J-1

Tukey Scheffé Bonferroni

Tests 1. Overall test of max 1. Overall test – one
comparison way ANOVA No overall test!

2. If sig, test remaining 2. If sig, follow up 1. Bonferroni contrasts

comparisons Scheffé contrasts

Confidence
𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎
intervals
𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎
 1. Single Factor Designs

Between Within

Tukey Post-hoc Planned Planned

(Scheffé) (Bonferroni) (Bonferroni)
 k = # of contrasts
Single Factor Within α = .05
ν𝟐𝟐 = N-1

Tukey Scheffé Bonferroni

Tests
No overall test!

1. Bonferroni contrasts

Confidence
𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎
intervals
𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎
 2. Factorial Designs

AxB B x (W) (A x B)
Between Between x Within Within

Post-hoc Planned Planned Planned

 2. Factorial Designs

AxB B x (W) (A x B)
Between Between x Within Within

Post-hoc Planned Planned Planned

 AxB
Post-hoc Planned
Scheffé Bonferroni
Standard Factorial 1. Two-Way ANOVA F – tests of
Analysis homoegenity hypotheses for A, B,
(A, B, AB) AB families

A(B) Family 1. Overall test for A(B) family or B(A)

A, A(b), AB family
or B(A) Family
B, B(a), AB 1. If Sig, follow up Scheffé contrasts

All factorial 1. Overall test for all factorial family

contrasts
A, B, A(b), B(a), AB 2. If Sig, follow up Scheffé contrasts
 AxB ν𝟐𝟐 = JK(n-1)

Post-hoc
Scheffé
Standard Factorial 1. Two-Way ANOVA F – tests of
Analysis homogeneity hypotheses for A, B, AB α* = α = .05
(A, B, AB) families ν1 = J-1 or K-1
or (J-1)(K-1)
2. If Sig, Scheffé contrasts – 3 separate
families 1.

A(B) Family 1. Overall test for A(B) family or B(A) 2. Fc =

𝑚𝑚
A, A(b), AB family (usually not run) α* = 1-(1−α)
or B(A) Family ν1 = K(J-1) or J(K-1)
B, B(a), AB 2. If Sig, follow up Scheffé contrasts

All factorial 1. Overall test for all factorial family α* = 1-(1−α)𝑚𝑚

contrasts (usually not run) ν1 = JK-1
A, B, A(b), B(a), AB
2. If Sig, follow up Scheffé contrasts
 AxB ν𝟐𝟐 = JK(n-1)

Planned
Bonferroni
Standard Factorial 1. Two-Way ANOVA F – tests of No overall tests

Analysis homoegenity hypotheses for A, B, AB

(A, B, AB) families 1. Planned Bonferroni contrasts – 3
α* = α = .05 separate families
1. Fc =
2. If Sig,Scheffé contrasts – 3 separate
families

A(B) Family 1. Overall test for A(B) family or B(A) No overall tests

A, A(b), AB family
or B(A) Family α* = mα 1. Planned Bonferroni contrasts – one family
k = # contrasts
B, B(a), AB 1. If Sig, follow up Scheffé contrasts

All factorial 1. Overall test for all factorial family No overall tests

contrasts α* = mα
A, B, A(b), B(a), AB 2. If Sig, follow up Scheffé contrasts 1. Planned Bonferroni contrasts – one family
 2. Factorial Designs

AxB B x (W) (A x B)
Between Between x Within Within

Post-hoc Planned Planned Planned

 B x (W) ν𝟐𝟐 = J(n-1)

Planned
Bonferroni
Standard Factorial No overall tests

Analysis
(A, B, AB) 1. Planned Bonferroni contrasts – 3
α* = α =
1. Fc = separate families
.05

A(B) Family No overall tests

A, A(b), AB α* = mα
or B(A) Family 1. Planned Bonferroni contrasts – one family
k = # contrasts
B, B(a), AB
All factorial No overall tests

contrasts α* = mα
A, B, A(b), B(a), AB 1. Planned Bonferroni contrasts – one family
 2. Factorial Designs

AxB B x (W) (A x B)
Between Between x Within Within

Post-hoc Planned Planned Planned

 (A x B) ν𝟐𝟐 = N-1

Planned
Bonferroni
Standard Factorial No overall tests

Analysis
(A, B, AB) 1. Planned Bonferroni contrasts – 3
α* = α =
1. Fc = separate families
.05

A(B) Family No overall tests

A, A(b), AB α* = mα
or B(A) Family 1. Planned Bonferroni contrasts – one family
k = # contrasts
B, B(a), AB
All factorial No overall tests

contrasts α* = mα
A, B, A(b), B(a), AB 1. Planned Bonferroni contrasts – one family
 And the best part of all of this….
You don’t need to remember it all
The table (final exam formula sheet)
Inference activity
Week 9/10 tutorial example
 Pre Post FU Mj
25

Tmt 1 10 20 12 14
20
Tmt2 12 18 18 16
15
Control 11 13 12 12
10
Mk 11 17 14 14
tmt1 tmt2 control
5

0
pre post FU
Significant contrasts: B1, W1, W2, B1W1, B2W2
 Pre Post FU Mj

Tmt 1 10 20 12 14

Tmt2 12 18 18 16

Control 11 13 12 12

Mk 11 17 14 14

B1: Participants who were given treatment 1 or treatment 2 had improved more
than those who were in the control group.
Referring to participants Missing DV
Not stating subsets clearly Not averaging across other factor
“improved” Not referring to average DV

(For the population of people seeking treatment for depression:)

B1: Average wellbeing scores are higher for treatments (tmt1 and tmt2
averaged) compared to control, averaged across occasions of measurement.
 Pre Post FU Mj

Tmt 1 10 20 12 14

Tmt2 12 18 18 16

Control 11 13 12 12

Mk 11 17 14 14

W1: Averaging across occasions of measurement, wellbeing scores are higher at

pre than at post.
Averaging across same factor
Wrong direction
Not stating average DV

(For the population of people seeking treatment for depression:)

W1: Averaging across type of treatment received, average wellbeing scores are
higher at post than at pre.
 Pre Post FU Mj

Tmt 1 10 20 12 14

Tmt2 12 18 18 16

Control 11 13 12 12
Assume: 3 DV units is the smallest difference of clinical importance.
Mk 11 17 14 14

W2: Averaging across types of treatment, scores are lower at post than follow
up by between -4.64 and -1.36 units and this is clinically important.
Wrong direction Negative scores not possible on DV
Missing DV Incorrect inference about importance
Not stating average DV

(For the population of people seeking treatment for depression:)

W2: Averaging across type of treatment, average wellbeing scores are higher at
post than at follow up by at least 1.36 and at most 4.64 points on the wellbeing
scale. This effect may or may not be of clinical importance.
 Pre Post FU Mj

Tmt 1 10 20 12 14

Tmt2 12 18 18 16

Control 11 13 12 12

Mk 11 17 14 14

B1W1: The size of the B1 effect (treatments compared to control) is greater at

post.
Not looking at diff. between 2 simple effects One subset in W1 not specified
Direction of B1 effect not specified Missing average DV

B1W1: The size of the B1 effect (higher average wellbeing scores with
treatments compared to control) is greater at post test than at pre test.

B1W1: The magnitude of the increase in average wellbeing scores from pre to
post (W1) is greater for treatments than for control.
 Pre Post FU Mj

Tmt 1 10 20 12 14

Tmt2 12 18 18 16

Control 11 13 12 12

Mk 11 17 14 14

B2W2: The magnitude of the decrease in wellbeing scores from post to follow
up (W2) is greater for treatment 1 than treatment 2. At post, average wellbeing
is higher for T1 than T2, whereas at follow up, average wellbeing is higher for T2
than T1.

B2W2: The magnitude

Implying that simpleofeffects
the decrease in wellbeing
have been tested scores from post to follow
up (W2) is greater for treatment 1 than treatment 2. For the sample, at post,
average wellbeing is higher for T1 than T2, whereas at follow up, average
wellbeing is higher for T2 than T1.
Inferences – Main effects
DO:
Clearly identify your subsets: higher/lower in X compared to Y
Refer to your DV
Refer to average DV scores
Include “averaging across other factor”

DO NOT:
Refer to sample (e.g., “participants who got drug 1”)
Refer to a generic DV (e.g., “by at least 2 and at most 4 units)
Talk about improvement/increase when it has not been measured
Inferences – Interactions
DO:
Express as the difference between two simple effects

DO NOT:
Imply simple effects have been tested when they haven’t
Refer to only 1 cell (e.g., DV scores are highest with Drug X and
Tmt1 are combined )
Comparing analyses and
theoretical issues
Planned vs Post-hoc Analyses
• Planned analysis: more power/precision
• Smaller Fc/CC

• Post-hoc analysis: more flexibility

• Similar logic applies to comparing analyses controlling PCER

vs FWER

• Example: Week 4 tutorial Ex3, Ex4; Final exam practice Q1

Standard vs Nonstandard Factorial Analyses
• Standard analysis: more power/precision

• Nonstandard analysis: interpretive gain

• Example: Weeks 8 and 10 tutorials

Multivariate vs Univariate
• Both are possible approaches to analyses of data from within-
subject designs

• Multiple vs single error term for contrasts

• Sphericity (assumption of univariate approach) is typically

violated – under/over-estimation of error terms

• Example: Topic 15, 16 and 17 lectures

Between vs Within Designs
• PSY output for a within-subjects design can tell us that it is
more powerful compared to a between-subjects design.
• Look at SS(Between) vs SSE for within contrasts.

• Example: Week 10 Tutorial