CHOOSING THE CORRECT STATISTICAL TEST

Contents

1. Introduction 2

2. Independent and Dependent Variables 2

3. Samples/Groups 5

4. Assumptions 8

5. Review of Test 10

6. Summary 12

(source: http://www.wadsworth.com/psychology_d/templates/student_resources/workshops/stats_wrk.html)
Introduction

This tutorial is designed to help you develop a framework for choosing the correct statistic to test
your hypothesis. The tutorial begins with a set of questions you should ask when selecting your
test. It is followed by demonstrations of the factors that are important to consider when
choosing your statistic. Once you understand how to choose among statistics, you should test your
knowledge by doing the practice exercises.

This tutorial assumes that you know all of the basic univariate statistics. The tutorial is not designed
to teach you the formulas and procedures for conducting these tests. Rather, it focuses on how to
select the appropriate statistic to test a particular hypothesis.

Why This Is Important? - Now that you know all of the basic univariate statistical tests, it can be
difficult to figure out which one you need to use for your research design. This workshop will give
you a broad overview of the different statistics available to help you choose which one is best for
your study.

Presented below are four questions you should ask and answer when trying to determine which
statistical procedure is most appropriate to test your hypothesis.

• What are the independent and dependent variables?

• What is the scale of measurement of the study variables?
• How many samples/groups are in the design?
• Have I met the assumptions of the statistical test selected?

To determine which test should be used in any given circumstance, we need to consider the
hypothesis that is being tested, the independent and dependent variables and their scale of
measurement, the study design, and the assumptions of the test.

Independent and Dependent Variables

Before we can begin to choose our statistical test, we must determine which is the independent and
which is the dependent variable in our hypothesis. Our dependent variable is always the
phenomenon or behavior that we want to explain or predict. The independent variable represents a
predictor or causal variable in the study. In any antecedent-consequent relationship, the
antecedent is the independent variable and the consequent is the dependent variable.
Study - It has been traditional for the man rather than the woman to receive the check when a
couple dines out. A researcher wondered whether this would be true if the woman was clearly in
charge, asking for the wine list, questioning the waiter about dishes on the menu, etc. A large
random sample of restaurants was selected. One couple was used in all restaurants, but in half the
man assumed the traditional in-charge role, and in the other half the woman was in charge. At each
restaurant, the couple recorded whether the check was presented to the man or to the woman.

Test the research hypothesis that the check will be presented to the person showing in-charge
behavior.

Analysis - The behavior that we are trying to explain is the presentation of the check. Did the
wait staff give the check to the man or the woman? This would be the dependent variable in the
study. The independent variable was manipulated as part of the experimental design. The
independent variable was who was in charge during the dinner.

Now that we have identified the independent and dependent variables, our next step is to determine
the scale of measurement.

Scale of Measurement
Once we have identified the independent and dependent variables, our next step in choosing a
statistical test is to identify the scale of measurement of the variables. All of the parametric tests
that we have learned to date require an interval or ratio scale of measurement for the dependent
variable. Many psychologists also apply parametric tests to variables with an approximately interval
scale of measurement. It is your decision whether to consider approximately interval or "scale"
scores as suitable for parametric tests of means. If you are working with a dependent variable that
has a nominal or ordinal scale of measurement, then you must choose a nonparametric statistic to
test your hypothesis.

The scale of measurement of the dependent variable helps us to choose the broad category of
statistical procedures appropriate for our hypothesis ( nonparametric vs. parametric ). The scale
of measurement of the independent variable helps us to determine which statistical procedure
within the broad category is appropriate.
Nonparametric statistics are used when our data are measured on a nominal or ordinal scale of
measurement. Chi-square statistics and their modifications (e.g., McNemar Test) are used for
nominal data. All other nonparametric statistics are appropriate when data are measured on an
ordinal scale of measurement. The nonparametric statistics that we learned in class are listed below.

In order to choose among these tests, we must next determine the number of samples in the study
design and the relationship between samples.

Parametric statistics are used when our data are measured on approximately interval, interval,
or ratio scales of measurement. No distinctions are made among these scales of measurement,
although all parametric statistics have assumptions that must be met before proceeding with
statistical analysis. The parametric statistics that we learned in class are listed below. This is not a
complete list -- we did not learn advanced statistical techniques.
In order to choose among these tests, we must next determine the number of samples in the study
design and the relationship between samples.

Once we have reviewed our options, the next step is to consider the study design. Let us now return
to our study example and determine the scale of measurement in our study.

Study - It has been traditional for the man rather than the woman to receive the check when a
couple dines out. A researcher wondered whether this would be true if the woman was clearly in
charge, asking for the wine list, questioning the waiter about dishes on the menu, etc. A large
random sample of restaurants was selected. One couple was used in all restaurants, but in half the
man assumed the traditional in-charge role, and in the other half the woman was in charge. At each
restaurant, the couple recorded whether the check was presented to the man or to the woman.

Test the research hypothesis that the check will be presented to the person showing in-charge
behavior.

Samples/Groups
Number of Samples - Once you have identified the scale of measurement of the dependent
variable, you want to determine how many samples or "groups" are in the study design. Designs
for which one-sample tests (e.g., Z test; t test; Pearson and Spearman correlations; chi-square
goodness-of-fit) are appropriate collect only one set or "sample" of data. There must be at least
two sets of scores or two "samples" for any statistic that examines differences between groups (e.g.
, t test for dependent means; t test for independent means; one-way ANOVA; Friedman
ANOVA; chi-square test of independence) .
One-Sample Tests - It is sometimes difficult to determine from a study description how many
samples of data were collected. Typically when the description refers to one type of person from a
larger population, the study design uses only one sample. An example would be if we wanted to
know whether Emory students are like college students in general. Emory students are a specific
sample from the larger population of college students. The specific test that we use now varies based
on whether we have collected data on only one dependent variable or on both independent and
dependent variables.

Dependent Variable - With single samples and one dependent variable, the one-sample Z test, the
one-sample t test, and the chi-square goodness-of-fit test are the only statistics that can be used.
Students sometimes ask, "but don't you have population data too, so you have two sets of data?"
Yes and no. Data have to exist or else the population parameters are defined. But, the researcher
does not collect these data, they already exist. So, if you are collecting data on one sample and
comparing those data to information that has already been gathered and is published, then you are
conducting a one-sample test using the one sample/set of data collected in this study. For the chi-
square goodness-of-fit test, you can also compare the sample against chance probabilities.

Independent and Dependent Variables - When we have a single sample and independent and
dependent variables measured on all subjects, we typically are testing a hypothesis about the
association between two variables. The statistics that we have learned to test hypotheses about
association include:
• chi-square test of independence

• Spearman's
• Pearson's
• bivariate regression and multiple regression

Multiple Sample Tests - Studies that refer to repeated measurements or pairs of subjects
typically collect at least two sets of scores. Studies that refer to specific subgroups in the
population also collect two or more samples of data. Once you have determined that the design
uses two or more samples or "groups", then you must determine how many samples or groups are in
the design. Studies that are limited to two groups use either the chi-square statistic, Mann-Whitney
U, Wilcoxon T, independent means t test, or the dependent means t test.

If you have three or more groups in the design, the chi-square statistic, Kruskal-Wallis H Test,
Friedman ANOVA for ranks, One-way Between-Groups ANOVA, Simple Repeated Measures
ANOVA, Factorial ANOVA, and Mixed Factorial ANOVA depending on the nature of the
relationship between groups. Some of these tests are designed for dependent or correlated
samples/groups and some are designed for samples/groups that are completely independent.

Dependent Means - Dependent groups refer to some type of association or link in the research
design between sets of scores. This usually occurs in one of three conditions -- repeated
measures, linked selection, or matching. Repeated measures designs collect data on subjects
using the same measure on at least two occasions. This often occurs before and after a treatment
or when the same research subjects are exposed to two different experimental conditions.

When subjects are selected into the study because of natural "links or associations", we want to
analyze the data together. This would occur in studies of parent-infant interaction, romantic
partners, siblings, or best friends. In a study of parents and their children, I would want my data to
be associated with my son's, not some other child's. Subject matching also produces dependent
data. Suppose that an investigator wanted to control for socioeconomic differences in research
subjects. She might measure socioeconomic status and then match on that variable. The scores on
the dependent variable would then be treated as a pair in the statistical test.

All statistical procedures for dependent or correlated groups treat the data as linked, therefore it is
very important that you correctly identify dependent groups designs. The statistics that can be used
for correlated groups are the McNemar Test (two samples or times of measurement), Wilcoxon t
Test (two samples), Dependent Means t Test (two samples), Friedman ANOVA for Ranks (three or
more samples), Simple Repeated Measures ANOVA (three or more samples), and Mixed Factorial
ANOVA (at least one factor is linked/correlated).

Independent Means - When there is no subject overlap across groups, we define the groups as
independent. Tests of gender differences are a good example of independent groups. We cannot be
both male and female at the same time; the groups are completely independent. If you want to
determine whether samples are independent or not, ask yourself, "Can a person be in one group at
the same time he or she is in another?" If the answer is no (can't be in a remedial education program
and a regular classroom at the same time; can't be a freshman in high school and a sophomore in
high school at the same time), then the groups are independent.

The statistics that can be used for independent groups include the chi-square test of independence
(two or more groups), Mann-Whitney U Test (two groups), Independent Means t test (two groups),
One-Way Between-Groups ANOVA (three or more groups), and Factorial ANOVA (two or more
independent variables).

Study - It has been traditional for the man rather than the woman to receive the check when a
couple dines out. A researcher wondered whether this would be true if the woman was clearly in
charge, asking for the wine list, questioning the waiter about dishes on the menu, etc. A large
random sample of restaurants was selected. One couple was used in all restaurants, but in half the
man assumed the traditional in-charge role, and in the other half the woman was in charge. At each
restaurant, the couple recorded whether the check was presented by wait staff to the man or to the
woman.

Test the research hypothesis that the check will be presented to the person showing in-charge
behavior.

Analysis - In this study, the sample consists of the wait staff at randomly selected restaurants. At
each restaurant either the man or the woman assumed the in-charge role. Thus, experimental
procedures resulted in two separated samples of subjects -- wait staff serving an "in-charge" man and
wait staff serving an "in-charge" woman. Given that a couple was dining together at each restaurant,
these are independent groups; one subject could not be assigned to both the "in-charge man" and
the "in-charge woman" experimental groups. Thus, the statistic selected must be one for
independent groups.

The scale of measurement of the dependent variable in this study is nominal (check presented to
man or woman), thus the appropriate statistic would be the chi-square test of independence. This
statistic allows us to determine whether the frequency of check presentation to the male or female
diner varied by whether the male or female was assuming the in-charge role.

Our last step in selecting the appropriate statistic for analysis is to determine whether the
assumptions of the statistical procedure have been met.

Assumptions
The final factor that we need to consider is the set of assumptions of the test. All parametric tests
assume that the populations from which samples are drawn have specific characteristics and
that samples are drawn under certain conditions. These characteristics and conditions are
expressed in the assumptions of the tests. Nonparametric tests make assumptions about sampling
(random) and the independence or dependence of samples (varies by test) but make no assumptions
about the population.

Parametric Assumptions - Listed below are the most frequently encountered assumptions for
parametric tests. Statistical procedures are available for testing these assumptions. The Kolmogorov-
Smirnov Test is used to determine how likely it is that a sample came from a population that is
normally distributed. The Levene test is used to test the assumption of equal variances. If we violate
test assumptions, the statistic chosen cannot be applied. In this circumstance we have two
options:

1. We can use a data transformation

2. We can choose a nonparametric statistic

If data transformations are selected, the transformation must correct the violated assumption. If
successful, the transformation is applied and the parametric statistic is used for data analysis.

Study - It has been traditional for the man rather than the woman to receive the check when a
couple dines out. A researcher wondered whether this would be true if the woman was clearly in
charge, asking for the wine list, questioning the waiter about dishes on the menu, etc. A large
random sample of restaurants was selected. One couple was used in all restaurants, but in half the
man assumed the traditional in-charge role, and in the other half the woman was in charge. At each
restaurant, the couple recorded whether the check was presented by wait staff to the man or to the
woman.

Test the research hypothesis that the check will be presented to the person showing in-charge
behavior.

Analysis - We selected the chi-square test of independence for data analyses because the dependent
variable is measured on a nominal scale of measurement and we have two independent groups in our
design (in-charge man and in-charge woman). The chi-square test of independence is a
nonparametric test, so we make no distributional assumptions about check presentation in the
population. The chi-square test of independence does require random sampling and independence
of observation. Our study meets both of these assumptions, so we can proceed to data analysis.

We have reviewed the steps needed to correctly select the statistic needed to test our hypothesis.
Proceed to the Practice Exercises to test your knowledge.

Review of Tests
We have learned a broad range of univariate inferential statistics. These statistics are divided into
nonparametric and parametric tests. Within nonparametric statistics, tests are futher divided into
those that are appropriate for nominal data and those that are applied to ordinal/rank data.
Parametric tests make no distinction between approximately interval, interval, or ratio data. These
are all considered "scale" data.

Nominal Data - The following tests can be used with nominal data. Which test you select is
determined by the number of samples and whether you are testing a hypothesis about group
differences or the association between independent and dependent variables. If you are testing a
hypothesis about group differences, you also must consider whether the groups/samples are
independent or dependent.
Ordinal Data - The following tests can be used with ordinal data. Which test you select is
determined by the number of samples and whether you are testing a hypothesis about group
differences or the association between independent and dependent variables. If you are testing a
hypothesis about group differences, you also must consider whether the groups/samples are
independent or dependent and how many samples you have in the study design.

Scale Data - The following tests can be used with all types of "scale"data. These include dependent
variables measured on approximately interval, interval, or ratio scales of measurement. Which test
you select is determined by the number of samples and whether you are testing a hypothesis about
group differences or the association between independent and dependent variables. If you are testing
a hypothesis about group differences, you also must consider whether the groups/samples are
independent or dependent and how many samples you have in the study design. Some parametric
statistics also permit analysis of more than one independent variable.
Summary
Five issues must be considered when choosing statistical tests.
• Scale of measurement
• Number of samples/groups
• Nature of the relationship between groups
• Number of variables
• Assumptions of statistical tests