Statistics and Research Methodology II

Ashoka University

Spring 2018
Instructor: Assistant Professor Kai Qin Chan |

Co-instructor: Aakanksha Mehta | Peer tutor: Dhairyya Singh

Course description

How would you go about to find out whether women talk more than men, whether
Indians are more interdependent than Americans, or whether nutrition improves IQ?
Quantitative data is merely a bunch of structured numbers; they cannot be interpreted
meaningfully until one understands and appreciates how the data came about. In this course,
we focus on methodological issues in sampling, designing experiments, correlation studies,
and ethics. This course is intended for students who already know the basics of statistical
analyses. The aim is to develop an eye for sound methodology, so that they can better
evaluate other scientist’s research, as well as their own.

Prerequisites

Pass in Statistics and Research Methodology I, or equivalent modules in math,

economics, or political science.

Course goals

By the end of this course, you will be able to:

 Compose a psychometrically valid questionnaire;

 Formulate hypotheses, test them, analyze the results, interpret them, and communicate
your results;
 Become a literate consumer of scientific information;
 Appreciate why meaningful interpretation of data relies on both statistical and
methodological understanding.

Required

 Zechmeister, J. S., Zechmeister, E. B., & Shaughnessy J. J. (2009). Essentials of

research methods in psychology. New Delhi: Tata McGrawhill. [Price: ~ Rs 650]
 RStudio and R
 JASP (download the Dec 2017 version)

Optional book
 Teetor, P. (2011). R Cookbook. New Delhi: O’Reilly. [Price: ~ Rs 420]

1
 Evaluation for students taking this as a 4-credit course:1

 [Individual] Quiz on syllabus: 5%

 [Group] Questionnaire Development Report: 15%
 [Individual] Attendance: 8% (0.5% for each class)2
 [Individual] Final Exam: 36%
 [Group] Final Project Report: 36%

Evaluation for students taking this as an audited course:

Students need to attend at least 80% of the classes, pass the quiz (50%) and the exam (50%) to
get the AU on their transcript. Audit students do not need to do the two reports.

Office hours

Not available on Mon/Wed 10 am – 1 pm; Tues/Thurs 8 am to 1 pm. My office is at Office

608 (old block). I prefer you to just drop by rather than email for appointment, unless you
really need to see me at a specific date/time. If you are afraid of making a wasted trip, you can
also call my office to see if I am in: 130-230-0394.

Seminar schedule

Week Date 1 Session 1 Date 2 Session 2

1 23/1 Introduction (Chap 1-2, & p. 25/1 Ethical issues in psychology
116-119) research (Chap 3)

Psychometrics: Dorst, E. A. Remember to install R and RStudio,

(2011). Validity and reliability in and also download Dec 2017 version
social science research. of JASP before the next class!
Education Research and
Perspectives, 38, 105-123.

In addition, we will take some time

in class to form up your project
groups for Questionnaire

1
Grades may be moderated. See subsection Explanatory Notes on Moderation on p. 9.
2
Excluding Week 1 and 2, there are 20 in-class sessions. For each class that you attend, you earn 0.5%, up to a
maximum of 8%. This means you can miss 4 in-class sessions without penalty. There will be no differences
between “excused” and “unexcused” absences. This implies that if you willfully decide to be absent 4 times
early on in the semester, you will be penalized from the 5 th time onwards even if you are genuinely ill, need to
participate in sports, have emergencies, have looming deadlines, etc. Bereavement and long-term medical or
psychological conditions will be considered separately. Inform the instructor in Week 1 if you need special
considerations.

2
 Development Report.
Complete an online
questionnaire, which will be
used in the demo session of
Week 2.1:
https://docs.google.com/forms/d/e/1F
AIpQLSdgpqpqNNUFLVImmM7jUpT-
qHdM9FNWgelVEz-
CsXWQ6W84JA/viewform?usp=sf_lin
k
2 30/1 Demonstration session [bring
laptop]
Install R and RStudio, and also
download Dec 2017 version of
JASP before class!
Analyzing the psychometric
properties (validity and
reliability) of an established
questionnaire using R and/or
JASP.
Form up groups for both your
Questionnaire Development
Project and Final Project.
Advice: You should start
working on refining your chosen
construct and writing questions
for the Questionnaire
Development Report.
[Attendance counted: 1]
3 6/2 Quiz on syllabus (5 mins)
Introduction to experimental
research
(Chap 6)
Advice: You should be analyzing
data and preparing for the group
presentation in this week.
1/2 Discuss: Kramer, A. D. I., Guillory,
J. E., & Hancock, J. T. (2014).
Experimental evidence of massive-
scale emotional contagion through
social networks. Proceedings of the
National Academy of Sciences, 111,
8788-8790.
Discuss: Loftus, E. F., & Pickrell, J.
E. (1995). The formation of false
memories. Psychiatric Annals, 25,
720-725.
Discuss: Martens, A., Kosloff, S.,
Greenberg, J., Landau, M., &
Schmader, T. (2007). Killing begets
killing: Evidence from a bug-killing
paradigm that initial killing fuels
subsequent killing. Personality and
Social Psychology Bulletin, 33,
1251-1264.
Advice 1: Focus on the methods, not
the introduction, statistics, or
discussion.
Advice 2: You should be collecting
data for your QDR.
[Attendance counted: 2]
8/2 Control problems in research
(Chap 6)
[Attendance counted: 4]
Reminder: Send list of priorities of
your project by 15 Feb!
3
 [Attendance counted: 3]
4 13/2 Stress-free student presentation
on Questionnaire Development
(ungraded)
Advice: Use feedback from your
classmates to improve your
report.
Due to time limits and the large
class size, extra session for 10:10
am class will be arranged.
[Attendance counted: 5]
5 20/2 Single-factor designs
(Chap 6)
[Attendance counted: 7]
15/2 Assignment: Analyze a research
design I
Discuss: Baker, L., & Lombardi, B.
R. (1985). Students' lecture notes
and their relation to test
performance. Teaching of
Psychology, 12(1), 28-32.
Discuss: Chenneville, T., & Jordan,
C. (2008). Impact of attendance
policies on course attendance among
college students. Journal of the
Scholarship of Teaching and
Learning, 8, 29-35.
Discuss: Mehl, M. R., Vazire, S.,
Ramírez-Esparza, N., Slatcher, R.
B., & Pennebaker, J. W. (2007). Are
women really more talkative than
men? Science, 317(5834), 82-82.
Use the Template for analyzing
research designs.docx to help you
deconstruct the methods.
Advice: Treat your readings
seriously. The skills you acquire are
precisely the ones tested in your
exam.
Advice: You should be finishing up
your Questionnaire Development
Report.
[Attendance counted: 6]
22/2 Assignment: Analyze a research
design II
Discuss: Ackerman, R., & Lauterman,
T. (2012). Taking reading
comprehension exams on screen or on
paper? A metacognitive analysis of
learning texts under time pressure.
Computers in Human Behavior, 28,
1816–1828.
Discuss: Wookey, M. L., Graves, N.
A., & Butler, J. C. (2009). Effects of
4

6 27/2 Factorial design I: Fully
independent, fully dependent
(Chap 7)
[Attendance counted: 9]
Supplementary reading
Fully between-subjects:
https://www.discoveringstatistics
.com/repository/twoway.pdf
Fully within-subjects:
https://www.discoveringstatistics
.com/repository/repeatedmeasure
s.pdf
a sexy appearance on perceived
competence of women. Journal of
Social Psychology, 149, 116-118.
Chen, P., Chavez, O., Ong, D. C.,
Gunderson, B. (2017). Strategic
resource use for learning: A self-
administered intervention that
guides self-reflection on effective
resource use enhances academic
performance. Psychological Science,
28, 774-785.
Use the Template for analyzing
research designs.docx to help you
deconstruct the methods.
Advice: Treat your readings
seriously. The skills you acquire are
precisely the ones tested in your
exam. Focus on the methods section.
[Attendance counted: 8]
1/3 Power analysis[bring laptop]
Install G*Power
http://www.gpower.hhu.de/en.html
Discuss: Sherman, G. D., Haidt, J.,
& Coan, J. A. (2009). Viewing cute
images increases behavioral
carefulness. Emotion, 9, 282-286.
Use the Template for analyzing
research designs.docx to help you
deconstruct the methods.
Advice: Treat your readings
seriously. The skills you acquire are
precisely the ones tested in your
exam.
Data analysis:[bring laptop]
In this tutorial, we will also analyze
a dependent groups design with
more than two levels
Dataset & R codes:
dataset_earthquake.csv &
dataset_earthquake.r
5

7 6/3 Factorial design II: Factorial
ANOVA – Between- and within-
subjects designs
Data analysis: In this tutorial,
we will analyze fully between-
subjects and fully dependent-
subjects factorial designs. [bring
laptop]
Dataset & R codes:
dataset_haloeffect.csv &
dataset_haloeffect.r
dataset_memory.csv &
dataset_memory.r
dataset_emotionVideo.csv (if
time permits)
[Attendance counted: 11]
8 Mid-semester break (10/3 – 18/3)
9 20/3 Correlational research, surveys
(Chap 5), observational (Chap 4)
10 27/3 Data analysis: We will analyze
more datasets. The focus is on
consolidating your knowledge
about factorial design analyses.
[bring laptop]
Datasets will be uploaded later;
R codes won’t be.
[Attendance counted: 10]
8/3 Factorial design III: Factorial
ANOVA – Mixed designs (Chap 7)
[bring laptop]
Data analysis: In this tutorial, we
will analyze mixed factorial designs,
as well as clear up any remain issues
about factorial ANOVAs.
Dataset & R codes:
dataset_stroke.csv &
dataset_stroke.r
dataset_bassin.csv &
dataset_bassin.r
[Attendance counted: 12]
Supplementary reading
Mixed ANOVA:
http://www.stat.cmu.edu/~hseltman/
309/Book/chapter11.pdf
[Attendance counted: 13]
22/3 Data analysis: We will analyze two
more datasets, but this time, the
focus is on learning how to write a
results section that is clear,
complete, and unambiguous.[bring
laptop]
dataset_potato.csv
dataset_pizza.csv
Advice: This activity prepares you
for Q3 of your exam.
[Attendance counted: 14]
29/3 Stress-free Research Proposal
Presentation (ungraded)
Advice: Use feedback from your
classmates to improve your research.
Due to time limits and the large
class size, extra session for 10:10 am
class will be arranged.
6
11 3/4 Quasi-experimental designs 5/4 Activity A (45 mins)
(Chap 9), small N designs (Chap SRM II exam review of key
8) concepts
[Attendance counted: 15] Activity B (45 mins)
I will hand out a sample exam paper.
We will try a few questions and
discuss answers in class

[Attendance counted: 16]

12 10/4 Activity A 12/4 Exam: All content up to and

You will learn how to write a including Week 11.
methods section that is clear,
complete, and unambiguous. [Attendance counted: 18]

We will watch a video of an

experiment on frogs and
“worms”. You will describe
what was done in the
experiment.

Advice: This activity prepares

you for Q1 of your exam and
Methods section of your Final
Project Report.

Activity B
Mock Exam: You will have 25
mins to write out the design of
an experiment to answer a
research question. We will then
spend 20 mins discussing your
answers. The research question
will mimic Q1 of your exam.

[Attendance counted: 17]

13 17/4 Data analysis & presentation
preparation (no classes)
Advice: Read Appendix B on
“Communication in Psychology”
19/4 Make-up week
Finale session: What is an average?
(45 mins)
Release of Final Exam results +
Open consultation (45 mins)

Read: Chap 10, and Molenaar &

Campbell (2009). The new person-
specific paradigm in psychology.
Current Directions in Psychological
Science, 18, 112-117.
14 24/4 Stress-free Final Project 26/4 Stress-free Final Project

7
 Presentation: Groups 1, 2, 3 Presentation: Group 4, 5

Due to time limits and the large Open consultation

class size, extra session for 10:10
am class will be arranged. [Attendance counted: 20]

[Attendance counted: 19]

15 1/5 [Make-up for SRM I] 3/5 No class, unless any class above is
Make- We will spend about 30 mins on cancelled.
up power analysis since this was not
week covered in SRM I.

Install G*Power before coming

to class and bring your laptop.

If you want, we can also practice

analyzing more datasets.

Other issues

Academic dishonesty
Do not plagiarize or cheat in exams and assignments. I do not grade plagiarized work, or give
a chance for resubmission. Ashoka University expects you to fulfill your academic obligations
through honest and independent effort. It is your responsibility to familiarize yourself with
Ashoka’s policy about academic dishonesty in your Student Handbook. And as a final
warning: Merely citing sources without paraphrasing is still plagiarism. Consult the CWC if
in doubt what plagiarism is. In the last 3 semesters, I have caught at least 5 students
plagiarizing (one was suspended). Do not become the next student.

Contingency plans
In the past few years, classes have been disrupted because of smog and social unrests in the
region. If classes are cancelled and make-up sessions are impossible to schedule, the
remaining attendance percentage will be distributed equally.

How to do well in this course?

This is the course where everything you learnt in SRM I falls into place, where you start to
appreciate why statistics and research methodology are both important in psychology. The
expected hours you need to put in is 10 hours per week. This 10 hours is divided
approximately as such, but this can vary week to week:
 3 hrs – in-class
 2 hrs – revising Session 1
 1 hr – preparing for Session 2
 4 hrs – working on your projects

8
You decide what you want to get out from the course, and work towards it. There are no
shortcuts in this course.

The math behind the statistics

In SRM I and II, we covered the math behind the statistics at a surface level. For those curious
about the math, you can read Cardinal (2004) here:
https://egret.psychol.cam.ac.uk/psychology/graduate/Guide_to_ANOVA.pdf. This is what
you will learn more in graduate school in psychology. If you were to do the same content in a
statistics department, the math behind it is even more complex, often involving matrix algebra
and triple summations.

How much of math do you really need to know? Not much – if you can understand
conceptually what variance partitioning is, that’s good enough for this level; but if you
struggle to understand why the formula for standard deviation makes sense, then I worry for
you. My personal take is that you only need to understand the super advanced math if you
want to specialize in quantitative psychology. Check out some of the world’s best statistical
psychologists such as David McKinnon, Andrew Hayes, Kristopher Preacher, Leona Aiken,
Steven West, Mike Cheung, etc. (By the way, quantitative psychologists are among the
highest paid and most employable psychologists.)

Am I shortchanged if I don’t know the math? Most carpenters don’t know ionic and covalent
bonds. Are they less competent as carpenters? Probably not. (You can turn the question
around: Would a scientist fully competent in ionic and covalent binds be a good carpenter?)

9
 Detailed components

The first step to doing well in this course is to take charge of your own learning; start by
knowing what is expected in this course. Knowing the syllabus of any course you take is
5-min Quiz

important because it helps keep yourself on track; you know what is expected of you,
when, and why. Unfortunately, many students do not read syllabus.

This is an open-paper quiz. There will be 10 factual questions, and answers can be
found in the syllabus.

The discussion articles equip you with a variety of skills to answer Q2 and Q3 of your
exam. But if you simply read passively, you won’t get much out of it. What is useful is
Discussion articles

to generate questions and try answer to answer them (or get your fellow classmates to
answer them!). Better still: Generate tricky questions that your classmates find it difficult
to answer.

Essentially, this is also how I set questions in Q2 and Q3: I pick an article, and generate
methodological and statistical questions. This is exactly the same way that will help you
if you are conscientious in reading the discussion articles.

There will be three questions.

a) Q1 will ask you to devise an experiment.
b) Q2 and Q3 will ask you to answer statistical and methodological questions. You
will not need to run any analysis. The mock exam will give you the style of my
questions.
Exam

c) Q3 will also ask you to write out the results of a statistical output.

Advice: Treat (c) seriously. It looks like a no-brainer, but it is challenging.

I emphasize answers that are clear, complete, and unambiguous.

Exam is open-book; internet connection is strictly prohibited.

10
 Questionnaire Development
Questionnaires are often used in social sciences. How do you compose a set of
questionnaires that is valid and reliable – and how would you know if they are? We will
learn about validity and reliability in class. Then you will work in groups to devise a set
of questionnaires and administer these questionnaires to your fellow classmates and
friends. Aim to collect as many responses as you can. After that, you will analyze the
validity and reliability of your questionnaire, present them in class (15 mins – ungraded),
and write a short group report (graded).

Constructs
Choose one of the following constructs. First, attempt to define the construct.
Think of whether there is a hierarchical structure involved (a hierarchy may not be
necessary). Drawing diagrams usually help at this stage, especially when there is a
hierarchy. Word the items, along with their numerical and verbal anchors. Aim for no
more than 10 items in total. Have your classmates complete them (and you should also
complete your classmates’ questionnaires). Collect additional data from other friends, if
necessary. Students who are curious about hierarchical structures may consult Google or
YouTube on how to do a factor analysis.
Questionnaire Development

Philosophical Political Slacker Entrepreneurial

correctness spirit

Perfectionism Sense of justice Grade anxiety Hypocrites

Sense of insecurity Distrust in science Environmentalism Social loafing

Adaptability Ingrate Back-stabber Foodie

Group sizes
8:30 am: 5 groups max
10:10 am: 10 groups max

Questionnaire Development Report (max. 1000 words)

Your report needs to explain your constructs. Draw diagrams if you have to.
After collecting and analyzing the data, discuss the psychometric properties of your
questionnaire. Attach your questionnaire items as an Annex, along with the appropriate
correlation matrix.
The aim is to produce a questionnaire that has good psychometric properties.
However, your grade is not determined on the outcome (i.e., the psychometric
properties), but rather the process. In other words, it is possible to get excellent grades
even if the psychometric properties are suboptimal.

Deadline - Questionnaire Development Report

19 Feb, 1800 hrs.

11
 Final Research Project
Statistical methods, particularly as they are applied in research, are best learned
through practice. In SRM I, you did secondary data analysis – analyzing data collected
by others. In SRM II, you will experience the whole research process from start to end.
The main intention is to practice formulating sound research questions, testing your
hypothesis, and communicating what you found via a presentations (ungraded) and a
report (graded).

Group Presentation – Research Proposal (ungraded)

Scientists often gather feedback from peers before launching an investigation.
As budding scientists, you will do the same. Your presentation will last 15 mins – you
will present for 10 minutes, allocating an additional 5 mins to gather feedback from your
classmates. You may decide on the number of speakers presenting. You will need to
present the following: Intro; Hypothesis; Design; Expected results (please graph it);
Analytic strategy.

Advice: Keep your presentation to 5 slides (excluding title slide)

Group Presentation – Research Results (ungraded)

After concluding a research, scientists often want to communicate their findings
Final Research Project

to their peers. As budding scientists, you will do the same. Your presentation can last up
to 25 mins. Allocate some time to solicit feedback from your classmates. You may
decide on the number of speakers presenting. You will need to present the following:
Intro; Hypothesis; Design; Results (please graph it); Limitations.

Advice: Treat your presentation seriously. In order to solicit effective feedback, you need
to communicate clearly your content.

Group sizes
8:30 am: 5 groups max
10:10 am: 10 groups max

Deadline – Research Report

1. Each group submits ONE group report.
2. To prevent social loafing, each individual of the group submits a confidential
assessment of each member’s contribution, including the individual him/herself.
The idea is not to “outcompete” others but to ensure everyone doe his/her fair
share of work. Thus, your grade may be moderated by others’ assessment of your
contribution. More details will follow as the semester goes on, but briefly, the
assessment includes items such as:
 Keeping abreast of group progress
 Sharing ideas
 Completing tasks on time
 Attending meetings
 Demonstrating respect for others
 Contributing to group discussions

7 May, 1800 hrs

12
 Explanatory Notes on Moderation

There are two families of grading scheme: absolute grading scheme (AGS) and
relative grading scheme (RGS). Simply put, absolute grading defines letter grades based on
predefined cut-offs scores, whereas relative grading defines letter grades based on a student’s
rank-ordered scores vis-à-vis other students. In reality, each grading scheme exist is a family
of grading techniques. For example, there are at least three ways to implement absolute
grading3, and many more ways to implement relative grading.

I use the relative grading scheme, specifically relative cluster grading, because it fits
my education philosophy and understanding of statistics. This means I rank order scores and
look for “precipitous” drops. These drops form my cutoff scores. Technically A and A- signal
categorical differences, but under an absolute grading system, this categorical system is
artificially created — a difference of 0.1 point can mean the difference between A and A-.
With a relative cluster grading, the categorical differences between letter grades is naturally
reflected in the data as the “precipitous drops”.

Because relative cluster grading is a data-driven technique, there is no fixed rule how
large a cluster can be, or how many grade bins there must be. Occasionally, I might skip a
grade bin if the deviation between two adjacent clusters is too large. How you measure “large”
is debatable. One can go for “≥ x point difference = large” or one can calculate average range
per cluster and use that range to define how large is large. For example, if the average range is
3.5, and at the border of two clusters you have scores that differ by 8, then you could drop 2
grades down. My personal definition is that a cluster must have adjacent scores ≥ 1.0. Here is
a real example from SRM I (2015):

Raw score RGS (cluster) AGS

57.98 D+ D+
58.26 D+ D+
58.36 D+ D+
62.25 C C-
63.12 C+ C-
65.44 B- C+
69.73 B C+
72.26 B B-
72.56 B+ B-
73.10 B+ B-
73.65 B B-
73.70 B+ B-
74.18 B+ B-
77.02 A- B
77.63 A- B
80.53 A- B+
84.98 A B+
85.04 A A-

3
Walvoord, B. E. (1998). Effective grading: A tool for learning and assessment in college. Jossey-Bass.

13
 87.57 A A-
95.13 A A

Here are some frequently asked questions about RGS:

Q: What are some advantages of relative cluster grading?

A: RGS offers some protection such that regardless of the course’s difficulty, there will be
A’s. Also, RGS fosters cooperative learning: If all scores fall into a tight cluster, all will get
the same score. TEAMwork: Together Everyone Achieves More.

Q: Are there situations where AGS is better over RGS suited?

A: Yes, when there are clearly right and wrong answers (e.g., MCQs, fill-in-the-blanks).

Q: Doesn’t relative grading system induce cut-throat competition?

A: This is true only if relative grading is inflexible e.g., there must be 10% A’s, 30% B’s, 30%
C’s, etc. This is sometimes known as “curving”. I do not use curving.

Q: Isn’t it impossible for everyone to get an A under relative grading?

A: If the distribution follows a uniform distribution, everyone gets an A. If the distribution

follows a near uniform distribution, everyone likely gets an A. The questions you need to ask
are: What is the distribution? How will anyone know the distribution before seeing the data?

Q: Does absolute grading creates certainty whereas relative grading creates

uncertainties?

A: Absolute grading does create certainty if and only if the predefined score is attainable (e.g.,
MCQs); relative grading creates uncertainties; that is true because you are unlikely to know
how well others are doing. How do you manage uncertainty? Do your best, don’t compare.

Q: Isn’t relative grading is statistically illogical in small samples?

A: It is true that when sample size, k  ∞, scores x ~ N (𝜇 = 0, 𝜎 = 1). But RGS ≠ grading
using normal distribution. RGS is a family of techniques, some of which (e.g., cluster grading)
do not rely on assumptions of normality.

14
Q: Isn’t RGS’ cut-off points are arbitrary?

A: This argument applies to both RGS and AGS.

Q: Grades are continuous variables, but RGS creates artificial categorical outcomes?

A: This argument applies to both RGS and AGS. This is the result of the letter-grade system,
not of RGS or AGS. Any downward change in the levels of measurement (continuous 
categorical) loses sensitivity. (This is why you should never dichotomize continuous variables
in an ANOVA but should instead use regression techniques to prevent Type I or Type II
errors.)

Q: Is AGS bad, whereas RGS is superior?

A: Each has its advantages and disadvantages. Neither is superior over the other. It depends
on the subject, instructor’s education philosophy, instructor’s understanding of statistical
distributions.

15