EMORY UNIVERSITY

QTM 110 – Fall 2021

Introduction to Research Design

Dr. Greg Sasso

gregory.sasso@emory.edu

Course Meeting Times:

Monday & Wednesday 10 – 11:15 AM on Zoom

Office Hours:
Thursday 9:30 – 10:30 AM on Zoom

Individual Office Hours:

By Appointment

Zoom Rooms:
For Class Sessions: https://emory.zoom.us/j/96152229572
Password: correlated
For Office Hours:  https://emory.zoom.us/j/96386046811

Password: causal

COURSE DESCRIPTION
QTM 110 is the first course in the sequence of requirements for the Quantitative Sciences (QSS)
major; as an introductory course, there are no prerequisites. The course is designed to introduce
students to the style of analytic thinking required for research and the concepts and procedures
used in the conduct of empirical research. In short, this course teaches a set of skills that are
essential for both understanding the research you will encounter in substantive classes, and
being able to produce high-quality original research of your own. Beyond simply learning how to
be a more critical participant in the academic research community, you will also be better
prepared for career opportunities using statistical tools and the products thereof. Whatever the
individual career goal, students will learn the principles of critical thinking essential for drawing
well-reasoned inferences from data.

The course is organized into four parts. In part one, students will be introduced to the key
concept of causality. What is causality and how is it different from correlation? What are
common inferential errors you will encounter when presented with research findings? In part
two, we examine the “gold standard” of causal inference, experimental empirical research.
While an excellent tool for identifying causal empirical relationships, we cannot always use
experiments to answer important empirical questions. Part three introduces the alternative,
observational empirical data analysis. Critically, we discuss inferential challenges to using
observational data, as well as some tools that can help overcome those challenges. Finally, the
fourth part covers some topics critical to doing empirical research whether experimental or
observational, including how we can use priors in empirical data analysis, problems of prediction,
and modern topics related to presenting research findings.

TEACHING AND LEARNING DURING THE PANDEMIC

Even though restrictions have been reduced, the world is not yet back to normal. Therefore,
many policies implemented due to the pandemic will continue through this semester. As in
previous semesters, communication is important. I will try my very best to respond to emails
and questions posted to Canvas within 48 hours. There will be well over 100 students across all
sections taking QTM 110 this semester. In order to facilitate efficient communication, I ask that
you post questions related to material and administrative policy to Canvas and to check Canvas
and the syllabus before asking a question. If you have a question of a personal nature then
please email me. I will likely be slower on weekends and it is usually not a great idea to ask
questions on a Friday night or right before something is due.

If your situation changes regarding health, housing, or in any other regard with respect to your
ability to participate in the class, please contact the appropriate Emory student support
organization first and then me as soon as feasible. It is easier for me to address your needs if I
know about them as soon as they arise. This does not mean I can successfully respond to every
request for consideration, but I emphasize that my goal is to treat you all equitably and do what I
can to help you succeed in this course.

ATTENDANCE POLICIES
Though attendance and participation in lectures is expected, there is no formal grade associated
with these aspects of the class. That said, previous semesters of this course have shown a strong
positive correlation between a student’s final grade and active participation in the lectures. 1

If you are not feeling well, please do not attend lectures in person! If you are sick, understand
that I will be flexible about attendance and keeping up with work. If you expect that illness or
other circumstances will prevent you from attending more than a single lecture, please make
sure to email me so that we can discuss your individual circumstances. I will be recording each
lecture and posting the recording to the course’s Canvas site.

However, these recordings are not intended to be a substitute for the actual lectures! Much of
the content in this course centers on discussion and group problem solving that occurs during
the lectures. The recording setup will not provide high quality recordings of these discussions

1 Note that this only a correlation derived from observation and there are most certainly confounding variables
that muddy the causal relationship between these two events. However, underlying beliefs about the likelihood of
this relationship should likely lead you to interpret this relationship as causal.
and there will be no further efforts made to improve the quality of the recordings - what you get
is what you get. If I determine that attendance has dipped below an acceptable level, I will stop
posting the recordings and distribute them to students by request only. If everyone uses these
recordings responsibly (as I expect will be the case), then this common good will benefit the class
greatly.2

ACCESSIBILITY AND ACCOMMODATIONS

As the instructor of this course, I endeavor to provide an inclusive learning environment. I want
every student to succeed. The Department of Accessibility Services (DAS) works with students
who have disabilities to provide reasonable accommodations. It is your responsibility to request
accommodations. In order to receive consideration for reasonable accommodations, you must
register with the DAS at http://accessibility.emory.edu/students/ . As a matter of courtesy and
to allow me to best address any specific needs, I am requesting that all students register and
ensure that DAS has sent me any accommodation letters in the first two weeks of the course.
Accommodations cannot be retroactively applied so you need to contact DAS as early as possible
and contact me as early as possible in the semester to discuss the plan for implementation of
your accommodations. For additional information about accessibility and accommodations,
please contact the Department of Accessibility Services at (404) 727-9877 or
accessibility@emory.edu.

HEALTH CONSIDERATIONS
At the very first sign of not feeling well, stay at home and reach out for a health consultation.
Please consult the campus FAQ for how to get the health consultation. As you know, Emory does
contact tracing if someone has been diagnosed with COVID-19. A close contact is defined as
someone you spend more than 15 minutes with, at a distance less than 6 feet, not wearing facial
coverings. This typically means your roommates, for example.

COURSE REQUIREMENTS

IMPORTANT NOTE: All times listed on this syllabus are given in Eastern Standard Time, the time
zone at Emory’s campus in Atlanta, GA. Please be sure to take any time zone differences into
account when submitting assignments. No special considerations will be made for missed
assignments due to time zone differences - adequate time for completion of assignments will be
given regardless of your time zone.

Lectures: Weekly lectures will assume that the week’s readings and case studies have already
been reviewed. We will begin each session with an overview of the week’s topic and a question
and answer session. Then, we will move into a more interactive application of each topic - case
studies, discussions of potential research snags, etc. This will often occur as smaller discussions
in groups and I expect that everyone will attempt to contribute to the discussions. Finally, I will
randomly select one group to lead off that week’s discussion.

2 We can avoid a tragedy of the commons if no one abuses the system.

Readings: Lectures will operate on the assumptions that all students have at least skimmed the
readings and other materials that have been posted on Canvas for that week. Some weeks will
have readings from the textbook that are long. Know that you are not responsible for all of the
details in the readings – I only expect that you’ve attempted to understand the main ideas of the
text. To prepare for lectures, I recommend writing 5 one-sentence bullet points that summarize
the main points of that week’s readings. Not only is this a great way to prepare to contribute to
the weekly discussion/prepare questions, it will also serve as a useful study tool for exams.

Lectures will often discuss specific case studies that have been posted to the course’s Canvas site
prior to the in-person meeting. These case study readings will be used to introduce real world
examples of the topics being discussed in class. Along with readings from the textbooks, it is
expected that students have at least familiarized themselves with the main points of the case
study materials. These articles will be used as a bridge to questions that are more like what you
will see on exams – they will bring up points about the messiness of research and how
misapplication of assumptions can lead to undesirable inferences about real-world phenomena.
In many ways, case studies are the most important component of this course. To get the most
out of these case studies, please prepare a few notes prior to the class discussion to help drive
our conversation. These can be summaries of the main findings, questions about the material,
disagreements you may have with the authors, etc. This class will be much more enjoyable if
these discussions are more of a conversation.

Discussion Threads: Discussion threads will be used to post questions about class material.
Direct emails to the instructor are reserved solely for questions pertinent only to that student.
This policy ensures that all students benefit from instructor feedback and clarification of
material, as well as allowing for fruitful interaction among students around that material.

Working Groups and Collaboration: Each student will be assigned to a group that they will work
with on graded group assignments (problem sets and preparing case study discussion points) for
the entire semester. Each group will consist of 3 members (and potentially 1 or 2 groups of 2).
In the first week of classes, I will post an introductory survey on the course Canvas site where I
will ask some general questions about your interests and previous background in statistics and
research. I will use this info to assign groups before the second week of the course.

Group assignments are important in a course like this as many jobs in data science and related
fields will require working with diverse sets of individuals to produce a single deliverable - aside
from the topics covered in this course, I hope to provide you with practice working in this kind of
environment. I am a firm believer in groups that are diverse in experience, background, culture,
interests, etc. For this reason, all groups will be randomly assigned by me.

Group assignments are fixed and will only be changed under extreme circumstances. Group
members are expected to participate in all group assignments and share work in an equitable
manner. As a way to ensure that grades are commensurate with effort and participation in
group assignments, each assignment should include a statement at the top attesting to which
students worked on the assignment. Please be sure to indicate if any group members did not
meet your group's workload expectations. Note that this should be decided among those who
participated in your group - there is no competition for high grades, so there is no incentive to
lie. Cases where group members do not fairly contribute will be dealt with on an individual
basis.
Should the situation arise where a group member is consistently shirking work, please let me
know as soon as possible and we will work to come up with an equitable solution. If all else fails,
the offending student will be required to do the rest of the course's work alone.

Group Problem Sets: You will complete four group problem sets during the class that will
account for 35% of your final grade. Each of these assignments will help you understand and
apply the conceptual material presented in the course material. These problem sets will contain
questions that range in difficulty, testing both your basic understanding of the material and your
ability to create more advanced extensions across topics. Group problem set dates are outlined
on the syllabus and there will be frequent reminders in class and on Canvas. The instructor will
assign a grade for the turned-in answers as a whole and, in most cases, all members of the
problem set group will receive the same grade. However, I reserve the right to alter this policy
in circumstances where it is clear that a member/members of the group are consistently free
riding. When computing final grades, your lowest problem set grade will be dropped.

Quizzes: There will be 5 short open book and open note quizzes over the course of the semester
(the Welcome Quiz is the first one). Quizzes will typically be posted on Fridays and be due by
11:59 PM the following Friday (seven days later). These quizzes are intended to provide short
but meaningful assessments of your understanding of the materials from the preceding weeks.
They will include a mixture of multiple choice, true/false, and quick computation questions (I.e.
autograded question on Canvas). These quizzes will account for 15% of your final grade. When
computing final grades, your lowest quiz grade will be dropped.

Exams: In this course, you will complete two 75 minute midterms and one 150 minute final
exam that will cumulatively account for 50% of your final grade. For each student, the exam
with the highest grade will count for 20% of the final grade while the other two exams will
account for 15% each. You will have 24 hours during which you can take the exams. Once you
start the exam, you will then have 90 minutes for the midterms and 175 minutes for final (the
extra 15 minutes is to mitigate any technical problems). The midterm exams will take place on
Wednesday, October 13-14th to and Wednesday, November 17-18th. The final exam will take
place during the scheduled final exam period, December 10-11 th. Students are expected to take
the exams at the assigned times – flexibility in exam time is reserved for only the most urgent
of circumstances. Please check the final exam schedule early for all of your final exam times at
the end of the semester. If you find that you have more than 2 scheduled for the 10 th, let me
know ASAP so that I can schedule an alternative time.

COURSE GRADES
Students are expected to adhere to the Emory Honor Code when completing all aspects of the
class, including the assignments and exams.
ACTIVITY POINTS
Quizzes 15%

Group Problem Sets 35%

Exams 50%

Note that all assignments will be turned in on Canvas.

Final grades will be determined using the following (estimated) grade ranges:

A: 94%−100% A-: 90%−93% B+: 87%−89%

B: 83%−86% B-: 80%−82% C+: 77%-79%
C: 73%-76% C-: 70%-72% D+: 65%-69%
D: 60%-64% F: Less than 60%

As assignment scores are computed and I get more of a feel for the average grades on each
assignment, the grading rubric may be updated. However, the grade scale will only be loosened
from this initial rubric (e.g. if your final average is 94%, then you will get an A for the course
regardless of rubric changes). Final grade percentages will be rounded to the nearest integer.
There is no curve for final course grades - if everyone in the class scores high enough to get an A,
then everyone will get an A. Hence, there is no “competition” for grades.

LATE ASSIGNMENT POLICY

You will lose 50% of the possible points for any assignment that you submit late without my
explicit prior approval. Students may submit assignments up to two weeks after the original due
date to receive half credit. Any assignments turned in more than two weeks after the original
assignment date will receive a grade of zero.

If issues arise over the course of the semester, please let me know early; I understand that the
current state of the world has brought about new challenges and demands for all of us. Beyond
these new challenges, life happens quickly and unexpectedly. Please keep me in the loop and I
will try my best to fairly adjudicate.

COURSE RESOURCES

As the semester progresses, class materials (e.g., asynchronous materials, lecture presentations,
notes, readings and links to other resources) will be posted on Canvas.

Required Textbooks:
Bueno de Mesquita, Ethan and Anthony Fowler. 2019. Thinking Clearly in a Data-Driven Age.
Unpublished manuscript (available on the course website) – Referenced as BDM-F

Bueno de Mesquita, Ethan. 2014. Thinking: A Practitioner’s Guide. Unpublished manuscript

(available on the course website) – This is an older version of the BDM-F text, we will use it
sometime as an additional optional resource. It will be referenced as BDM-2014

Recommended Textbooks:
Ellenberg, Jordan. 2014. How Not to Be Wrong: The Power of Mathematical Thinking. Penguin
Books.

Cunningham, S., 2018. Causal inference: The mixtape. Yale University Press. - Available freely
online from the author here.

Additional readings listed in the course schedule will be provided in PDF format on the Canvas
site.

If you have any special needs, please contact me at the beginning of the semester and we will
discuss the necessary arrangements (see Emory’s Access, Disability Services and Resources).

Software and Technical Requirements: Information on the software we will be using and the
technical requirements for the online class are located at the end of the syllabus. Please note
that there are video tutorials on Canvas for each of the software packages (Zoom; Canvas).

One more important piece of information….

Canvas may be down occasionally for maintenance. Please plan ahead!

PROCEDURE FOR APPEALING A GRADE

If you believe that your grade on any assignment or exam question is incorrect or unfair, you
should submit your concerns, in writing, to the professor. There are two types of appeals:
1. Addition errors or other administrative snafus: Please let me know if you catch one of
these ASAP. I’ll confirm the error and change your grade in the gradebook.
2. Disagreement about grading criteria: Please wait two days (or the nearest non-weekend
day) after receiving your grade before submitting a written appeal. Do not leave
comments on the Canvas assignments for the TAs as they are not responsible for grade
appeals or score corrections. The written appeal should fully summarize what you
believe the problems are and why. The professor will consider your appeal. If you are
not satisfied with the response, you may resubmit the assignment for regrading in its
entirety by the professor. This grade will be final. Note that grades may go up or down
during an appeal. Failure to comply with this appeal process means that you forfeit your
appeal.
A note about grading in this course: I will have a team of graders that will help me grade
assignments. The graders are excellent and follow the rubric I give them as closely as possible.
Typically, the graders will give an initial grade and I will go back over every single grade to ensure
that students receive the points that I believe are appropriate. Only after this final pass will I
post the final grades to Canvas. Any grades that you may see in the comments on your
assignment are not final until I post them to the Canvas site (e.g. you get a message from Canvas
saying that the assignment’s grades have been posted). For your own sanity (and mine), please
do not worry about the grade you add up based on the Canvas comments and wait until they
have been posted.

I may make errors in this process – I am only human, after all. I will do my best to make sure all
students receive the grade that they deserve. I ask, in return, that you be patient with any errors
and respectfully explain your reasoning for any disagreements you may have.

FREQUENTLY ASKED QUESTIONS

Q: How much math do I need to know to succeed in this course?

A: Basic arithmetic is as much as you’ll need to successfully complete the course (mostly
averages and conditional averages). The purpose of this course is not to give you a thorough
introduction to the mathematics behind the statistical procedures we’ll discuss in class - rather,
this course will discuss the logic behind scientific evidence and what it means to claim that
something causes or predicts something else. Another way of thinking about it is that this class
is more about the “Why?” questions than the “How?” questions.

Q: Do I need to know any programming for this course?

A: No! We will not be using any software packages or coding in the course. There may be some
situations where you might want to make a graph for an assignment and you can choose to use
R, Excel, Python, or any other software of your choice. However, you can also draw things by
hand to receive full credit.

Q: I’ve heard this class has lots of reading and it can be difficult to keep up with it all. How do
you suggest that I approach the readings to maintain a reasonable workload?

A: There is a lot of text used in this class. However, there is no expectation that you read and
memorize every single detail from the posted readings. In fact, this is a bad strategy for
succeeding in this class and in many other classes. Rather than reading everything thoroughly, it
is important that you learn to skim and narrow in on the important points. If this is something
that you find yourself struggling with, feel free to drop by office hours to discuss the readings
and strategies that can be used to more efficiently gather the important points.

Q: What are office hours and why are they useful?

A: Office hours are specified times that the professor (I.e. me) has set aside to commit to 1-on-1
and small group meetings outside of lecture. Office hours typically do not require an
appointment and are windows where you are free to come and go as you please.

The most common purpose of office hours is to ask the professor questions about the material
and to receive more personalized answers and instruction. However, office hours can also be
used to just chat with the professor, discuss other related work you’re doing/interested in, etc. I
will be available during my office hours (either in my office or online) even if no students are
there, so there’s no reason to not take advantage of this time!

In many ways, office hours is a bit of a misnomer – it would be better if they were called
“Student Hours” because they are windows of my schedule that are set aside for you! 1

Q: I’ve fallen behind on the work for QTM 110 and I’ve done poorly on the midterms. Is there
any way that I can do an extra credit assignment?

A: Unfortunately, it isn’t fair to offer extra credit to any subset of students. I believe that the
grade scheme for this course gives ample opportunities for a student who stays on top of the
work for this course, attends lectures, and studies for the exams to add points to their final
grade. Therefore, I don’t intend to add extra points to the already specified rubric.

Q: I like courses where questions on exams have one clear answer. I feel like this class presents
ideas and methods that only lead to more questions and it can be difficult to write answers that
fully address a question. Is there any approach I can take that will ensure that I get full points for
homework/exam questions?

A: This class is about critical thinking and logic. Given a set of loose rules about what a good
scientific answer looks like, we will reason through why certain concepts like causality are
desirable. However, there is no silver bullet for the scientific process – every choice comes with
a set of pros and cons. As such, it is important to be able to compare and contrast approaches
and think about cases where the stuff we discuss in class won’t even begin to fully answer a
question of interest!

This said, there are often many different answers to questions that are acceptable. However,
some are better than others and more effectively engage with the material that has been
presented in the course. The best way to receive full credit for answers is to engage the course
material thoroughly and note any strengths and weaknesses of your proposed answer.
Transparency is key!

August 2021
SUNDAY MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY
1 2 3 4 5 6 7

8 9 10 11 12 13 14

15 16 17 18 19 20 21

22 23 24 25 26 27 28
First meeting;
Introductory
Comments;
Welcome
Quiz
Posted

29 30 31
Correlation;
Welcome
Quiz
Due

September 2021
SUNDAY MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY

1 2 3 4
Correlation to
Causation

5 6 7 8 9 10 11
Labor Day; Quiz #1 Comparing
No Class Posted; Apples to
Group Apples
Assignments
Posted

12 13 14 15 16 17 18
Common Confounding
Mistakes and and the
ABC (Always Fundamental
Be Problem of
 Comparing); Causal
Quiz #1 Due Inference
Problem Set
#1

Posted

19 20 21 22 23 24 25
Bias, Precision, Average
and Averages: Treatment
An Important Effects and
Equation; How
Problem Set #1 Randomization
Due; Solves
Problems
Quiz #2 Posted

26 27 28 29 30
Experiments Internal
and Inference Validity,
– How can I External
say that a Validity, and
treatment Chance
works with Imbalance
confidence?;
Quiz #2 Due;
Problem Set #2
Posted

October 2021
SUNDAY MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY

1 2

3 4 5 6 7 8 9
Experimental Review for
Ethics and Midterm #1
Catch-Up
Problem Set #2
Du
e
 10 11 12 13 14 15 16
Fall Break; Midterm #1
No Class

17 18 19 20 21 22 23
Confounding More on
Revisited; Regression and
Observational Matching
Data;
Intro to
Regression

24 25 26 27 28 29 30
Time Series More on DID
and Diff-inDiff; and intro to
Quiz #3 Posted RDD

31

November 2021
SUNDAY MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY

1 2 3 4 5 6
Regression Instrumental
Discontinuity Variables and
Designs; Natural
Quiz #3 Due; Experiments
Problem Set #3
Posted

7 8 9 10 11 12 13
Bayes’ Theorem The Base Rate
and Prior Fallacy and
Beliefs; Other Bayesian
Problem Set #3 Oddities
Du
 e

14 15 16 17 18 19 20
Bayes’ Theorem Midterm #2
and Prior Beliefs

21 22 23 24 25 26 27
Comparing No class
Prediction and
Causal
Inference;
Quiz #4 Posted

28 29 30
More Prediction
and Bridging the
Gap between
Prediction and
Causal
Inference;
Problem Set #4
Posted

December 2021
SUNDAY MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY

1 2 3 4
P-Hacking and
Multiple
Comparisons;
Quiz #4 Due

5 6 7 8 9 10 11
Science in the Optional Final Final Exam
Age of Social Exam Review from 3:00 PM
Media; Session; to 5:30 PM
Last Class; Time TBA
Problem Set #4
Du
e
12 13 14 15 16 17 18

19 20 21 22 23 24 25

26 27 28 29 30 31