Data Collection In the Home

A TEI Toolkit
Module 3: Collecting High-Quality Data

Tool 3.11 Inter-Rater Agreement Tool is a Modifiable Excel Template With Instructions
template for calculating and reviewing agreement between raters for observational too

The development of this toolkit was funded by the Office of Planning, Research and Evaluation (OPRE) at the Admin
Contract Number HHSP23320095644WC. ACF is a division of the U.S. Department of Health and Human Services. In
Evaluation Institute (TEI) to provide   guidance, leadership, and support to promote excellence in community-based
serve American Indian/Alaska Native (AIAN) children and families through the Tribal Maternal, Infant, and Early Chil
The views expressed in this publication do not necessarily reflect the views or policies of the Office of Planning, Rese
and Families, or U.S. Department of Health and Human Services. This toolkit was developed in response to needs an
emerged through TEI’s work with Tribal MIECHV grantees. TEI members from two institutions collaborated to creat
and Johns Hopkins Center for American Indian Health.
 AGREEMENT
Sub themes RATER A RATER B (1 = yes; 0 = no)
Question 1 Agree Agree 1
Question 2 Agree Agree 1
Question 3 Agree Agree 1
Question 4 Agree Agree 1
Question 5 Agree Disagree 0
Question 6 Agree Agree 1
Question 7 Agree Agree 1
Question 8 Agree Agree 1
Question 9 Agree Agree 1
Question 10 Agree Agree 1
Question 11 Agree Agree 1
Question 12 Agree Agree 1
Question 13 Agree Agree 1
Question 14 Agree Agree 1
Question 15 Agree Agree 1
Question 16 Agree Agree 1
Question 17 Disagree Disagree 0
Question 18 Agree Agree 1
Question 19 Agree Agree 1
Question 20 Agree Agree 1
Question 21 Agree Agree 1
Question 22 Disagree Agree 0
Question 23 Agree Agree 1
Question 24 Agree Agree 1
Question 25 Agree Agree 1
 INTRUCTIONS

Step 1: Replace the Question # place holders with the actual

question headers from your instrument.

Step 2: Count the total number of questions you have and put
that number is cell G23 (in the example text there are 18
questions).

Step 3: Enter in the actual scores per question for each rater.

Step 4: Enter in a 1 when the Raters agree and a 0 when they do

not in column D. (Agreement can be defined as matching exactly
for some measures or as being within a given range for others.
Make sure to follow the guidelines from your data collection
instrument).

Step 5: Review your interrater reliability in G24 and discuss.

Agreement rates of 80% or better are desireable. Reconcile
together questions where there were disagreements.

Sum of Agreement Rating 22

Total number of possible questions: 25
Agreement 88%
General Goal is to have 80% agreement
 AGREEMENT
PARTICIPANT RATER 1 RATER 2
(1 = yes; 0 = no)
Parent speaks in a warm tone of voice 1 1 1
Parent smiles at child 2 2 1
Parent praises child 0 0 1
Parent is physically close to child 0 1 0
Parent uses postitive expressions with child 1 2 0
Parent is engaged in interacting with child 2 1 0
Parent shows emotional support toward child 1 1 1

4
This example uses questions from the PICCOLO observational measure
of developmental parenting. Agreement here is established as one to
one correlation so the numbers being the same. You will see below that

Sum of Agreement Rating 4

Total number of possible questions: 7
Agreement 57%
General Goal is to have 80% agreement