Beyond Theory of Change

Tips and Strategies for Collecting, Aggregating, and Reporting PM Outcomes

SW National Service Training Conference July 24, 2013 Susan Hyatt, Core Thought www.core-thought.com

Session Description
Knowing WHAT data to collect to demonstrate the

change your program creates is important.

However, data collection, aggregation and reporting tools

and processes are critical to insure you have valid and reliable data on which to base your findings either makes or breaks your efforts.
This session will provide you the step-wise information to

help you insure your programs current performance measurement approach is in alignment with your theory of change and can yield the information you need to clearly and confidently show your outcomes.

Learning Objectives
By the end of the session, you will:

Know the four types of outcomes and identify your options

for collecting data based on the outcome to be measured;

Be able to identify the most appropriate data source and

cost effective data collection methodology to use to measure your intended outcome; and

Know strategies for aggregating and summarizing

performance measurement data to make powerful statements that make your program more attractive for grants and other resources.

Making the Case for Support

CNCS funds program interventions

with a high likelihood of creating an intended change in a community need

o Evidence-based interventions: Will

the proposed intervention (design, dosage) likely lead to the desired outcome?
o Whats the cause and effect?

o What to measure to demonstrate

change?

Measuring Program Performance (Outcomes)

TOC to Clarify Desired
Outcome What is Measure -ment Question ?

Who Would Know?

What Does Info Tell How to You? Tweak Collect Program Info? & Tell Then Do Your It! Story

What are Outputs? Bean Counting

Measure the AMOUNT of service provided (Did you do what you said?)
Number of people served (usually best)
Number of products/events/etc. created

What are Outcomes? The So What?

Measure the CHANGES or BENEFITS

that occur because of service provided

(individuals, organizations, communities, or the environment)
Increase - Decrease More - Less Something is different

Keep It Simple....You Cant Measure EVERYTHING

3rd Grade Reading Tutoring: Whats the Best Outcome to Measure? Can read at grade level

Graduates from HS

How Clear Are You About What to Measure?

Program Activities/Inte rvention Problem/Ne Theory of Change: ed Tool to Identify
Cause and Effect Relationship

Desired Outcome

What Are You Trying to Change? (K-ABC)

Knowledge Skill
Learning new information or how to do something

10

What Are You Trying to Change? (K-ABC)

Thoughts or feelings about something

11 Learning new information or how to do something

Attitude Belief

Knowledge Skill

What Are You Trying to Change? (K-ABC)

Thoughts or feelings about something

12 Learning new information or how to do something

Attitude Belief

Knowledge Skill

Actions, conduct, or habits

Behavior

What Are You Trying to Change? (K-ABC)

Thoughts or feelings about something

13 Learning new information or how to do something

Attitude Belief

Knowledge Skill

Actions, conduct, or habits

Behavior

Condition

Circumstance s or status

What to Measure? Teen Health Program

More willing to eat vegetables

14

Attitude Belief

Knowledge Skill

Increased knowledge of healthy food

More frequent exercise

Behavior

Condition

Improved physical condition

15

What Do These Questions Measure?Identifying

Outputs and Outcome Types

ACTIVITY

Now What?
You identified the desired change (outcome) through your theory of change.
Are you clear about WHAT to measure? AND

16

Are you measuring it the BEST way?

Asking the Right Question is Critical

Intervention: ABS driving school gives a 10 week course that meets twice a week for 60 minutes that includes classroom-based and on road lessons on driving skills Desired Outcome: Students have basic driving proficiency.

17

Asking the Right Question

Question: Do you like driving? Answer: I LOVE driving!!
Do you like driving? gets information about an attitude not skill level.

18

Asking the Right Question

Question: Are you a skilled driver? Answer: I think I am a GREAT driver!
Asking, Are you a skilled driver gets information about self perception, a thought not actual skill level.
Self-ratings are subjective NOT objective.

19

Asking the Right Question

Question: Do you know the Colorado driving laws? Answer: I got 100% correct on my written drivers test!
Knowing state driving laws reflects knowledge

20

not actual skill level even though it is objective.

Asking the Right Question

Question: Have you demonstrated you are a skilled driver? Answer: I passed my road test!
An on-road driving test DOES measure skill level or proficiency and is objective.

21

Qualitative Data
Describes or characterizes through words
Focuses on meaning, experience or attitudes Collected through focus groups, interviews, opened ended questionnaire items, and other less structured situations. Not the same as anecdotal Information

22

Qualitative Quality

Quantitative Data
Focus on numbers and frequencies.
Data which can be measured Length, height, area, volume, weight, speed, time, temperature, humidity, sound levels, cost, ages, etc.

23

Quantitative Quantity

Whats the Best Way to Ask the Question?

24

Qualitative
Please describe how you have been feeling lately.

Quantitative
Thinking about the past week, how depressed would you say you have been on a scale from 0 to 5, where 0 means "not at all" and 5 means "the most possible?

I feel great! I love my new job. Ive lost 20 pounds and feel healthier than I have in years.

25

Whats Your Outcome and Measurement Question? ACTIVITY

Whos Your Best Source of Information?

Data Source = The person, group or organization

26

that has information on whether the intended outcome occurred.

Can you access this data source? How?
What are the pros and cons of relying on this data

source?
What alternatives are available? What tradeoffs come

with relying on an alternative data source?

Once youve chosen an appropriate data source, make

sure youre asking them the right questions!

School Attachment Who Would Know?

Teachers
Students
(preferred data source)

27

Parents

Tutors

28

Who is Your Data Source? ACTIVITY

What Method Will You Use?

Data Collection Method: the process used to systematically collect data
Match method with: Measurement question for desired outcome Data source Resources (time, dollars, etc.)

29

Surveys
Types of Questions:
Open ended

30

Closed ended Scales Ranking Multiple Choice

Interviews & Focus Groups

31

Observations

32

Other Commonly Used Methods

Logs and Tally Sheets
Standardized Tests Program Developed Pre/Post Tests Rubrics Secondary Data Other?

33

Choosing or Designing an Instrument

Which ever method you select, what instrument will you use?
Borrow vs. Develop Does it ask the right questions to get at your
desired outcome?

34

Does it have all the necessary components? What information will each question yield? How will you use information, if not related to
outcome?

Instrument Mapping
Look at each question on your data collection tool and ask:

35

Does this help us measure the desired outcome?

Is there one question? More than one? Subjective? Objective? Quantitative? Qualitative?

What kind of data will we get?

If it doesnt measure the outcome, do we really need to ask

it? How will we use the answer? Nice to know or internal use? How will we analyze this?

36

Instrument Mapping ACTIVITY

Data Collection Steps

After identifying a data source, method and

37

instrument(s):
1.Identify who you need to work with to collect data 2.Set a schedule for collecting data 3.Train your data collectors 4.Implement data collection

Good practice is to pilot test your tool.

FOR BEST RESULTS make key decisions about how to implement data collection BEFORE your program begins!

Program PM Targets
Target describes amount of change you anticipate achieving. Based on indicator specific, measurable amount of change that will signify success

38

Apply target to your specific instrument(s) If instruments vary by site - which questions get at the desired outcome? Need to apply program target to data collection tools used by each site Need to report aggregate outcome for program - not each site separately

39

Data Aggregation and Reporting ACTIVITY

Summary of Key Points

Use your theory of change as a guide to outcome

40

measurement. The type of outcome you want to measure (attitude, knowledge, behavior, condition) influences your choice of data source, method, and instrument. Be very clear about the measurement question to ask Choose methods based on ability to measure your desired outcome and ease of use. Map your instrument Remember: Continuous Improvement NOT

Ways to Connect

41

Susan Hyatt
shyatt@core-thought.com 303-512-0808

@Susan_Hyatt http://www.facebook.com/CoreThoughtinc www.core-thought.com