MONITORING AND EVALUATION: CALCULATING

AND INTERPRETING COVERAGE INDICATORS BY DR JOHN P OYORE

Learning Objectives
By the end of the session, participants will be able to:

Identify sources of data for calculating coverage indicators Estimate denominators for routine coverage estimates Calculate and interpret coverage indicators from routine data Use online resources for estimating coverage indicators Assess the quality of relevant data sources Reconcile coverage estimates from different data sources

Maternal Health Coverage Indicators

Proportion of pregnant women who received at least two antenatal care visits Proportion of deliveries occurring in a health facility Proportion of deliveries with skilled attendant at birth Proportion of women attended at least once during postpartum period (42 days after delivery) by skilled health personnel for reasons related to childbirth

Why Coverage Indicators Are Important

Understand how effective program is See if one target group is reached more effectively than another Identify underserved area/regions

Child Health Coverage Indicators

Immunization Programs
DTP3

vaccine coverage Measles vaccine coverage BCG vaccine coverage OPV3 coverage HepB3 coverage Fully immunized child

Nutrition programs? Control of diarrheal disease programs?

Coverage Indicators for HIV/AIDS Care & Treatment Programs

Number of clients receiving public/NGO VCT services Number of clients provided with ARVs Percent of children in need receiving cotrimoxazole prophylaxis Percent of HIV patients receiving DOTS Coverage of PMTCT programs?

Where Do We Get the Data?

Censuses Surveys Registrations Health management information systems Program statistics Patient registers

ESTIMATING COVERAGE FROM ROUTINE DATA

Indicators From Program Statistics: Numerators

HMIS and routine reports give information on numerators Numerators: number of deliveries in health facilities, measles vaccinations, pills distributed, voluntary counseling and testing clients etc. Denominators: ?

Example: Importance of denominator

Town A vaccinated 200 infants

Town B vaccinated 400 infants

Town C vaccinated 600 infants

Population size: Town A= 10,000 Town B= 30,000 Town C= 60,000

Indicators From Program Statistics: What Denominators Are Needed?

Denominators: population composition

Population

composition How many women are of childbearing ages? How many children are under five? How many adolescents? 15-19? 20-24? How many men are 15-59 years? How many children are of school going age? How many infants are there? How many babies are born each year?

How Do We Get Denominators?

Population registers Censuses Population projections Population growth rate (r) Rate of natural increase = crude birth rate (CBR) minus the crude death rate (CDR) Net migration rate: inmigration - outmigrants per 1000 population CBR: no. of births per 1000 population in 1 year CDR: no. of deaths per 1000 population in 1 yr Population growth = rate of natural increase + net migration rate

Spectrum Model

DemProj: projects population of country/region by age and sex based on assumptions about fertility, mortality, and migration
Urban

and rural population projections can also be prepared

EasyProj: supplies data needed to make a population projection from estimates provided by the Population Division of the UN www.tfgi.com

Spectrum

Calculating Denominators

Population at time t: P(t) = P(0) * exp(r*t), where:

P(t)

is the population size after t years P(0) is the population size at the last census

Example:
300,000

people at census Growth rate = 3% (0.03), What is the population after 10 years? 404,958 people

Estimating Number of Live Births

Where data on the number of live births are unavailable:

Total expected births = Total population x crude birth rate

Where the crude birth rate (CBR) is unknown:

Total expected births = Total population x 0.035

Estimating Number of Surviving Infants

Target population for childhood immunization:

Surviving infants <12 months of age in a year

Where data on the number of surviving infants are unavailable:

Total expected number of surviving infants = Total population x CBR x (1 infant mortality rate)

Estimating Number of Surviving Infants: CBR Known

Total population: 5,500,000 CBR: 30/1000 Infant mortality rate (IMR): 80/1000 Number of surviving infants = Total population x CBR x (1 IMR) = 5,500,000 x 30/1000 x (1 - 0.080) = 5,500,000 x 0.030 x 0.920 = 151,800

Estimating Number of Surviving Infants: CBR Unknown

Where data on the number of surviving infants, CBR or IMR are unavailable, multiply total population by 4%:

Expected no. of surviving children < 12 months = Total population x .04

If the total population is 30,000, then the number of children under one year = 30,000 x 4/100 = 1200
Source: WHO, 2002

Estimating the Monthly Target Population

Monitoring immunization and vitamin A coverage should be done monthly at the facility and district levels, requiring estimations of the monthly target population Monthly target population = Estimated number of children under 1 year of age divided by 12 Example: Annual target population of children < 12 months = 1200 Monthly target = 1200/12 = 100

Example: Immunization Coverage From Routine Data

Total population of district in 1990 = 99,000 CBR = 40 per thousand IMR = 80 per thousand Population growth (r) = 3% per year 3,000 measles vaccinations were given to infants in district in 1998 What is the measles coverage rate for 1998? Numerator: No. immunized by 12 months in a given year Denominator: Total no. of surviving infants < 12 months in same year

Immunization Coverage From Routine Data: Answer

Estimate district total population in 1998 Pop1998 = 99,000 * exp(.03*8) = 125,410 Estimate number of surviving infants in 1998 125,410 x (40/1000) x (1 - .080) = 4615 Estimate measles coverage rate Measles coverage = 3000/4615 x 100 = 65%

Case Study 1: Immunization Coverage from Facility Data

Estimate total population in 2003 Calculate coverage for DTP1, DPT3, and measles vaccine in 2003 Evaluate trends in coverage Estimate drop-out rates Analyze the problems in 2003 Is coverage low or falling? What are possible causes? What are the differences in coverage in different areas? What action can managers take if coverage data indicate problems?

Challenges in Estimating Coverage from Routine Data

Limited knowledge of target pop/denominators Low timeliness & completeness of reporting Poor data quality
Lack

of written standard reporting procedures No systematic supervision on data management

Dual reporting systems (EPI, HMIS) Inclusion of data from private sector

Assessing Reliability of Routine Coverage Indicators

Understand how denominators are derived Understand the process of collecting the information Look for inconsistencies and surprises

Assessing Reliability of Routine Coverage Indicators

Look for reliable data from other sources to use as a basis for comparison Cross-check

ESTIMATING COVERAGE FROM SURVEY DATA

Survey Tools for Coverage Estimation

WHO-EPI surveys Lot quality coverage surveys Large-scale population-based surveys

USAID

Demographic and Health Surveys UNICEF Multiple Indicator Cluster Survey Arab League PAPCHILD surveys CDC Reproductive Health Surveys

Seventy-five household survey Knowledge-Practice-Coverage Surveys Other local surveys

How Do Administrative Data Compare With Survey Data?

100 90 80 70 60 50 40 30 20 10 0
Nairobi Central Coast Eastern N/ Eastern Nyanza Rift Valley Western

Percent immunized

Survey (2002)

Routine Cumm Sep 2002

Reconciling Coverage Estimates From Different Data Sources

Age group & geographic scope Health cards versus recall Different sources for different purposes Not all coverage data can be compared in constructive way Differences in inclusion of private sector Selectivity

On-line Resource: STATcompiler

Innovative online database tool Allows users to select numerous countries and hundreds of indicators to create customized tables that serve specific needs Accesses nearly all population and health indicators published in DHS final reports

http://www.measuredhs.com/statcompiler

STATcompiler

On-line Resource: DOLPHN

DOLPHN: Data Online for Population, Health and Nutrition Online statistical data resource Quick access to frequently used indicators from multiple sources, including: DHS, BUCEN, CDC, UNAIDS, UNESCO, UNICEF, World Bank, WHO

www.phnip.com/dolphn

Advantages and Disadvantages of Routine-based Coverage

Advantages

Provides information on more timely basis Makes use of data routinely collected Can be used to detect and correct problems in service delivery Denominator errors Poor quality reporting

Disadvantages

Advantages and Disadvantages of Survey-based Coverage

Advantages

Avoids problems with denominators Includes information from non-reporting facilities

Coverage survey has low precision Larger standard errors at sub-national levels Irregular and expensive Survey timing may affect coverage rates

Disadvantages

Case Study 2: Estimating Vitamin A Coverage

Calculate coverage from routine data Use tally sheets to determine number of children who received vitamin A compared to target population Compare coverage estimates from routine data with estimates from survey data Estimate missed opportunities

References
WHO. 1999a. Indicators to Monitor Maternal Health Goals: Report of a Technical Working Group, Geneva, 8-12 November 1993. Division of Family Health Geneva: WHO. WHO. 1999b. Reduction of Maternal Mortality: A Joint WHO, UNFPA, UNICEF, World Bank Statement. Geneva: WHO. WHO (2002) Increasing Immunization at the Health Facility Level. Geneva, Switzerland: World Health Organization