Step 1: Identifying a problem

Over the past decade, many developing countries have expanded primary school access, but
improvements in school access and enrollment may not always translate into improved learning
outcomes for all students if the quality of education is poor.
Ironically, the difficulty in improving the quality of education may be complicated by success in
getting more children to attend school, as in many cases neither the pedagogy nor the curriculum
has been adapted to take into account the quantity and characteristics of the influx of new
children.
Step 2: Arriving at a solution
Precipitating and perpetuating factors, whether there are previous interventions that have
addressed the problem + work along the guidelines of the same. If not, independently design an
intervention program. Intervention hypothesis

 Schools do not have enough resources and often have inappropriate curricula.
 There are too few teachers and some are poorly trained.
 There are also too few classrooms, teaching materials, textbooks, notebooks, and pencils.
 The curricula are often not adapted to the lack of resources or to the local context.
 Teachers are often absent or make little effort when present. A countrywide survey found
that one quarter of all public primary school teachers were absent from school on any
given day and that only half of those present were teaching.
 As more children enroll, pupil-teacher ratios worsen and teachers cannot give extra
attention to pupils who may need it to follow the lesson.
 What’s more, when the class size is larger, more of the teacher’s attention has to be spent
on ancillary classroom issues, such as discipline and simply getting the pupils
coordinated and focused.
 Not only are the classes large, but they also often include students of varying
achievement or even grade levels. This makes it even more difficult to adapt the material
and the pace to the learning needs of the pupils. The less-prepared pupils may need
different instruction or a slower pace or even remedial education. But if the teacher
focuses on their needs, the more-prepared students would be learning less.
 Low school quality often translates into poor learning. In Mumbai, 25 percent of children
in grades 3 and 4 in public schools cannot recognize letters, and 35 percent cannot
recognize basic numbers; in Vadodara, only 19.5 percent of grade 3 students can
correctly answer questions testing grade 1 math competencies.
 And a nationwide survey found that 44 percent of the in-school children aged 7 to 12
cannot read a basic paragraph and 50 percent cannot do simple subtraction.

Step 3: Setting goals and designing the intervention

Program activities, objectives (short term), goals (long term), what objective will the program
activity help meet?
The objective of the program was to improve basic math and reading competence. In particular,
Pratham (the NGO behind the Balsakhi program) wanted to make sure the program led to
improvements in basic number recognition, counting, ordering one- and two-digit numbers, and
solving basic word problems. Pratham also wanted to learn as much as possible about the
channels through which the program achieves its impact. They also wanted to help at-risk
children acquire the basic skills they need to participate fully in the classroom.
The decision to take part in the program was left up to the schools. There was a pilot research
done that showed that there was a general need for remedial education in the municipal schools.
Thus, a general consensus emerged among the stakeholders that all schools had to receive
balsakhis during the evaluation period.
The program provided tutors for at-risk children in government schools. The tutor, called a
balsakhi, or “child’s friend,” was typically a young woman hired from the local community.
Balsakhis were paid between 500 and 750 rupees a month. All the balsakhis had completed at
least secondary school, and they were given two weeks’ training at the beginning of the school
year.
The program would be targeting 122 public primary schools in Vadodara and 77 schools in
Mumbai.
There was also some concern that schools would only be willing to take part in the evaluation—
for example, allow Pratham to conduct achievement tests in the school—if they received some
program assistance. Designing the evaluation considering the opportunities and the constraints
whatever evaluation design was adopted, it had to ensure that all schools in the sample received
the program and that somehow half the sample would be a comparison group not receiving the
program.
The program targeted children who had reached grades 3 and 4 without mastering grades 1 and 2
reading and math competencies, including spelling simple words, reading simple paragraphs,
recognizing numbers, counting up to 20, and subtracting or adding single-digit numbers.
Children who were lagging behind—identified as such by the teacher—were pulled out of the
regular class in groups of 20 and sent for remedial tutoring, spending half the school day with the
tutor. Tutoring followed a curriculum designed by Pratham to help the children acquire the
grades 1 and 2 skills they needed to follow their regular lessons. But because the 20 pupils are
pulled out of the regular classroom, the program could have two other potential effects. Pulling
out the children created two classes, each smaller than the original. So, for half of the school day,
the class size was reduced. Pulling out the weakest children created two streams, each with
children of comparable achievement.
This amounted to tracking: For half the school day, a child in the regular class (the higher-ability
track) temporarily had peers at an equal or more advanced learning level. Therefore, the impact
of the program, if any, could come through one or more of the following channels: the remedial
instruction delivered by the balsakhi, the reduction in class size, and the ability tracking.

In the 2001 school year in Vadodara, approximately half of the schools were given a tutor for
grade 3, and the other half were given a tutor for grade 4, while in Mumbai during that same
year, approximately half of the schools received a tutor for grade 3, and the other half received a
tutor for grade 2 in both cities, which school received which tutor was randomized.

In 2002, the schools were given a tutor for the previously untreated grade. In determining
program impact, grade 3 students in schools that only received a tutor for grade 4 were compared
to grade 3 students in schools that had tutors for grade 3, and so on. Academic achievement was
measured through two annual tests, administered at the start and end of the academic term.

Pratham identifies the following features as key to the design of the Balsakhi program.

 In a small class, the balsakhi can provide more individualized attention, and as a member of the
local community, the balsakhi is more familiar with and socially linked to the children.

 Removing children from the classroom benefits nontargeted children by reducing the effective
student teacher ratio and by allowing the school instructor to proceed to more advanced topics.

 An effective balsakhi will eventually allow targeted children to return to the mainstream
classroom with reinforced basic literacy and numeracy skills.

 The program is easily replicated. Balsakhis are paid roughly $10/month, are recruited locally,
and require relatively little training. Balsakhis also adapt to local space constraints, so there is
low overhead and capital costs.

 The balsakhi turnover rate is high (on average a one year stay), so it is unlikely that the
program’s success depends largely on the ability of a few enthusiastic individuals.

 There is existing evidence that official teachers appreciate the extra help from the balsakhi in
reducing class size and helping out with some other basic administrative tasks at the school.
Furthermore, because of the high turnover rate and relatively low level of training, there is little
threat that they will take over the official teacher’s job.

Step 4: Implementing the intervention

Implementation of the activities, degree of effectiveness can be evaluated
Researchers expected two possible effects of the balsakhi program on schooling: improved
attendance and increased test scores. Removing remedial students for part of the day has two
potential effects: (1) Lower achieving students are given closer, individualized attention from a
local village resident and (2) Students taught by non-balsakhi teachers benefit from a smaller
class size for a portion of the day and the ability of teachers to focus on more advanced material,
thereby encouraging attendance of higher achieving students.
To test for any educational benefit from the balsakhi in the Vadodara sample, Pratham developed
a two-part exam that tested math and language skills separately. This exam covered different
skills that the Vadodara Municipal Corporation designated as “compulsory” for each of the grade
levels. For example, the math skills test covered topics ranging from basic number recognition,
counting, and ordering of single and double-digit numbers to basic addition and word problems.
Similar exams were administered in Mumbai. Tests were administered to all students in the study
schools in the grades of interest both at the beginning, middle and end of the school term. This
allowed the impact evaluation to focus on improvements rather than the level of performance.
Actual Impacts of the intervention:

Impact on Education: The program had substantial positive impacts on children' s academic

achievement. In both Vadodara and Mumbai, the Balsakhi program significantly improved
overall test scores; by 0.14 standard deviations in the first year and 0.28 standard deviations in
the second year, with the largest gains in math. Moreover, the weakest students, who were the
primary target of the program, gained the most. Researchers estimate that the entire effect of the
program was due to a very large (0.6 standard deviations) improvement in average test scores
among the children who were sent for remedial education. In contrast, there was no measurable
impact for their classroom peers, who did not receive remedial tutoring, but were "treated" with
smaller class sizes and a more homogenous classroom.

Balsakhi Turnover: There was rapid turnover among the Balsakhi tutors, with each tutor staying
on average for just one year, typically until they got married or got another job. Despite the high
turnover among tutors, the program still resulted in significant gains in student learning, which
suggests that the success of the program did not depend on a handful of very determined and
enthusiastic individuals.

Cost-Effectiveness: The Balsakhi program was very inexpensive, since the main cost of the
program was the tutors' relatively small salaries. Overall, the Balsakhi program cost
approximately US$2.25 per child per year, significantly less than the cost per child of a
Computer Assisted Learning program, evaluated by Pratham at the same time. In terms of cost
per improvement in test scores, researchers estimate an attractive cost-effectiveness of about
US$0.67 per standard deviation increase in test scores.1 The Balsakhi program has since been
adapted, re-evaluated, and scaled up across India.
In J-PAL's comparative cost-effectiveness analyses, the Balsakhi program led to a 3.07 standard
deviation improvement in test scores and did not have an impact on years of education per $100
spent. Reducing class sizes did not have an impact on test scores. 

Research Design:
Given this design, in each year, children in grade 3 in schools that received the program for
grade 4 form the comparison group for children that receive the program for grade 3, and vice
versa. While the assignment strategy ensures treatment and comparison groups are comparable,
the estimates of the program effect would be biased downwards if the schools reassigned
resources from one grade to the other in response to the program. In practice, the way schools are
organized in urban India (and, in particular, in Vadodara and Mumbai) makes this extremely
unlikely: schools have a fixed number of classes (a group of students and a teacher) per grade.
All students are automatically promoted so that the principals have no discretion in the number
of students per class or the number of teachers per grade. Most schools have just enough
classrooms for each class, and in Vadodara the balsakhi class typically met outside or in a
hallway. Teachers were assigned to classes before the program was implemented, and we
observed no instance of subsequent reassignment to a different standard. There are essentially no
other resources to speak of that the head teacher could allocate to the grade that did not receive
the balsakhi. Thus, we are confident that there was no reallocation of resources to the grade that
did not receive the balsakhi, which makes these students a good comparison group. Note that this
design allows us to estimate both one-year and two-year effects of the program, since a child
entering grade 3 in a school where the program was offered in grade 3 in year 1 (Group A
school) would remain in the treatment group in the second year, when in grade 4.
Estimates of the impact: