J-PAL Southeast Asia Webinar

Talk Show Script

“Preliminary Findings from the impact evaluation of BPNT”
March 31st 2021, 08.00 – 9.30 am WIB

Time Session

8.00- 8.05 Opening from J-PAL SEA Executive Director

8.05 - 8.30 Opening remarks

● PMK (5 mins)
● Bappenas (5 mins)
● Kemensos (5 mins)
● USAID (5 mins)
● DFAT (5 mins)

8.30 - 8.35 Introduction Video - The story of BPNT evaluation

8.35 - 9.05 SESSION 1 – Talk show with panelists

● Context of the BPNT program and evaluation design (by Principal
Investigators)
● Discussion of preliminary findings (by Principal Investigators)
● Relating findings to policy (by a representative from Bappenas)

9.05 -9.25 SESSION 2 - Q&A with Audience

All panelists respond to live audience question

9.25 - 9.30 Closing

Moderator Opening
● Beras Sejahtera or Rastra is Indonesia’s largest targeted social assistance program, which
covered 15.5 million beneficiaries in 2016. Initially, beneficiaries received 15 kg rice per month
at subsidized price under Subsidi Rastra. The program changed to Bansos Rastra whereby
beneficiaries will receive 10 kg of free rice per month in 2018.
● Now, to better assist the most vulnerable households, the Government is looking to reform the
delivery of the program. Instead of receiving assistance in the form of rice, beneficiaries will
instead receive a voucher amounting to 110k per month that they can redeem for rice and eggs.
The reformed program was called BPNT (Bantuan Pangan Non Tunai), which then is known as
Program Sembako since 2020.
● With this transition, perhaps what policymakers are concerned about is whether or not this new
program can better target the beneficiaries and improve their welfare - hence reducing the
overall poverty level.
● If BPNT could indeed do all of these, it is even more imperative for us to carry out the program,
especially with the Covid-19 taking place. The poorest segment of our society are now even
more vulnerable due to the threat of job loss, unemployment and reduction of working hours.
● To understand the impact of this transition, J-PAL affiliated researchers worked together with
the government to conduct randomized evaluation across 105 districts.
● Some of the findings from this study have been presented in the video, but we’ll go deeper into
these results and to understand the overall impact of the transition
● In this talk show, we will be speaking with Elan Satriawan, Sudarno Sumarto, Ben Olken, and
Rema Hanna, researchers who worked with the government in evaluating this program.
● We will also be speaking with Maliki, the Director for Alleviating Poverty and Development of
Social Welfare at Bappenas about how the findings from this evaluation can be used to inform
policy.
● At the end of this webinar, we will have a 25 minutes question and answer session.

Session I - Talkshow with panelists (30 minutes)

Part 1: Context of the study & research design (6 minutes)

Moderator transition: Context of the study & research design

● Before delving more into the result, it is important for us to first understand the background of
the study itself. We will start off by examining the larger context of this reform and how this
study was designed.

2 minutes Q1: (To pak elan/pak darno - In Indonesian) You are a (position) of TNP2K as well as one of the
Principal Investigators for this study. Can you tell us about the policy background for the transition to
BPNT and how this study began?
Answer pointers:
● Brief introduction into context of the transition: reason why the GoI sees this transition as an
important step in optimizing food assistance
● Role of TNP2K: How tnp2k plays a role in advising for this policy and the need for this evaluation
(its importance in informing future policy decisions)
 ● How the PIs of this evaluation came to work together with the GoI: story on how this PI team
was put together

2 minutes Q2: (To pak elan/pak darno - In Indonesian) This evaluation is very important in producing
evidence for future decision making. What exactly are the questions that you are exploring?
Answer pointers:
● What are the impacts of the transition on the program’s targeting? In social assistance
programs we are always concerned about whether assistance is going to the right beneficiaries,
ie. the households who are most vulnerable. There are many reasons why the transition to BPNT
can change targeting relative to Rastra, including the way assistance is delivered -- for example,
(this will be explored more further later) less intermediaries involved, strict beneficiary identity
verification -- and the nature of the assistance itself in the form of debit cards that is more
difficult to redistribute to those excluded from the beneficiary list compared to bags of rice.
● How much subsidized food is actually received by the beneficiaries? How is the quality? We of
course want to understand whether those who receive assistance are receiving more or less
compared to Rastra and whether the rice is of a different quality. The value of the BPNT
voucher, Rp 110k, is approximately similar to 10kg of rice, which is the amount of the Rastra
assistance. However, the mechanism of how beneficiaries under BPNT can receive a different
amount of subsidy compared to Rastra is similar to the first question, ie. looking at the way
assistance is delivered. On top of that, the quality of rice beneficiaries receive may also change
compared to Rastra because the e-warong where beneficiaries can redeem their assistance can
source their rice (and eggs as well) from the private market as opposed to exclusively from
BULOG.
● Do beneficiaries change their consumption because of the “flexibility” that the BPNT vouchers
give them to purchase eggs in addition to rice? Although we’ll see later that there is no
mechanical reason to expect this (ie. the amount of subsidy is already below what households
would normally spend on rice, therefore there should be no reason for them to switch their
purchase to eggs) , we do find interesting results.
● We will also share our calculation on the administrative cost of implementing BPNT compared
to Rastra.

2 minutes -Q3: (To Ben/Rema - in English) There are a lot of questions you explore in this research. How
was the evaluation designed to answer all of them and how did you ensure that the results are unbiased?
Answer pointers:
● First, to answer how we ensure rigorous results, we use a randomized evaluation method.
When conducting an impact evaluation the concern is often whether or not the results we see is
because of the program, in this case the transition from Rastra to BPNT, or whether they are
caused by other factors.
● [show chart] We have confidence in claiming causality in this evaluation because, of the 105
areas that were involved in this evaluation, they were randomly assigned to receive the BPNT
either in 2018 or in 2019. The areas that receive BPNT in 2018 is our treatment group (consisting
of 42 areas), and those that receive in 2019 is the comparison group (consisting of 63 areas).
Because assignments to these two groups are random, we know that the two groups are equal
to each other in all important measures of characteristics prior to the transition. Therefore, any
 difference between them that arise after the transition, is due to the transition and not any
prior differences. (Visual aid here)
● To ensure that we can answer all of the important questions, we utilized two datasets: the
UDB and the SUSENAS. The UDB is a database used by the Ministry of Social Affairs for the
targeting of both Rastra and BPNT constructed in 2015. The SUSENAS is a bi-annual national
sample survey on socio economic indicators.
● The government merged these two datasets using NIK, we then used the de-identified version
of this merged data to understand whether a household is likely to be eligible or not to receive
assistance (using information from the UDB) and whether they actually received assistance
(using information from SUSENAS). From here we can understand the program targeting, ie.
whether assistance is going to the poorest households. We then use more information from
SUSENAS to see whether those who did receive assistance changed their consumption patterns,
and their quality rating for the rice they received as assistance.
● We’d like to note the significant collaboration with the government on adding an extra set of
questions within SUSENAS on social assistance that enabled us to essentially conduct this
evaluation.

Part 2: Findings on targeting, quality of assistance, and poverty (6 minutes)

Moderator transition: Findings on targeting, quality of assistance, and poverty

● Now that we have learned about the framework of the study, we will proceed with the findings
● One of the main challenges of in-kind program is that beneficiaries tend to receive considerably
less food compared to what they are entitled to. This may be due to an ineffective delivery
system, or a monitoring process that is too lenient.
● On the other hand, BPNT is believed to be able to better target the beneficiaries - ensuring that
subsidies are indeed allocated to the most vulnerable households. However, the question now is
how.

3 minutes Q4: (To pak elan/pak darno - In Indonesian) As you previously explained, the first concern with
social assistance is always whether or not the poorest households are the ones actually receiving benefits.
Previous evaluation of in-kind assistance in the form of rice found that many non-poor households also
received rice and many beneficiaries weren’t receiving the full amount of assistance. Did this change with
the transition to BPNT?
Answer pointers:
● Yes. We found that under BPNT targeting improved because poorer households experienced
substantial gain in the total amount of subsidy received. Somewhat fewer households receive
assistance, but those that receive assistance receive substantially more, and usually the full
amount they were entitled to.
● [show chart] We found that under BPNT, the probability for a household to receive aid at all is
smaller compared to in-kind areas, but the vouchers did in fact concentrate aid to the targeted
groups because the probability of receiving aid decreased most for households who are not in
the bottom 30% of the population. A household with a PMT score of under 30, meaning
households who are in the bottom 30% of the population, are 16% less likely to receive
assistance compared to in-kind recipients, however, those who have a PMT score of over 30,
meaning those who are not in the bottom 30%, are 49% less likely to receive assistance.
 ● [show chart] On the other hand, those who did receive assistance, received substantially more
benefits compared to in-kind recipients. We found that conditional upon receiving assistance,
beneficiaries received 84 percent percent more in voucher areas than in in-kind areas on
average over the previous 4 months. Furthermore, recipients in voucher areas were more likely
to receive the full benefits. Combined, this resulted in a dramatic concentration of benefits
among the poor: poorer households (at baseline) as a group received 45 percent more
assistance in voucher areas than in in-kind districts.
● Hence, under the BPNT program, poorer households experienced substantial gains in the total
amount of subsidy received. This is more pronounced for those who correspond to the bottom
18 percent of the population. One the other hand, households in the middle of the distribution
received about the same amount of subsidy on average, and those without PMT scores
(households that the community saw were too wealthy) saw the amount of subsidy they receive
fall considerably.

1 minute Q5: (To pak elan/pak darno - In Indonesian) So BPNT does improve targeting by concentrating
aid among the poorest households. What is the possible mechanism for this improvement in targeting?
Answer pointers:
● In in-kind districts, assistance is subdivided into much smaller amounts, so that the typical
beneficiary receives only about 1/3 of the intended transfer size. The additional funds are
used to give many more households assistance, despite the fact that this is explicitly against the
stated rules. In the in-kind program areas, sacks of rice were brought to local government
warehouses, where they were then packaged into individual sized portions and delivered to
beneficiaries. This process is difficult for the central government to control.
● In voucher areas, this practice of subdividing aid is substantially less prevalent. Debit cards
were used for the voucher program, and each card is pre-printed with the name of the
beneficiary. Therefore it is more difficult to subdivide this, as compared to giving out sacks of
rice.

1 minute Q5: (To pak elan/pak darno - In Indonesian) The video in the beginning mentioned that the
quality of rice that beneficiaries received is higher in BPNT areas. Coupling this improvement in quality and
the flexibility that a voucher offers compared to in-kind transfers, are there any concerns that households
who are above the cutoff would try to obtain assistance as well?
Answer pointers:
● The poor quality of rice was a frequent complaint about the Raskin program. On the other
hand, the low quality of rice means that there could be more self-targeting in such in-kind
programs, as richer households opt themselves out.
● In BPNT districts, household received higher quality of rice - due to a more flexible sourcing
mechanism (i.e. agents may source their rice from BULOG, private market or a combination of
both) Through the SUSENAS survey, we find that households rated the voucher rice substantially
higher quality, by about 32 percent. This suggests that the quality-adjusted amount of subsidy
received by targeted beneficiaries in voucher areas is even higher. With this improvement in
quality and the flexibility to purchase eggs in addition to rice this should have made the program
more appealing to richer households as well.
● However, the better targeting in voucher areas occurred despite this. The administrative
benefits of being able to better enforce the targeting of the electronic transfers thus swamped
the self-targeting benefits of the poor quality of food in this context.
1 minute Q6: (To Pak Elan/Darno - In Indonesian) We’ve learned that poorer households are receiving
more benefits and the rice is also of a higher quality. What is the overall impact on poverty?
Answer pointers:
● To understand this poverty question, we compute whether each household is below the
official Government of Indonesia poverty line, which is set separately for rural and urban
households in each province.
● [show chart] We found that BPNT reduced poverty among poor and near-poor households,
with the greatest reduction found among the poorest households. The increase in the amount
of subsidy led to a 20% (4.3 percentage points) reduction in poverty for households with PMT
score <= 15, or the bottom 15% of the population. For the very poorest – those with PMT scores
0-5 at baseline, we see a 24 percent reduction in the share of households in poverty,

Part 3: Findings on consumption decisions, rice price, leakage and admin cost (6 minutes)

Moderator transition: Findings on consumption decisions, rice price, and leakage and admin cost
● Thank you very much to everyone for the detailed explanation on how BPNT can target the
beneficiaries more effectively, as well as the impact of the program to the quality of rice as well
as the poverty reduction among poor and near-poor households
● Looking at this, it is possible that switching to BPNT can be the key in delivering a more effective
social assistance program.
● However, before we draw conclusions, let’s now go into the next set of the findings, specifically
on how the transition can impact household consumption decisions, the price of rice and of
course, the administrative costs of delivering the program.

2 minutes Q7: (To Ben/Rema - In English) We now understand the flexibility that the vouchers offer did not
negate targeting enforcement by attracting richer households. However, for households who did receive
assistance, did they actually change their consumption?
Answer pointers:
● We found interesting results on this question because even though there is no mechanical
reason to expect households to change their consumption, that is in fact what happened.
Almost all households consume more rice than what the amount of assistance allows for, which
is 10 kgs of rice per month. Even with households with PMT score below 30, we find similar
patterns; in fact, only 3.3% of these households consume less than 10kgs of rice. With this
consumption pattern, we should have no reason to expect that households use their 110k
assistance to purchase eggs.
● However, we find that targeted voucher households consume more of their subsidy in the
form of eggs, about 33 grams per month of egg protein for households with PMT scores <= 30,
with no change in total consumption of rice.
● We might wonder whether the change in consumption of eggs comes from income effects
rather than “stickiness” from the voucher. We find that this is not the case since we don’t find
systematic increase in other food consumption. This suggests the results we find on egg
consumption really are coming from the fact that eggs, and only eggs, were added to the set of
goods that the transfer could be used on.
 ● This suggest that governments seeking to improve nutrition can potentially do so through the
set of foods they chose to include or not include in vouchers, even if, in fact, basic price theory
says that unconstrained households could undo these constraints through their other purchase
decisions
● As per 2020 BPNT has been transformed to “Program Sembako” that includes a broader set of
food with a variety of nutrition (eg. animal & plant-based proteins like fish, chicken, eggs &
beans, vegetables and fruits, and rice.

2 minutes Q8: (To Ben/Rema - In English) This voucher effectively reduces the supply of subsidized rice and
instead increases demand for private rice in shops. We might worry that the price of rice would increase, is
this what happened?
Answer pointers:
● We might worry about price increase for areas where the switch to vouchers would cause a
negative supply shock, which are areas where subsidized rice consists of a large share of the
overall supply of rice, or areas where supply is inelastic, which are areas that are remote.
● [show chart] First we look at the supply shock measure, which is the share of subsidized rice
over overall rice consumed within the district. For districts above the median in this supply
shock measure, we find no observable effect in price. For districts that are higher in this supply
shock measure (75th percentile or higher); we observe a larger effect in magnitude (Rp. 554
rather than Rp. 180), but it remains statistically insignificant and still relatively small in
magnitude (representing a 5.8 percent price increase.
● [show chart] Looking at district remoteness, In above median districts, the effect is positive
but small (1.6 percent) and insignificant. For those that are very remote (i.e, 75 percentile or
th

higher), we observe a Rp. 338 (3.6 percent) increase in voucher areas compared to in-kind
areas. However, even the 3.6 percent price increase observed in remote locations is not enough
to negate the benefits gained from the greater concentration of the program.

2 minutes Q9: (To Ben/Rema - In English) One of the key challenges of the in-kind program is the
complexity of the food distribution which makes it difficult to monitor. So much food can ‘fall off the truck’.
Can the voucher program reduce such leakage? And last but not least, is there any difference in the
administrative costs of running both programs?
● To examine whether vouchers reduced such leakage, we compare the actual subsidy received
by the households as measured by SUSENAS survey with the intended subsidy for each
district, we find that there is no significant change in leakage compared to in-kind transfers.
We obtained these results by also looking at whether market prices are inflated thus decreasing
the “real” assistance beneficiaries receive, and adjusting that market price for quality.
● In terms of administrative costs, yes - the cost of delivering the voucher program is much less
than the in-kind program. In general, the administrative cost for the in-kind program is
approximately 4.1 percent of the total benefits. We managed to gather this data by looking at
the operating cost from BULOG’s annual report. However, it is important to note that the
national government only facilitates delivery to the district or sub-district capital, and local
governments are required to cover the costs of their own pick-up and delivery.
● [show chart] On the other hand, the administrative costs of the voucher program is 0.74
percent of the total benefits. The program has two main costs: (1) the costs of printing the
debit card (IDR 12,500/person) - which we can assume will last for three years, since targeting is
conducted every three years; and (2) the monthly rental costs of the EDC machines for the
 agents. However, this is if we assume the agents did not use the machine for other transactions.
In reality, research shows that 77 percent of the agents already existed prior to the program,
and paid rent on the machines for other transactions as well.

Part 4: Findings to policy (9 minutes)

Moderator transition : Findings to policy

● Previously, we’ve learned in great detail the impact of transitioning from in-kind to voucher
program - especially on how it helps in better targeting the beneficiary, reducing the poverty
level among the poorest households, increasing the quality of rice, and lowering the
administrative costs associated with the program.
● Now we would like to hear more about how these findings can contribute to policy making
decisions. Moving on to our next speaker, we have Pak Maliki from Bappenas.

3 minutes Q10: (to Pak Maliki - in Indonesian) Pak Maliki, as the Director for Alleviating Poverty and
Development of Social Welfare, can you tell us more about Bappenas’ role in the BPNT program?
Answer pointers:
● Provide a brief explanation on how this study links with the work that Bappenas is doing
● Is there a particular target for this program in the next couple of years (i.e. in terms of number
of beneficiaries, value of vouchers)
● Is there anything in particular that could be improved in the current BPNT program
● What are the main challenges faced by the BPNT program (especially in relation to the current
Covid-19 pandemic)

3 minutes Q11: (to Pak Maliki - in Indonesian) Are there any particular findings from this study that you
find interesting?
Answer pointers
● Possible answers could be related to one of the findings: targeting the beneficiaries, reducing
poverty level, increasing quality of the rice, changing consumption decisions, reducing
administrative costs, or its impact to the price of rice

3 minutes Q12: (to Pak Maliki - in Indonesian) Looking at the results of this study, the transition from
Rastra to BPNT could indeed transform the lives of millions. In your opinion, what would be the potential
use of these preliminary findings for the government, especially related to policy making process?
Answer pointers:
● There is strong evidence on how BPNT could prove to be an effective social assistance program,
provide explanation on how to implement this evidence at the government level
● Provide explanation on how the result of the findings could transform policies related to social
assistance