The Journal of Development Studies

ISSN: 0022-0388 (Print) 1743-9140 (Online) Journal homepage: https://www.tandfonline.com/loi/fjds20

Note – Walking on Solid Ground: A Replication

Study of ‘Housing Health and Happiness’

Maria Pía Basurto, Ramiro Burga, José Luis Flor Toro & César Huaroto

To cite this article: Maria Pía Basurto, Ramiro Burga, José Luis Flor Toro & César Huaroto (2019)
Note – Walking on Solid Ground: A Replication Study of ‘Housing Health and Happiness’, The
Journal of Development Studies, 55:5, 1042-1046, DOI: 10.1080/00220388.2018.1506579

To link to this article: https://doi.org/10.1080/00220388.2018.1506579

Published online: 13 Sep 2018.

Submit your article to this journal

Article views: 511

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=fjds20
The Journal of Development Studies, 2019
Vol. 55, No. 5, 1042–1046, https://doi.org/10.1080/00220388.2018.1506579

Note – Walking on Solid Ground: A Replication

Study of ‘Housing Health and Happiness’
MARIA PÍA BASURTO*, RAMIRO BURGA**, JOSÉ LUIS FLOR TORO†
& CÉSAR HUAROTO††
*Department of Economics, University of California Santa Cruz, Santa Cruz, CA, USA, **Department of Economics,
University of Virginia, Charlottesville, VA, USA, †Department of Economics, Northwestern University, Evanston, IL, USA,
††
Instituto de Economía, Pontificia Universidad Católica de Chile, Santiago City, Chile

ABSTRACT This note summarises our replication study ‘Housing, Health, and Happiness’, henceforth
HHH2009, which constitutes an important paper in the literature of housing and slum upgrading. The original
authors conduct a quasi-experimental impact evaluation of ‘Piso Firme’, an intervention that replaced in-house
dirt floors with cement in Mexico. We conduct a Pure Replication (PR), a Measurement and Estimation Analysis
(MEA), and a Theory of Change Analysis (TCA). In our PR, we did not find any major discrepancy with the
original study. In the MEA, we generally find the results to be strongly robust to different types of alternative
analysis. Finally, in TCA we explore a dimension that was not reported on the published version of the study and
found that households with high initial levels of cement-floor coverage benefitted significantly less from Piso
Firme’s intervention. These findings are discussed in greater detail on International Initiative for Impact
Evaluation’s (3ie) working paper version.

1. Motivation
Cattaneo, Galiani, Gertler, Martinez, and Titiunik (2009), from now on HHH2009, constitutes an
important paper in the economic literature for three main reasons and it is thus a great candidate for a
replication study. First, the topic of the paper is of great relevance as it affects millions of people
worldwide. By 2014, over 880 million people in the developing world lived in an urban slum. In Latin
America and the Caribbean – the region where the study takes place – more than 100 million people
lived in urban slums in 2014 (UN – HABITAT, 2016). A house is considered a slum if it lacks access
to improved water, access to improved sanitation facilities, sufficient living area, durable housing
(quality of the building), and/or secure tenure. Inadequate housing is a multidimensional problem that
affects a significant proportion of the developing world.1
According to the 2012 Inter-American Development Bank Flagship Report, the Latin American
region would have to spend over $300 billion, or nearly 8 per cent of its gross domestic product, to
provide adequate housing for all its citizen. In particular, around 12 per cent of households in the
region consider that their main housing problem are poor quality materials, and about half considers
the floor as the main one (Inter-American Development Bank, 2012).
Second, HHH2009 constitutes an important milestone in the literature on the relation between
housing conditions and household wellbeing. Before HHH2009, this literature had been mainly
developed by the medical literature using cross-sectional relationships (see Turley, Saigh, Bhan,
Rehfuess, & Carter, 2013; for a literature review, and other relevant papers such as: Galiani, Gertler,
& Schargrodsky, 2005; Galiani, Gonzales-Rozada, & Schardgrodsky, 2009; Devoto, Duflo, Dupas,
Parienté, & Pons, 2012; Galiani et al., 2017). HHH2009 is one of the few papers that provides causal
evidence on the positive effects of low-cost housing interventions and it has been extensively cited.2

Correspondence Address: César Huaroto, Instituto de Economía, Pontificia Universidad Católica de Chile, Avda. Vicuña
Mackenna 4860, Macul, Santiago, Chile. Email: cahuaroto@uc.cl

© 2018 Informa UK Limited, trading as Taylor & Francis Group

 Walking on solid ground 1043

Third, and no less important, HHH2009 results had been used to inform policy directly. The
programme was scaled-up nationwide. According to a well-known media source, by 2012 the
programme had reached over 2.7 million households.3 In addition, HHH2009 results helped the non-
governmental organisation TECHO (previously known as Un Techo Para Mi País translated as roof
for my country), which builds low cost housing, raise additional funding and expand its operations
internationally.4 Currently, TECHO operates in 19 countries in Latin America and the Caribbean and
has administrative offices in the United States and London, England.5 This makes this study a great
candidate for replication.
Finally, the motivation for the MEA and TCA (Brown, Cameron, & Wood, 2014) sections lies in the
possibility of exploring alternative methods to check for robustness and conduct further analysis of the
dataset. Even though the original authors conducted a series of robustness checks, we find that there is
still room to further explore robustness and heterogeneous effects. Regarding robustness, we use
Multiple Imputation as an alternative to the authors’ choice for imputing variables. This method is
considered the most unbiased for imputation. In addition, we study the direct effects of cement
flooring (as opposed to the programme’s) on household wellbeing, using an instrumental-variables
(IV) approach. Then we also look for heterogeneous effects with respect to initial coverage of cement
flooring. It is important to mention that, at the time of conducting the replication, we were not aware
that the original authors had also discussed the IV estimates in a working paper version of the study
and had removed that section after a request during the review process at the American Economic
Journal: Economic Policy.
A detailed description on the decision to replicate this study can be found in our replication plan
(available on 3ie`s website).6

2. Results
The results can be categorised into three main parts: (i) Pure Replication, (ii) Measurement and
Estimation Analysis, and, (iii) Theory of Change.

2.1. Pure replication

The codes and raw data provided by the authors allowed us to replicate variable construction by
providing us their raw data and cleaning data codes to check for coding mistakes. We find some minor
errors that do not affect the main conclusions of the study. It is important to mention that the authors
had already made their dataset available on the journal webpage, but we requested access to the raw
data and cleaning codes to complete the PR analysis. We also replicated all analyses the authors did
with secondary data that we gather ourselves from the Mexican National Statistics Office.
Overall, our pure replication analysis concludes that results in HHH2009 can be replicated. After
thorough inspection of the code, we only found three minor coding mistakes, which do not alter the
original author’s main results.

2.2. Measurement and estimation analysis

For the measurement and estimation analysis, we conduct three robustness checks that we described in
our replication plan prior to the beginning of the MEA analysis.
First, we use an alternative method for imputing missing values: multiple imputation (MI).
HHH2009 uses an imputation strategy known as Dummy Adjustment Imputation (DAI), this method
has been criticised in the literature because it can introduce bias to estimates and tends to increase
standard deviations of regression parameters (Enders, 2010; Jones, 1996; Schafer & Graham, 2002). In
addition, the authors use ‘Arithmetic Mean Imputation’ (AMI) for three variables: (i) per capita cash
transfers from government programmes, (ii) total per capita value of household assets, and (iii) total
1044 M. P. Basurto et al.

per capita consumption. In the literature, AMI has been found to also introduce bias in estimates and
overstate the estimates’ variance.
Instead, we use MI which is considered the most unbiased method (Enders, 2010; Jones, 1996;
Schafer & Graham 2002), and its impact on empirical results has been gaining interest among social
researchers (see Lall, 2016; Breitwieser & Wick, 2016; Arel-Bundock and Pelc, 2018, Romaniuk,
Patton, & Carlin, 2014; Sterne et al., 2009).
After thorough inspection, we found that the differences between results in HHH2009 and those
using MI are not sizable, which strongly supports robustness of the original results to an alternative
imputation method. However, the similarity of results using the different imputation methods is not
surprising ex post since the proportion of data that was imputed was not sizable (less than 1% of
values of most variables were imputed, and only one variable had about 12% of imputed values). It is
important to mention that, prior to this replication study, we could not have anticipated the scope of
missing values imputation as the publicly available dataset did not identify the imputations.
Second, we estimate an ordered multinomial model for outcomes measured in categorical variables.
Such a model can shed light on shifts in between the finer categories that explains most of the impact
found on the aggregated binary indicators. We use this alternative method instead of the ordinary least
squares estimation used by HHH2009, for which they group categories on variables such as maternal
satisfaction with housing characteristics and overall life quality to construct dummy variables instead.
In HHH2009, those variables were recoded into two categories: satisfied (1) and unsatisfied (0), while
we use the underlying four categories always ranging from 1 (best) to 4 (worst). The estimators for
treatment effects from an ordered probit and an ordered logit imply aggregate effects very similar to
those in the original paper. This supports the robustness of results found by HHH2009 with regard to
the specification with the dummy variables in the original author’s specification.
Finally, we use the dummy variable of offering households Piso Firme’s intervention as an
instrumental variable for in-house cement flooring in 2005 (post programme). This estimation
shows the impact of having rooms with cement floors on the outcomes of interest, as opposed to
the intent to treat estimates that shows the effects of being offered Piso Firme. The results from the
instrumental variables estimation are the same as in the working paper version of HHH2009 (Cattaneo
et al., 2007). Other than that, results are robust to inclusion of the initial share of rooms covered with
cement, which supports the specification chosen by the original authors not to include said variable in
the final regressions for the 2007 working paper.7
Overall, we find that results of HHH2009 do not change significantly with the three robustness
checks, therefore our evidence strongly supports robustness of the original study results.

2.3. Theory of change analysis

For the theory of change analysis, we explore heterogeneous effects of ‘Piso Firme’ due to differences
in the initial share of cement floor (pre-treatment) among the treated households. While some house-
holds might have gone from 0 per cent to 100 per cent share of cement-floored rooms, others might
have gained less marginal coverage from the programme, because of the difference in initial cement
coverage and the cap on how much cement flooring the programme granted (50 sq. mt.).
Unfortunately, there was no data available on the total area of the house or the area with cement
floor in year 2000, but only a variable with per cent share of rooms with cement floor in year 2000.
Therefore, we explore the heterogeneity by constructing a dummy to identify households above the
median on the per cent share of rooms with cement floor in year 2000, and interact this dummy with
the treatment variable to check for different treatment effects.
As expected, results suggest that households initially better off benefitted less from Piso Firme’s
intervention. First, we find that Piso Firme’s beneficial health effects on children from better-off
households were significantly lower for anaemia and the height-for-age z-score. For other measures of
children’s health, point estimates show smaller programme’s effects for the initially better-off, but said
differences are not statistically different. Second, we also find that the programme had smaller, but not
statistically significant, impacts on maternal mental health measures for initially better-off households.
 Walking on solid ground 1045

Using a back-of-the-envelope calculation, the programme’s effect on the initially better-off was
approximately half of what was reported by HHH2009 for both satisfaction with floor quality and the
Perceived-Stress Scale. By that same calculation, Piso Firme’s effect on satisfaction with house quality
and satisfaction with quality of life, as reported in HHH2009, is mainly driven by the initially worse-
off households, since the interaction term for the initially better-off is negative and cancels out the
average effect. Lastly, there are no significant differences for the initially better-off in terms of
Depression Scale.
Hence, results support the idea that Piso Firme’s effect was smaller for children and mothers from
households initially better-off in terms of the share of rooms with cement floor in year 2000 prior to
the programme. Better-off households, especially those with almost 100 per cent of rooms with cement
floors, may have received less cement as part of the programme and used it to cement less important
rooms in the household.

3. Conclusion
In our pure replication, we did not find any major discrepancy with the results shown in HHH2009.
Importantly, the minor coding issues we found were not part of the most important variables used for
the paper’s results. The rest of our work sought to check the robustness of HHH2009 to different
specifications, imputation methods and explore heterogeneous effects. We consider that this additional
analysis is useful to check the validity of results and further explore the programme’s effects. All the
analysis to be conducted in our replication study was prepared and published in our replication plan
before having access to the raw dataset and codes.
We systematically find results in HHH2009 to be robust to different strategies for dealing with
missing values, alternative specification, and estimation procedures, which included using an alter-
native IV specification which was used by the original authors in the working paper version of the
study. The heterogeneity analysis shows that households that were initially better off benefited less
from Piso Firme and thus the programme had a more sizable effect on initially worse-off households.

Acknowledgements
This replication study was funded and facilitated by 3ie’s replication programme. We are grateful to
the original authors for kindly sharing their codes, raw dataset, and methodological documents. We are
also thankful to the 3ie Replication Program staff, especially Benjamin Wood and Annette Brown, the
anonymous External Project Advisor, original authors and anonymous reviewers from this journal for
helpful feedback.

Disclosure statement
No potential conflict of interest was reported by the authors.

Funding
This work was supported by the International Initiative of Impact Evaluation Second Replication Window which
agreement and contract was signed on November 2013.

Notes
1. Replication draft: http://www.3ieimpact.org/media/filer_public/2015/09/16/rps_7_-study_on_piso_firmes_impact.pdf.
2. According to Google Scholar it has been cited 214 times and 40 according to Web of Science (May 2018).
3. Data from BBC post: http://www.bbc.com/mundo/noticias-38250971.
1046 M. P. Basurto et al.

4. http://cega.berkeley.edu/assets/cega_events/19/E2A_Cement_Floors_Brief.pdf.
5. https://www.techo.org/techo/que-es-techo/.
6. Replication Plan URL: http://www.3ieimpact.org/media/filer_public/2014/01/13/basurto_revised_replication_plan.pdf.
7. Working paper: https://openknowledge.worldbank.org/bitstream/handle/10986/7295/wps421401update1.pdf?sequence=
1&isAllowed=y.

References
Arel-Bundock, V., & Pelc, K. J. (2018). When can multiple imputation improve regression estimates? Political Analysis, 26(2),
240–245.
Breitwieser, A., & Wick, K. (2016). What we miss by missing data: Aid effectiveness revisited. World Development, 78(1),
554–571.
Brown, A., Cameron, C., & Wood, B. (2014). Quality evidence for policymaking: I’ll believe it when I see the replication.
Journal of Development Effectiveness, 6, 215–235.
Cattaneo, M., Galiani, S., Gertler, P. J., Martinez, S., & Titiunik, R. (2007). Housing, health, and happiness. World Bank Policy
Research Working Paper No. 4214.
Cattaneo, M., Galiani, S., Gertler, P. J., Martinez, S., & Titiunik, R. (2009). Housing, health, and happiness. American Economic
Journal: Economic Policy, 1(1), 75–105. https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD010067.pub2/abstract
Devoto, F., Duflo, E., Dupas, P., Parienté, W., & Pons, V. (2012). Happiness on tap: Piped water adoption on urban morocco.
American Economic Journal: Economics Policy, 4, 68–99.
Enders, C. (2010). Applied missing data analysis. New York: The Guilford Press.
Galiani, S., Gertler, P., Cooper, R., Martinez, S., Ross, A., & Undurraga, R. (2017). Shelter from the storm: Upgrading housing
infrastructure in Latin America slums. Journal of Urban Economics, 98, 187–213.
Galiani, S., Gertler, P., & Schargrodsky, E. (2005). Water for life: The impact of the privatization of water services on child
mortality. Journal of Political Economy, 113, 83–120.
Galiani, S., Gonzales-Rozada, M., & Schardgrodsky, E. (2009). Water Expansion in Shantytowns: Health and Saving.
Economica, 76(304), 607–622.
Inter-American Development Bank. (2012). Room for development. Housing markets in Latin America and the Caribbean
(IADB Flagship Report). Washington, DC: IADB.
Jones, M. (1996). Indicator and stratification methods for missing explanatory variables in multiple linear regression. Journal of
the American Statistical Association, 91(433), 222–230.
Lall, R. (2016). How multiple imputation makes a difference. Political Analysis, 24, 414–433.
Romaniuk, H., Patton, G., & Carlin, J. (2014). Multiple imputation in a longitudinal cohort study: A case study of sensitivity to
imputation methods. American Journal of Epidemiology, 180(9), 920–932.
Schafer, J., & Graham, J. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147–177.
Sterne, J., White, I., Carlin, J., Spratt, M., Royston, P., Kenward, M., … Carpenter, J. (2009). Multiple imputation for missing
data in epidemiological and clinical research: Potential and pitfalls. British Medical Journal, 338, b2393.
Turley, R., Saigh, R., Bhan, N., Rehfuess, E., & Carter, B. (2013). Slum upgrading strategies involving physical environment and
infrastructure interventions and their effects on health and socio-economic outcomes (Review). Cochrane Database of
Systematic Reviews, Issue 1. Art. No.: CD010067. Retrieved from https://www.cochranelibrary.com/cdsr/doi/10.1002/
14651858.CD010067.pub2/abstract
UN – HABITAT. (2016). World cities report 2016: Urbanization and development – emerging futures. Nairobi, Malta:
Gutenberg Press Ltd.