Training materials for experimental research

Experimental impact evaluation using Stata

Coalition for Evidence-Based Policy “Social Programs That Work”

International Initiative for Impact Evaluation (3ie) 

J-Pal Poverty Action Lab

The Development Impact Evaluation (DIME) Initiative

• DIME Analytics
• DIME Wiki
• ietoolkit
• World Bank Group GitHub repositories
• Microdata Catalog Releases

This is a list of links to institutional sites rather than to technical training resources, these may include evidence bases or other general information on experimental research techniques.

Coalition for Evidence-Based Policy “Social Programs That Work”

Impact Evaluation Network (IEN)

Innovations for Poverty Action (IPA) 

Inter American Development Bank Office of Evaluation and Oversight (OVE)

International Initiative for Impact Evaluation (3ie)

J-Pal Poverty Action Lab

The What Works Clearinghouse (WWC) Registry of Randomized Controlled Trials (RCTs)

Millennium Challenge Corporation

Center for Effective Global Action: CEGA

Global Innovation Fund

Africa Evidence Network

• Acemoglu, D. (2010). Theory, General Equilibrium, and Political Economy in Development Economics. The Journal of Economic Perspectives, 17-32
• Angrist, J. D., & Pischke, J. S. (2008). Mostly harmless econometrics: An empiricist's companion. Princeton University Press
• Angrist, J., & Pischke, J. S. (2010). The credibility revolution in empirical economics: How better research design is taking the con out of econometrics (No. w15794). National Bureau of Economic Research
• Angrist, J. D., & Krueger, A. B. (1999). Empirical strategies in labor economics. Handbook of Labor Economics, 3, 1277-1366
• Banerjee, A. V., & Duflo, E. (2009). The Experimental Approach to Development Economics. Annual Review of Economics, 1, 151-78
• Blair, Cooper, Coppock, Humphreys (2018). Declaring and diagnosing research designs. DeclareDesign.org
  • DeclareDesignwebsite
  • Declaring and diagnosing research designs Development Impact Blog, World Bank
• Burtless, G. (1995). The case for randomized field trials in economic and policy research. The Journal of Economic Perspectives, 63-84
• Card, D., DellaVigna, S., & Malmendier, U. (2011). The Role of Theory in Field Experiments. The Journal of Economic Perspectives, 25(3), 39
• Deaton, A. (2010). Instruments, randomization, and learning about development. Journal of Economic Literature, 424-455
• Dhaliwal, I., & Tulloch, C. (2012). From research to policy: using evidence from impact evaluations to inform development policy. Journal of Development Effectiveness, 4(4), 515-536
• Duflo, E., Glennerster, R., & Kremer, M. (2007). Using randomization in development economics research: A toolkit.Handbook of development economics, 4, 3895-3962
• Glennerster, R., & Takavarasha, K. (2013). Running randomized evaluations: A practical guide. Princeton University Press
• Gerber, A. S., & Green, D. P. (2012). Field experiments: Design, analysis, and interpretation. WW Norton
• Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. (2011). Impact evaluation in practice. World Bank Publications
• Heckman, J. J. (1991). Randomization and Social Policy Evaluation (No. 0107).National Bureau of Economic Research, Inc
• Heckman, J. J. (1996). Randomization as an Instrumental Variable. The Review of Economics and Statistics, 78(2), 336-341
• Heckman, J. J., & Vytlacil, E. J. (2007). Econometric evaluation of social programs, part II: Using the marginal treatment effect to organize alternative econometric estimators to evaluate social programs, and to forecast their effects in new environments. Handbook of Econometrics, 6, 4875-5143
• Imbens, G. W., & Wooldridge, J. M. (2009). Recent Developments in the Econometrics of Program Evaluation. Journal of Economic Literature, 47(1), 5-86
• Kasy, M. (2013). Why experimenters should not randomize, and what they should do instead (No. 36154)
• Khandker, S. R., Koolwal, G. B., &Samad, H. A. (2010). Handbook on impact evaluation: quantitative methods and practices. World Bank Publications
• Krueger, A. B. (1999). Experimental Estimates of Education Production Functions. Quarterly Journal of Economics, 497-532.
• LaLonde, R. J. (1986). Evaluating the econometric evaluations of training programs with experimental data. The American Economic Review, 604-620
• Ravallion, M. (2009). Should the randomistas rule?. The Economists' Voice, 6(2)
• Rodrik, D. (2008). The New Development Economics: We Shall Experiment, but How Shall We Learn? Harvard University, John F. Kennedy School of Government
• Ravallion, M. (2007). Evaluating anti-poverty programs. Handbook of Development Economics, 4, 3787-3846
• Rubin, D. B. (1978). Bayesian inference for causal effects: The role of randomization. The Annals of Statistics, 34-58.
• Vasilaky, K. N. & Brock, J. M. (2019). Power(ful) Guidelines for Experimental Economists. European Bank for Reconstruction and Development (EBRD), Working Paper No. 239
• White, H., (2014). Current Challenges in Impact Evaluation. European Journal of Development Research (2014) 26, 18–30

• De Mel, S., McKenzie, D., & Woodruff, C. (2008). Returns to capital in microenterprises: evidence from a field experiment. The Quarterly Journal of Economics, 1329-1372
• Green, D. P., Ha, S. E., & Bullock, J. G. (2010). Enough already about “black box” experiments: Studying mediation is more difficult than most scholars suppose. The Annals of the American Academy of Political and Social Science, 628(1), 200-208
• Imai, K., Keele, L., Tingley, D., & Yamamoto, T. (2011). Unpacking the black box of causality: Learning about causal mechanisms from experimental and observational studies. American Political Science Review, 105(04), 765-789
• Ludwig, J., Kling, J. R., & Mullainathan, S. (2011). Mechanism Experiments and Policy Evaluations. The Journal of Economic Perspectives, 17-38

• Casey, K., Glennerster, R., & Miguel, E. (2011). Reshaping institutions: Evidence on aid impacts using a pre-analysis plan (No. w17012).National Bureau of Economic Research
• Humphreys, M., de la Sierra, R. S., & Van der Windt, P. (2013). Fishing, commitment, and communication: A proposal for comprehensive nonbinding research registration. Political Analysis, 21(1), 1-20
• Olken, B. A. (2015). Promises and Perils of Pre-Analysis Plans. The Journal of Economic Perspectives, 29(3), 61-80

• Abdulkadiroğlu, A., Angrist, J. D., Dynarski, S. M., Kane, T. J., & Pathak, P. A. (2011). Accountability and flexibility in public schools: Evidence from Boston's charters and pilots. The Quarterly Journal of Economics, 126(2), 699-748
• Gibson, J., McKenzie, D., &Stillman, S. (2013). Accounting for selectivity and duration-dependent heterogeneity when estimating the impact of emigration on incomes and poverty in sending areas. Economic Development and cultural change, 61(2), 247-28

• Aker, J. C., Boumnijel, R., McClelland, A., & Tierney, N. (2012). Zap it to me: The impacts of a mobile cash transfer program
• Chaubey, Y. P. (1993). Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment. Technometrics, 35(4), 450-451
• Gibson, J., McKenzie, D., & Stillman, S. (2011). The impacts of international migration on remaining household members: omnibus results from a migration lottery program. Review of Economics and Statistics, 93(4), 1297-1318
• Kling, J. R., Liebman, J. B., & Katz, L. F. (2007). Experimental analysis of neighborhood effects. Econometrica, 75(1), 83-119
• Lee, S. & Shaikh, A. (2013). Multiple Testing and Heterogeneous Treatment Effects: Re-Evaluating the Effect of PROGRESA on School Enrollment. Journal of Applied Econometrics, 29(4), 612-626
• O'Brien, P. C. (1984). Procedures for comparing samples with multiple endpoints. Biometrics, 1079-1087
• Schochet, P. Z. (2008). Guidelines for Multiple Testing in Impact Evaluations of Educational Interventions. Final Report. Mathematica Policy Research, Inc.

• Angelucci, M., & De Giorgi, G. (2009). Indirect Effects of an Aid Program: How Do Cash Transfers Affect Ineligibles' Consumption?. The American Economic Review, 486-508.
• Baird, S., Bohren, J. A., McIntosh, C., & Ozler, B. (2014). Designing Experiments to Measure Spillover Effects. World Bank Policy Research Working Paper, (6824).
• Gine, X., & Mansuri, G. (2011). Together we will: experimental evidence on female voting behavior in Pakistan. World Bank Policy Research Working Paper Series
• Lee, S. & Shaikh, A. (2013). Multiple Testing and Heterogeneous Treatment Effects: Re-Evaluating the Effect of PROGRESA on School Enrollment. Journal of Applied Econometrics. 29. 10.2139/ssrn.2214773
• Miguel, E., & Kremer, M. (2004). Worms: identifying impacts on education and health in the presence of treatment externalities. Econometrica, 159-217
• Özler, B. (2018). Most good you can do. But for whom? Measuring total causal effects (TCE), Development Impact Blog, World Bank

• Baird, S., De Hoop, J., & Özler, B. (2013). Income shocks and adolescent mental health. Journal of Human Resources, 48(2), 370-403
• Benedetti, F., Maggi, G., Lopiano, L., Lanotte, M., Rainero, I., Vighetti, S., & Pollo, A. (2003). Open versus hidden medical treatments: The patient's knowledge about a therapy affects the therapy outcome. Prevention & Treatment, 6(1), 1a
• Chassang, S., Miquel, G. P. I., & Snowberg, E. (2012). Selective Trials: A Principal-Agent Approach to Randomized Controlled Experiments. American Economic Review, 102(4), 1279-1309
• Jones, S. R. (1992). Was there a Hawthorne effect?. American Journal of Sociology, 451-468
• Kaptchuk, T. J., Friedlander, E., Kelley, J. M., Sanchez, M. N., Kokkotou, E., Singer, J. P., & Lembo, A. J. (2010). Placebos without deception: a randomized controlled trial in irritable bowel syndrome. PloS one, 5(12), e15591
• Kompier, M. A. (2006). The" Hawthorne effect" is a myth, but what keeps the story going?. Scandinavian Journal of Work, Environment & Health, 402-412
• Leonard, K., & Masatu, M. C. (2006). Outpatient process quality evaluation and the Hawthorne Effect. Social Science & Medicine, 63(9), 2330-2340
• Leonard, K. L., & Masatu, M. C. (2010). Using the Hawthorne effect to examine the gap between a doctor's best possible practice and actual performance.Journal of Development Economics, 93(2), 226-234
• Levitt, S. D., & List, J. A. (2011). Was There Really a Hawthorne Effect at the Hawthorne Plant? An Analysis of the Original Illumination Experiments. American Economic Journal: Applied Economics, 3(1), 224-38
• Muralidharan, K., & Sundararaman, V. (2011). Teacher Performance Pay: Experimental Evidence from India. The Journal of Political Economy, 119(1), 39-77
• Muralidharan, K., & Sundararaman, V. (2010). The Impact of Diagnostic Feedback to Teachers on Student Learning: Experimental Evidence from India. The Economic Journal, 120(546), F187-F203
• Saretsky, G. (1972). The OEO PC experiment and the John Henry effect. Phi Delta Kappan, 579-581
• Schumacher, J. E., Milby, J. B., Raczynski, J. M., Engle, M., Caldwell, E. S., & Carr, J. A. (1994). Demoralization and threats to validity in Birmingham's homeless project. New Directions for Program Evaluation, 1994(63), 41-44
• Zwane, A. P., Zinman, J., Van Dusen, E., Pariente, W., Null, C., Miguel, E., Kremer, M., Karlan, D., Hornbeck, R., Gine, X., Duflo, E., Devoto, F., Crepon, B.& Banerjee, A. (2011). Being surveyed can change later behavior and related parameter estimates. Proceedings of the National Academy of Sciences,108(5), 1821-1826
• Wickström, G., & Bendix, T. (2000). The" Hawthorne effect"—what did the original Hawthorne studies actually show?. Scandinavian Journal of Work, Environment & Health, 363-367

• Caplan, A. L. (2009). Is disease eradication ethical?. The Lancet, 373(9682), 2192-2193
• Emanuel, E. J., Wendler, D., & Grady, C. (2000). What makes clinical research ethical?. Jama, 283(20), 2701-2711
• Fraser, D. (2001). Beyond ethics: why we need evaluation standards. Evaluation Journal of Australasia, 1(1), 53-58
• Freedman, B. (1987). Equipoise and the ethics of clinical research. New England journal of medicine, 317(3), 141-145
• Hessler, R. M., Donnell-Watson, D. J., & Galliher, J. F. (2011). A case for limiting the reach of institutional review boards. The American Sociologist,42(1), 145-152
• Holden, C., (1979). Ethics in social science research. Science 206 (4418): 537-538
• Indian Council of Medical Research (2006). Ethical Guidelines for Biomedical Research on Human Participants. New Delhi 110 029
• Nama, N., & Swartz, L. (2002). Ethical and social dilemmas in community‐based controlled trials in situations of poverty: a view from a South African project. Journal of community & applied social psychology, 12(4), 286-297
• National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research (1979). The Belmont Report. Ethical Principles and Guidelines for the Protection of Human Subjects of Research
• Nelson, R. M., Beauchamp, T., Miller, V. A., Reynolds, W., Ittenbach, R. F., & Luce, M. F. (2011). The concept of voluntary consent. The American Journal of Bioethics, 11(8), 6-16
• Newman, D.L. and Brown, R.D. (1996) Applied Ethics for Program Evaluation, Thousand Oaks CA and London: Sage
• Osrin, D., Azad, K., Fernandez, A., Manandhar, D. S., Mwansambo, C. W., Tripathy, P., & Costello, A. M. (2009).Ethical challenges in cluster randomized controlled trials: experiences from public health interventions in Africa and Asia. Bulletin of the World Health Organization, 87(10), 772-779
• Simons, H. (2006). Ethics in Evaluation. The SAGE Handbook of Evaluation, 243

• Allcott, H. (2012). Site selection bias in program evaluation (No. w18373).National Bureau of Economic Research
• Aronow, P. M., & Samii, C. (2015). Does Regression Produce Representative Estimates of Causal Effects?. American Journal of Political Science
• Banerjee, A. V., Cole, S., Duflo, E., & Linden, L. (2007). Remedying Education: Evidence from Two Randomized Experiments in India. The Quarterly Journal of Economics, 122(3), 1235-1264
• Banerjee, A., Duflo, E., Goldberg, N., Karlan, D., Osei, R., Parienté, W., & Udry, C. (2015). A multifaceted program causes lasting progress for the very poor: Evidence from six countries. Science, 348(6236), 1260799
• Bloom, N., Eifert, B., Mahajan, A., McKenzie, D., & Roberts, J. (2013). Does Management Matter? Evidence from India. The Quarterly Journal of Economics, 128(1), 1-51
• Bold, T., Kimenyi, M., Mwabu, G., Ng'ang'a, A., & Sandefur, J. (2013). Scaling up what works: Experimental evidence on external validity in Kenyan education. Center for Global Development Working Paper, (321)
• Deaton, A., & Cartwright N., (2018). “Understanding and misunderstanding randomized controlled trials” Social Science & Medicine 210 (August 2018): 2-21
• Dhaliwal, I., Duflo, E., Glennerster, R., & Tulloch, C. (2011). Comparative cost-effectiveness analysis to inform policy in developing countries: a general framework with applications for education. Cambridge, MA: MIT Press
• Evans, D., & Popova, A. (2014). Cost-effectiveness measurement in development: accounting for local costs and noisy impacts. World Bank Policy Research Working Paper, (7027)
• Hjort, J., Moreira, D., Rao, G., & Santini, J.F.. How Research Affects Policy: Experimental Evidence from 2,150 Brazilian Municipalities. (2019).
• McEwan, P. J. (2014). Improving Learning in Primary Schools of Developing Countries A Meta-Analysis of Randomized Experiments. Review of Educational Research, 0034654314553127
• Morduch, J., Ravi, S., & Bauchet, J. (2013). Substitution Bias and External Validity: Why an innovative anti-poverty program showed no net Impact. NYU Wagner Research Paper, (2317322)
• Muller, S. (2014). Randomised trials for policy: a review of the external validity of treatment effects (No. 127). Southern Africa Labour and Development Research Unit, University of Cape Town
• Olsen, R. B., Orr, L. L., Bell, S. H., & Stuart, E. A. (2013). External validity in policy evaluations that choose sites purposively. Journal of Policy Analysis and Management, 32(1), 107-121
• Rothwell, P. M. (2005). External validity of randomised controlled trials:“to whom do the results of this trial apply?”. The Lancet, 365(9453), 82-93

• Baird, S., & Özler, B. (2012). Examining the reliability of self-reported data on school participation. Journal of Development Economics, 98(1), 89-93
• Thomas, D., Witoelar, F., Frankenberg, E., Sikoki, B., Strauss, J., Sumantri, C., & Suriastini, W. (2012). Cutting the costs of attrition: Results from the Indonesia Family Life Survey. Journal of Development Economics, 98(1), 108-123

• Bardasi, E., Beegle, K., Dillon, A., & Serneels, P. (2011). Do labor statistics depend on how and to whom the questions are asked? Results from a survey experiment in Tanzania. The World Bank Economic Review, 25(3), 418-447
• Blair, G., Imai, K., & Lyall, J. (2014). Comparing and combining list and endorsement experiments: Evidence from Afghanistan. American Journal of Political Science, 58(4), 1043-1063
• Blair, G., & Imai, K. (2012). Statistical analysis of list experiments. Political Analysis, 20(1), 47-77
• Dillon, B. (2012). Using mobile phones to collect panel data in developing countries. Journal of International Development, 24(4), 518-527
• Fitzsimons, G. J., & Morwitz, V. G. (1996). The effect of measuring intent on brand-level purchase behavior.Journal of Consumer Research, 1-11
• Kilic, T., &Sohnesen, T. P. (2015). Same question but different answer: experimental evidence on questionnaire design's impact on poverty measured by proxies. World Bank Policy Research Working Paper, (7182)
• List, J. A., Sadoff, S., & Wagner, M. (2011). So you want to run an experiment, now what? Some simple rules of thumb for optimal experimental design. Experimental Economics, 14(4), 439-457
• Miller, K., Msiyaphazi Zulu, E., & Cotts Watkins, S. (2001). Husband—Wife Survey Responses in Malawi. Studies in Family Planning, 32(2), 161-174
• Phillips, A. E., Gomez, G. B., Boily, M. C., & Garnett, G. P. (2010). A systematic review and meta-analysis of quantitative interviewing tools to investigate self-reported HIV and STI associated behaviours in low-and middle-income countries. International Journal of Epidemiology, 39(6), 1541-1555
• Sana, M., & Weinreb, A. A. (2008). Insiders, outsiders, and the editing of inconsistent survey data. Sociological Methods & Research, 36(4), 515-541

• Barrios, T., Diamond, R., Imbens, G. W., & Kolesar, M. (2012). Clustering, spatial correlations, and randomization inference. Journal of the American Statistical Association, 107(498), 578-591
• Behaghel, L., Crépon, B., Gurgand, M., & Le Barbanchon, T. (2012). Please call again: Correcting non-response bias in treatment effect models. Review of Economics and Statistics
• Bertrand, M., Duflo, E., & Mullainathan, S. (2004). How Much Should We Trust Differences-In-Differences Estimates?. Quarterly Journal of Economics, 119(1), 249-275
• Blair, Cooper, Coppock, Humphreys (2018). Cluster randomized trials can be biased when cluster sizes are heterogeneous. Blog post on DeclareDesign.org
• Bloom, N., Eifert, B., Mahajan, A., McKenzie, D., & Roberts, J. (2013). Does Management Matter? Evidence from India. The Quarterly Journal of Economics, 128(1), 1-51
• Bowers, J. (2011). Making effects manifest in randomized experiments. Cambridge Handbook of Experimental Political Science, 459-80
• Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2008). Bootstrap-based improvements for inference with clustered errors. The Review of Economics and Statistics, 90(3), 414-427
• Cameron, A. C., & Miller, D. L. (2015). A practitioner’s guide to cluster-robust inference. Journal of Human Resources, 50(2), 317-372
• Cohen, J., & Dupas, P. (2010). Free Distribution or Cost-Sharing? Evidence from a Randomized Malaria Prevention Experiment. Quarterly Journal of Economics, 125(1)
• Freedman, D. A. (2008). On Regression Adjustments in Experiments with Several Treatments. The Annals of Applied Statistics 176-196
• Freedman, D. A. (2008). On regression adjustments to experimental data. Advances in Applied Mathematics, 40(2), 180-193
• Ho, D. E., & Imai, K. (2006). Randomization inference with natural experiments: An analysis of ballot effects in the 2003 California recall election. Journal of the American Statistical Association, 101(475), 888-900
• Imai, K., King, G., & Nall, C. (2009).The essential role of pair matching in cluster-randomized experiments, with application to the Mexican universal health insurance evaluation. Statistical Science, 24(1), 29-53
• McCloskey, D. N., & Ziliak, S. T. (1996). The standard error of regressions. Journal of Economic Literature, 97-114
• Samii, C., & Aronow, P. M. (2012). On equivalencies between design-based and regression-based variance estimators for randomized experiments. Statistics & Probability Letters, 82(2), 365-370
• Solon, G., Haider, S. J., & Wooldridge, J. M. (2015). What are we weighting for?. Journal of Human Resources, 50(2), 301-316

• Abadie, A., Athey, S., Imbens, G. W., & Wooldridge, J. M. (2014). Finite Population Causal Standard Errors (No. w20325). National Bureau of Economic Research
• Abadie, A., Chingos, M. M., & West, M. R. (2013). Endogenous stratification in randomized experiments (No. w19742). National Bureau of Economic Research
• Barrios, T. (2013). Optimal Stratification in Randomized Experiments
• Barrios, T., Diamond, R., Imbens, G. W., & Kolesar, M. (2012). Clustering, spatial correlations, and randomization inference. Journal of the American Statistical Association, 107(498), 578-591
• Bertsimas, D., Johnson, M., & Kallus, N. (2015). The Power of Optimization Over Randomization in Designing Experiments Involving Small Samples. Operations Research
• Brodeur, A., Lé, M., Sangnier, M., & Zylberberg, Y. (2012). Star wars: The empirics strike back. Paris School of Economics Working Paper, (2012-29)
• Bruhn, M., & McKenzie, D. (2009). In Pursuit of Balance: Randomization in Practice in Development Field Experiments. American Economic Journal: Applied Economics, 1(4), 200-232
• Crump, R. K., Hotz, V. J., Imbens, G. W., & Mitnik, O. A. (2008). Nonparametric tests for treatment effect heterogeneity. The Review of Economics and Statistics, 90(3), 389-405
• De Chaisemartin, C. (2014). Tolerating defiance? Local average treatment effects without monotonicity. Warwick Economics Research Paper Series, (1020)
• Imbens, G. W. (2010). Better LATE than nothing. Journal of Economic Literature, 48
• McKenzie, D. (2012). Beyond baseline and follow-up: The case for more T in experiments. Journal of Development Economics, 99(2), 210-221

• Handbook of field experiences: webpage collecting together all the working papers version of the chapters, and will also link to the final versions as they become available. 
• Curated List of Postings on Technical Topics World Bank Blog on Development Impact: technical postings on variety of impact evalution methods and other useful topics.
• How to Write the Introduction of Your Development Economics Paper Centre for Global Development Blog