The key characteristic of both approaches is the experimental framework, conducted in a real-world environment. Central to this is the notion of random assignment between treatment groups (who receive the intervention) and control groups (who do not). Random assignment ensures that the two groups are comparable and thus all differences noted after the intervention can be confidently ascribed to the intervention. This method is considered the gold standard for policy evaluation, because it produces unbiased results.
The experimental framework requires careful design of the intervention and collection of baseline data on both groups well before the intervention takes place. This framework is typically used to test new interventions in a pilot phase, to explore their impacts and to compare alternative intervention designs (i.e. by distinguishing several different treatment groups). Results can then be used to determine if and how to eventually scale up the new intervention.
Randomized controlled trials (RCTs)
RCTs use random assignment to allocate the program or intervention being analyzed. They are generally considered the gold standard for policy evaluation, producing unbiased results. To quantify the impact of the intervention, the outcomes of the units receiving the program are compared against those who did not receive it.
RCTs are generally large-scale endeavors with the aim of evaluating the effectiveness of policy packages. These projects are often expensive, multi-year, and are carried out by governments or multilateral organizations wanting to test whether a program comprised of several interventions achieves the outcome it was set to achieve or not.
For example, Conditional Cash Transfer (CCT) programs targeted at vulnerable populations combine conditionalities, income support and some sort of health assistance with the aim of increasing schooling. An RCT designed to evaluate the effectiveness of CCTs would be implemented in the form of a pilot, where program participation is assigned randomly, and where results refer to the effect of the whole package of interventions on schooling.
Field experiments (FEs)
FEs also use random assignment to allocate treatment as well as controlled environments that make it possible to clearly identify the effect of a treatment and/or learn about micro-motives for behavior.
While sharing a common methodology, field experiments are generally smaller scale endeavors than their RCT counterparts.
FEs may also be set up to evaluate a policy, but they are more frequently devised to learn about the effectiveness of a given mechanism to implement a program or to test economic hypothesis. Thus, they are an important tool to design better policies. Due to their focused nature, FEs can be implemented in shorter periods of time and generally require less funding than RCTs.
For example, in the case of a government trying to make decisions about a Conditional Cash Transfer program, an FE could be carried out to find out whether the income support portion of the program has a higher impact on schooling when paid to mothers or fathers of children, for example.
The PEP experimental group encourages researchers to think of creative ways to adapt well-known experimental protocols to answer research questions relevant to local barriers faced by subjects in developing countries.
Both types of projects (RCTs and field experiments) require a good institutional knowledge of the program under study, in terms of:
- Eligibility rules and the target population
- Objectives of the intervention and its potential unintended effects
- Outcomes that may potentially be affected by the program
- The intervention calendar and the timing of effects (short-term, medium-term, long-term)