Using experimental methods for evidence-based policy making

Context
Abhijit Banerjee, Esther Duflo and Michael Kremer win 2019 Nobel Economics Prize for study on poverty.

Background

• Nobel Economists have redeveloped the field of development economics using new experimental methods that put researchers in direct contact with the poor.
• There are three big shifts in their field of work:
• First, development challenges are now viewed through an appropriate lens, the lives of the poor (rather than large statistical models), with a special focus on how incentives, information and constraints shape actual choices.
• Second, their use of Randomized Control Trials (RCTs) has lent credibility to poverty research and helped solve old riddles of causality.
• Third, their expertise has led to the formulation of policies that go by evidence, not assumptions.

How is RCT used in the field of development economics?

• Simply put, the idea is to understand the impact of interventions to achieve desirable outcomes. Or, why does a particular scheme/intervention work, while the other does not?
• There are four parts to this story which the new experimental method attempts to answer:
  • First, there is need to identify the causes of issue (here, poverty),
  • Second have the necessary interventions in place which can address the issue
  • Third, carry out field experiments which work so that those which don’t can be abandoned.
  • The last is the cost-benefit analysis to evaluate efficacy. Accordingly, those interventions which work can be persevered.

Examples where controlled experiments were successfully used

• Case of Immunisation in Rajasthan, India
  • Despite offering free immunisation, women were not bringing their children into government clinics to avail the facility.
  • Research team started distributing dal packets to women who came to get shots for their children.
  • Soon, word spread and immunization rates soared.
• Case of de-worming drive in Kenya
  • Researchers found that parents in low-income countries are much less likely to give their children de-worming pills when the medication is heavily subsidized, than when it is available for free.
  • Benefits of de-worming were be staggering, way beyond the costs of such an intervention.
• Case of Remedial Tutoring in India
  • The reason was that most government schools have classrooms with children of different abilities – some have Maths skills of Class 2, some 3 and some 6 – so the class teacher doesn’t know quite where to pitch the lesson.
  • RCT established that providing remedial tuitions for individual students improved overall test scores dramatically.
• Case ofpublic goods provision in local governments with women leaders.
  • Researchers had no idea about the direction of causality between, progressive villages with better provision of public goods and villages with women leaders.
  • By studying a massive data-set from West Bengal and Rajasthan, they proved that the provision of local public goods, like water supply, improves in statistically significant ways in villages where women are elected to lead.

What are the lacunas in Indian policy making and how can experimental research fill in the gaps?

• Gaps in Indian policy making
  • Recently, the government has undertaken bold steps in policy-making but it has done it too quickly without enough homework.
  • Schemes and new ideas are often not put on the table for public debate.
  • Innovations of previous governments are often not continued. They must be acknowledged and scaled up.
  • Lessons are from past mistakes are not learnt. A good example is the Banking Correspondent Model (BCM), which is the right complement to Jan DhanYojana.
  • It is problematic to announce the scheme first and see the fallout later, like it happened in case of demonetization and implementation of GST.
  • There is a tendency to set monetary or financial targets, which are achieved without being too effective in delivering outcomes.
• Application to policy making
  • In India we have an array of problems where experimental research can be carried out- health, family planning, education, credit, agriculture, social security and so on.
  • Once the problem is identified, we need to carry out experiments to see how possible solutions may work, and integrate them
  • Niti Aayog can take the thesis of these awardees as a template for evaluating all the programmes of the government.
  • Such an evaluation will help to sieve out programmes that have not worked. The same funds can be routed to others which have worked, or alternatively new programmes can be constructed.
  • It is important to make every-rupee expenditure with a definite cost-benefit analysis in place.
  • Issue of corruption must be seriously addressed in order to make policies effective.
  • Since randomised trials require a lot of work on the ground, adequate mechanisms and structure must be put in place.
  • A new generation of RCTs are going beyond programme evaluation and can help seek answers to how individuals react to changes in prices, contracts, and new information in the context of specific markets such as land or credit.
  • In light of the new approach, role of income transfers can be re-looked at, when strategising anti-poverty measures.

What are some criticisms of randomized controlled trials?

• Solutions may not be universally applicable; they may not work in other geographies; nor is it clear why a particular solution works.
• Some argue that an obsession to trace the causal links of one phenomenon to another is a confusing and futile exercise.
• Field trials tend to miss the big structural changes that influence the political economy of a country.
• Some of the solutions are intuitive, and can also be achieved even without the use of RCT.
• RCT-type of experiment does not always give a clear answer on preferable policy option.
• Some critiques contend that RCT is more suited for research in the physical sciences than social science.

Conclusion

• Despite all criticism, the potential of successfully using experimental methods in development economics; and consequently policy-making, cannot be ignored. Laying micro-foundations is important to understand the big picture. The need for evidence-based policies stems from the fact that, smaller and more precise, questions are often best answered via carefully designed experiments among the people who are most affected. This is overall a case for an inclusive strategy to revive growth.