July 2009 David McKenzie, DECRG-FP
The World Bank, IFC, and National Governments implement many policies and projects each year designed to promote private sector and financial development. Yet we have almost no rigorous evidence about which policies work, which don’t, why, and how we can make them better.
Rigorous impact evaluations, which compare the outcomes of a program or policy against an explicit counterfactual of what would have happened without the program or policy are one of the most important tools that can be used along with appropriate economic theory for understanding “what works”. Despite this, until recently impact evaluations have been extremely rare. This is now particularly apparent in the area of finance and private sector development, where the recent financial crisis has prompted renewed attention to knowing what works in terms of getting finance to consumers and firms, and in getting the private sector growing again.
Why haven’t impact assessments been common?
- There is a perception that many finance and private sector development policies and projects lend themselves less to formal evaluations. Changes in laws or regulations may occur at an economy-wide level, or a large loan may only be given to one or two banks or firms.
- There has been a tendency by both bankers and government officials to measure success of projects by the number of loans given out, number of businesses trained, or number of business procedures reduced, with insufficient incentives being present for the harder task of measuring whether this increases firm productivity or lowers poverty.
Yet a vanguard of recent efforts show both the feasibility and value of evaluations
A handful of recent impact studies (highlighted in our new Finance & PSD Impact Note series together with a new overview paper (McKenzie, 2009) dispel the myth that rigorous evaluations of finance and private sector projects are not possible.
- In many cases randomized experiments are possible, which offers many advantages for evaluation. In particular, they ensure that the only reason why some firms, consumers, or other units are subject to a policy or program and others are not is pure chance, making the results easy to communicate to policymakers. Recent examples include randomized provision of grants to small business owners, crop insurance to poor farmers, business training to entrepreneurs, and marginal borrowers to receiving credit or not. These experiments have generated many important findings, including several that challenge the central precepts of microfinance.
- Even when randomization isn’t possible (or is theoretically possible but wasn’t done), modern evaluations methods often will still allow for formal evaluation. Regulations may be implemented in some regions and not others, or apply only to firms of a certain industry or size. Generally available programs or policies may have low take-up that can be raised through targeted interventions. Taking advantage of these ideas, recent evaluations have examined the success of business registration reform in Mexico, bankruptcy reform in Colombia, and the introduction of a credit bureau in Guatemala.
What are the lessons for future impact evaluations?
- Evaluation is feasible and useful for a wide range of projects and policies. Senior government officials and Bank staff need to make it a priority in project development, so that evaluation not only tells whether a project worked, but also helps in ensuring it can work even better next time.
- Most existing evaluations have been limited to consumers, microenterprises, or poor borrowers. There is a big need for evaluation in policies directed at SMEs and banks.
- Evaluation designs need to recognize the low take-up of many finance and private sector policies. Not all households or firms will want or need a loan, attend training sessions, register formally, or purchase insurance. Low take-up presents challenges for both identification and interpretation of program impacts, which can be overcome provided the issue is recognized upfront.
The few incipient efforts in rigorous evaluation have yielded high payoffs in terms of both new knowledge generation and better policy design. Given the World’s focus in getting the private sector growing again after the financial crisis it is now more important than ever to know what works and why. We hope our new research and Impact Note series will convince more operational staff, NGOs, and Government officials of both the feasibility and desirability of doing so.
For further reading, see:
McKenzie, David (2009) “Impact Assessments in Finance and Private Sector Development”, World Bank Research Observer, forthcoming.
(also available as World Bank Policy Research Working Paper No. 4944)