Francisco Campos, Aidan Coville, Ana Fernandes, Markus Goldstein, and David McKenzie,
Our latest note illustrates a case where attempting to do impact evaluations can provide valuable knowledge, even if the impact evaluations ultimately are unable to be completed.
A typical matching grant consists of a partial subsidy - most commonly covering 50 percent of the cost - provided by a government program to a private sector firm to help finance the costs of activities to promote exports, innovation, technological upgrading, the use of business development services, and, more broadly, firm growth.
Matching grant programs are one of the most common policy tools used by developing country governments to actively facilitate micro, small, and medium enterprise competitiveness, and have been included in more than 60 World Bank projects totaling over US$1.2 billion, funding over 100,000 micro, small and medium enterprises.
Yet despite all the resources spent on these projects, there is currently very little credible evidence as to whether or not these grants spur firms to undertake innovative activities that they otherwise would not have done, or merely subsidize firms for actions they would take anyway.
Since firms self-select into whether they apply for such programs, and then the programs decide which applicants receive funding, attempts to compare outcomes for matching grant recipients to non-recipients are likely to be biased.
We set out to design randomized experiments to prospectively evaluate seven matching grant programs in six African countries. Five were to be supported through World Bank loans and technical support, while two stemmed from a direct engagement with the government.
In theory, matching grants satisfy a number of conditions that make randomization a possibility: i) they involve selection of individual firms; ii) the numbers of firms involved can be large enough to potentially generate enough statistical power for measuring impacts, and iii) data on key outcomes may be measured reasonably well through firm surveys.
Given that the government is effectively giving away free money to firms, one might expect significant demand for this funding, resulting in the need for projects to select which firms receive it. Since we believe there is substantial uncertainty over which firms would best benefit from receiving these funds, our suggestion was for randomized evaluation based on an oversubscription design. The idea here would be to make the matching grant programs open for all firms meeting certain basic eligibility criteria, and then randomly select which firms would be awarded the grants from among eligible applicants.
Out of the seven projects that we discussed impact evaluations with, five initially agreed to implement projects with an oversubscription-based randomization experiment included, while the other two had encouragement designs planned.
However, in practice, we were unable to implement any of the randomized experiments successfully. The main reasons were:
- Lack of applications: despite giving away free subsidies, programs struggled to get enough applicants, resulting in insufficient eligible applicants to randomize among.
- Repeated delays and changes in personnel: implementation delays of over a year or more led to changes in government personnel, reversing some of the buy-in for the evaluation; it also meant we ran up against impact evaluation funding.
Why is it so hard to give away free money?
- Political economy and capture: these subsidies were viewed by some governments and partners as something to give their constituents: so we found Chambers of Commerce lobbying to keep eligibility conditions such that only their members would be eligible, while local governments competing with national governments in programs.
- Overly strict eligibility criteria: requiring firms to be registered with audited tax accounts tended to exclude most firms in many African countries: criteria were set on the idea of which firms would grow fastest, not which firms would see the greatest benefit from the program.
- Last mile issues and red tape: many of the conditions imposed by the World Bank and governments made it difficult and burdensome for even eligible firms to apply and receive money. Examples include: requiring firms to get letters from the tax department to show they were current on taxes; requiring firms to go through procurement processes like getting three bids in writing for services, and paying upfront for services with delayed reimbursement; and restricting what grants could be used for.
- Incentives facing project staff: project staff were typically on fixed wage contracts, giving no incentive to try and get more applications.
- Matching grant programs need to change the mindset from picking winners to picking positive treatment effects: the latter is even harder to judge, making impact evaluation even more important.
- Focus more on eligibility criteria and making it easy for firms to apply, and to get the money once they are awarded it.
- There are also several lessons for designing impact evaluations of such projects: these include using methods for small samples; having more realistic expections on time frames; and conducting “little IE” or impact evaluation on different program design features.
For further reading see:
Francisco Campos, Aidan Coville, Ana Fernandes, Markus Goldstein and David McKenzie (2012) “Learning from the experiments that never happened: Lessons from trying to conduct randomized evaluations of matching grant programs in Africa” World Bank Policy Research Working Paper no. 6296.