|Dec 2006, Shaohua Chen, Ren Mu, and Martin Ravallion
Publicly-supported grants and loans to poor areas have long been an important vehicle for development assistance. For example, China’s anti-poverty policies have emphasized such poor-area programs since the mid-1980s, motivated by the observation that the country’s success against poverty over the last 25 years has been geographically uneven, with marked disparities in living standards emerging.
Advocates of such programs claim that credit constraints in poor areas perpetuate their poverty and that targeted aid can relieve those constraints. By this view, capital-market failures in poor areas entail that the investments made under such a program would be infeasible otherwise, implying both efficiency and equity gains.
It remains an open question how much impact can be expected. While not perfect, capital markets may still work well enough to assure that marginal products of capital come into rough parity between poor and non-poor areas in steady state. Then the problem of lagging poor areas is not so much lack of capital as low productivity of capital, such as due to poor natural conditions or poor policies.
And even with credit constraints, some people are clearly more constrained than others. Then beneficiary selection will be crucial to the outcomes. If those selected are not credit constrained, their participation is voluntary, and the interest rate is no different from other credit sources, then there will be no net gain from the extra availability of credit through poor-area programs. We know very little about how well these interventions have performed in reaching the most credit-constrained households in poor areas, who need not be the poorest.
Assessing the long-term impacts of aid to poor areas
Policy research working paper 4084 provides the first rigorous assessment of the longer-term impacts at the micro (household and village) level of a large poor-area program. The program is the World Bank financed Southwest China Poverty Reduction Project — the Southwest Program (SWP) for short. This comprised a package of multi-sectoral interventions targeted to poor villages using community-based participant and activity selection. The overall aim was to achieve a large and sustainable reduction in poverty. The paper reports results from an intensive survey data collection effort over 10 years, initiated by two of the authors and done in close collaboration with the Rural Survey Organization of China’s National Bureau of Statistics.
Assessing aid effectiveness at the project level raises a number of challenges, which the paper tries to address. A long-term commitment to collecting high-quality longitudinal survey data is crucial, but it is not sufficient. Impact can only be meaningfully assessed relative to a counterfactual; here the counterfactual is the absence of the SWP, which means that the authors assess the incremental impacts, on top of pre-existing governmental spending.
As in any observational study, there are concerns about selection bias, i.e., differences in counterfactual outcomes between SWP participants and non-participants. The study’s data collection effort allows the authors to “difference out” the time-invariant component of the selection bias (arising from non-random placement). However, they argue that it is implausible that the bias would be constant over time, given that the initial village characteristics that attract the program (such as poor infrastructure) also influence the growth rate under the counterfactual. The authors use both propensity-score weighted regression and kernel-matching methods to balance the observable covariates between sampled SWP and non-SWP villages.
A further problem is that development projects are likely to violate the common assumption in impact evaluations (both experimental and non-experimental) of no interference with the comparison units. Mobility between the targeted poor areas and the comparison areas is one possible source of interference, although its relevance here is questionable, given China’s restrictions on mobility, reflecting long-standing policies related to household registration and administrative (non-market) land assignment.
A more plausible source of interference in this setting is through local public-spending spillover effects to non-SWP villages. The authors propose and implement a test for spillover effects.
Given that it is rare to assess impacts by repeated observation over a long period, this study also provides an opportunity to study less costly evaluation methods, based on respondent recall using subjective-qualitative questions.
Figure 1 summarizes the results. The figure gives mean income (“Y”) and mean consumption (“C”) over time for both SWP villages and non-SWP villages.
We see in Figure 1 a sizeable and statistically significant impact on mean household income in the participating villages during the disbursement period (1995-2000). However, there was a much smaller impact on consumption during that period; the short-term income gains were largely saved.
Four years after disbursements had ended, both project and non-project villages had seen sizeable economic gains, with only modest net gain to mean income attributed to the project. Indeed, the authors cannot reject the null hypothesis that the longer-term average impact was in fact zero, although they do find evidence of longer-term impacts on income in-kind from animal husbandry. These findings were robust to various corrections for selection bias.
The most plausible interpretation of the study’s empirical findings appears to be as follows. The high savings rate from the initial income gains reflected uncertainty about the future impacts — no doubt compounded by the uncertainty about the project’s loan repayment and interest obligations, given uncertain contract enforcement at local level. Farm animals were an important form of saving as well as being the main source of the short-term income gains.
No doubt the relevant uncertainties were resolved in the longer term. Productivity gains turned out to be small. The initial income gains proved to be transient for most households, although there was some persistence in the income gains from animal husbandry. The mean consumption gains over the longer-time period are in rough accord with what one would expect from the (modest) increment to permanent income attributable to the project.
The study’s results indicate significant and lasting income gains among the subset of households who were initially poor and relatively well educated. Presumably these households had more productive investment options, which could not be financed otherwise given the liquidity constraints facing the poorest. The program’s community-based selection process favored the better educated, but expanded coverage of those who were also poor could have greatly enhanced the program’s overall impact.
Given the heterogeneity in returns, the implied (ex-post) deficiencies of the community-based selection process help explain the program’s disappointing overall impact. While the program performed well in selecting poor villages, overall impacts were greatly attenuated by inadequate coverage of the (educated) poor within poor villages.
Lessons for future impact evaluations
Some generic lessons emerge for assessing the impacts of development projects. The importance of investing in longer-term survey-data collection is plain. The authors show that rapid appraisal methods based on a single post-intervention survey using subjective-qualitative questions are vulnerable to severe recall-error biases, stemming from the fact that respondents’ perceptions of how their living conditions have changed give far too high a weight to current circumstances.
The results of this study also point to the importance of taking account of the participants’ inter-temporal behavior in response to the uninsured risks often associated with a development project. Those responses can cloud impacts in both experimental and non-experimental evaluations. An evaluation that focused solely on the income or consumption gains during the disbursement period (as is commonly the case) can give a deceptive picture of the true impacts.
And the findings illustrate how the responses of local development agents can cloud identification of the long-term impacts of geographically-placed projects (whether randomly placed or targeted). The authors found evidence of positive spillover effects on the comparison villages through the displacement of other development spending.
Such interference suggests that the classic impact evaluation methods will systematically underestimate the long-term impacts. In this case, it is unlikely that these effects are imparting a large bias on our impact estimates, under plausible assumptions on the relevant parameters. But this may well be a bigger problem in other settings.