A Poverty Research featured article, April 2007
Organizations like the World Bank are often asked if they know what has been achieved through development aid. Is aid money effective? What exactly is gained from each dollar spent?
A recent example of such evaluation work is a long-term assessment of a World Bank project in rural southwest China. The evaluation, conducted by the Bank’s Development Research Group, was based on specially-designed survey data spanning ten years and collected for the purpose of the Bank’s evaluation by China’s National Bureau of Statistics. Households in two types of villages were studied – those that participated in the project and those that hadn’t.
In this evaluation, researchers Chen, Mu, and Ravallion make the first rigorous attempt to assess the longer-term impacts of a large development program—at the ‘micro’ or village and household level—in a low-income area.
“Research work that evaluates projects is important because the knowledge we gain from it feeds back into better-informed Bank lending for development,” said L. Alan Winters, Director of the Development Research Group.
Paramedic in front of village health clinic built under the project
|Did China’s South West Poverty-Reduction Project have a lasting impact?|
In the late 1990s, the Southwest China Poverty Reduction Project (SWPRP) offered a package of interventions that involved community-based participation and activity selection in areas such as farming, animal husbandry, infrastructure, and social services.
The SWPRP like others of its kind, aimed to achieve a large and sustainable reduction in poverty. The project covered Guangxi, Guizhou and Yunan – one of the poorest areas in China with a population of 120 million across 800,000 square kilometers.
China’s anti-poverty policies have emphasized poor-area programs for over 20 years, spurred by the geographical unevenness of the country’s success against poverty. Advocates of these programs have asserted that credit constraints in poor areas perpetuate their poverty and that targeted aid can relieve those constraints.
The surveys data collected for the evaluation showed that average household income in the project villages increased significantly between 1995 and 2000 (the project’s disbursement period). The gains were significantly larger than found in the matched control villages, which did not get the project.
However, most of these short-term income gains were saved rather than spent. As a result, the project’s impact was not evident in current living standards in the project villages.
Modest gains in consumption did occur in the longer term, in step with the modest long-term increase in income that can be linked to the project.
|Capital and technical assistance, but no insurance|
In common with other development projects, the SWPRP provided the capital and technical assistance, but did not provide insurance.
Many of the project’s activities were likely to entail significant income risk depending on vagaries of the weather, uncertain demand for new products, and risks associated with outward migration.
Participants felt that a large share of the income gains during the disbursement period was likely to be transient – hence the high savings rate.
|Four years after project disbursements ended, both project and control villages had seen sizeable economic gains, but only modest net gains to average income were attributable to the project.|
The results also show that when households were relatively well-educated as well as poor, there were more significant and lasting income gains. Presumably, they had productive investment options that could not otherwise be financed given their cash constraints.
But the program’s community-based selection process did not give sufficient weight to the types of households that turned out to be the relative “winners” from the project. Expanded coverage of those who were better educated, and also poor, could have greatly enhanced the program’s overall impact.
Bank researcher, Shaohua Chen, interviewing a project participant
|Lessons for future impact evaluations|
Some Some generic lessons emerge from this research:
Invest in long-term data collection. The authors show that quick assessment methods with a single post-program questionnaire are prone to severe recall errors. Respondents’ perceptions of how their living conditions have changed give far too high a weight to current circumstances.
Take into account responses to uninsured risks. How participants respond to uninsured risks often associated with a development project can cloud evaluation. For instance, an evaluation that focused solely on the consumption gains during the disbursement period (as is commonly the case) can give a deceptive picture of true impact.
Watch out for the effects of other development spending. The authors found that there were spillover effects on the comparison villages as a result of other development spending. Changes in local development spending in response to external aid can make it hard to detect the long-term impacts of such geographically-placed local development projects.
Shaohua Chen is a Senior Information Officer with the World Bank’s Development Research Group. She is an expert on poverty data for developing countries, including China. More.
Ren Mu is an economist who specializes on applied econometrics and worked on this project as a consultant to the Development Research Group.
Martin Ravallion is Senior Research Manager in the Development Research Group of the World Bank. His main research interests over the last 20 years have concerned poverty and policies for fighting it. He has advised numerous governments and international agencies on this topic, and he has written extensively on this and other subjects in economics. More