When Do Environmentally-Focused Aid Projects Achieve their Objectives? Evidence from World Bank Post-Project Evaluations
Researchers:
- Mark Buntaine
- Brad Parks
Sample:
146 environmentally-focused World Bank projects, 1994-2009
Timeline:
2010-2011
Themes:
Aid Effectiveness, World Bank Environmental Projects, Climate Finance
Overview:
Scholars and practitioners alike have paid a great deal of attention to the factors that lead to successful outcomes in environmentally-focused aid projects. Previous attempts to explain the success of these projects have identified a number of potentially important factors, including host country commitment, capacity and institutional quality, the severity of environmental pressures, contracting dynamics with donors, project characteristics, and civic participation in the environment sector. While theoretical development regarding environmental aid effectiveness is well-advanced, previous studies have not leveraged the large number of standardized post-project evaluations that have been produced by development banks. We compile World Bank evaluations and use the resulting dataset to test several prominent hypotheses about the factors that promote successful environmental aid outcomes. This study represents one of the first attempts to subject theories of environmental aid effectiveness to a general empirical test, with results that should be instructive to donor organizations that design and implement environmental aid projects. We find that the two most important factors predicting the successful implementation of environmental projects are good governance in the borrowing country and less focus on achieving global targets.
Research Question:
Over the last decade, calls for a massive "scaling up" of environmental assistance have built to a sharp crescendo (Baer, Athanasiou, Kartha, & Kemp-Benedict, 2008; Barrett, 2003; Green, 2008; Roberts 2009; UNDP, 2007; World Bank, 2009). The U.N. recently called on wealthy countries to help mobilize $128-167 billion a year by 2030 to support a wide array of mitigation and adaptation activities related to climate change in the developing world: building wind farms, weatherizing homes, attaching scrubbers to coal-fired power plants, constructing sea dikes to protect coastal communities, planting drought- and flood-resistant crops, etc. (UNFCCC 2007). International negotiators insist that it will be nearly impossible to negotiate an effective global climate agreement without a dramatic increase in external financing. Yvo de Boer, the Executive Secretary of the U.N. Framework Convention on Climate Change, stated in no uncertain terms during the weeks preceding the December 2009 Copenhagen round of global climate negotiations: "Finance is the key to a deal in Copenhagen. ...Money, in fact, is the oil that encourages commitment and drives action" (Kanter 2009).
Despite this groundswell of support for environmental assistance, there is remarkably little evidence that stand-alone environment projects funded by outside donors have substantially improved environmental outcomes in the developing world. Indeed, the limited evidence that does exist suggests that externally-funded projects seeking to protect or remediate the environment are generally less successful than other types of traditional development projects. In 2005, the World Bank's Independent Evaluation Group (IEG) reported that across nine sectors, environmental projects were the least successful projects in the World Bank's FY01-FY03 portfolio. Only 25 percent of Bank-financed environment projects during this period received a "satisfactory" project outcome rating, compared with 100 percent of education projects, 86 percent of health projects, and 87 percent of infrastructure projects (World Bank 2005).
In this paper, we conduct a large-N empirical analysis of the conditions under which environmentally-focused projects are most successful. Previous attempts to explain the success of these projects have identified a number of potentially important factors, including host country commitment, capacity and institutional quality, the severity of environmental pressures, contracting dynamics with donors, project characteristics, and civic participation in the environment sector. However, most of this research has been conducted using case studies and small-n methods. To our knowledge, no one has taken advantage of the large number of standardized post-project evaluations that have been produced by the World Bank and the regional development banks. We compile 146 evaluations of environmentally-focused World Bank projects implemented since 1994 and use the resulting dataset to test several prominent hypotheses about the factors that promote successful environmental aid outcomes. As such, this study represents one of the first attempts to subject theories of environmental aid effectiveness to a general empirical test.
Theory and Hypotheses:
Scholars and development practitioners have identified a wide range of factors that may contribute to effective implementation of (environmental) aid projects.
| Hypothesis | Variable | Description |
|---|---|---|
| H1: Ability to Provide Public Goods | World Bank Institute's Government Effectiveness index | This index "measures the quality of public services, the quality of the civil service... the quality of policy formulation and implementation..." Higher values represent better governance. |
| H2: Open Economic Policies | The Heritage Foundation's Trade Freedom index | This index is "a composite measure of the absence of tariff and non-tariff barriers that affect imports and exports of goods and services." Higher values represent more trade freedom. |
| H3a: Importance of Environment to Recipient | Annualized local environmental damages as a proportion of GDP | The World Bank Environment Department calculates the amount of national savings that is lost to environmental degradation each year, based on forest depletion, CO2 output, and air pollution damages. We have re-calculated this measure without CO2 output in order to measure the extent of local environmental damages. |
| H3b: Demand for Environmental Projects | Binary measure of World Bank project financing source (IBRD or IDA) | IBRD loans are offered at commercial interest rates, indicating borrower demand, whereas IDA loans are highly concessional. |
| H4a: Project Integrates Development Priorities | Percent of project financing targeting environmental objectives | This indicator measures the percent of project financing targeting environmental objectives, as coded by Powers et al. (2009). |
| H4b: Emphasis on Physical Outputs | Binary measure of the project's environmental risk category (Category A or Category B) | Every World Bank project is assigned a risk category based on expected environmental impacts. Category "A" and "B" projects are expected to have such impacts and thus have tangible outputs. |
| H4c: Project for Global Benefits | Percent of project falling under World Bank biodiversity or climate change theme codes | Every World Bank project is assigned to a variety of themes, ranging from economic management to conflict prevention. This variable is measured as the percent of the project budget assigned to either a biodiversity or climate change theme. |
| H5: Civic Participation in the Environment Sector | Freedom House civil liberties index multiplied by the number of environmental NGOs per million population | Civil liberties are rated on a 7-point scale, using qualitative information about the extent of freedom of expression, association, rule of law, and individual rights. Fredriksson, Neumayer, & Damania (2005) report the total number of environmental NGOs, using the 1994, 1997 and 2001 editions of the Environment Encyclopedia and Directory. The authors separately compiled data on the total number of environmental NGOs from the 2005 edition. |
| H6a: Size of Project | Total amount of project financing | This indicator measures the total amount of money allocated to a project, including all IBRD, IDA, GEF, grant and counterpart financing. |
| H6b: Size of Project & Governance | Total financing, multiplied by WGI Gov't Effectiveness | This indicator measures the total project amount multiplied by the Government Effectiveness variable described in H1. |
Data Collection:
One of the immediate difficulties involved with modeling the predictors of environmental project success is identifying the set of environmental projects to which our hypotheses apply. Powers et al. (2009) provide a unique measure of the percentage of individual World Bank project budgets allocated for environmental purposes. Using the coding scheme for "environmental strictly defined" and "environmental broadly defined" donor expenditure reported in Hicks et al. (2008), they derive a measure of environmental funding within World Bank projects. These financial estimates are available for 3,817 projects approved between 1994 and 2007. For this analysis, we have defined the universe of "environment projects" as those projects with a ratio of environmental spending-to-total spending that equals or exceeds 50%. 274 projects meet this criterion and form the universe of potential projects in our study.
Once we identified the universe of projects to include in our analysis, we collected the overall effectiveness rating given to each project in its Implementation Completion Report (ICR), which is the official closing report of the World Bank team responsible for implementing a project. The stated purpose of these reports is to "document the results achieved; the problems encountered; the lessons learned; and the knowledge gained from carrying out the project... The report describes and evaluates final project outcomes. The final outcomes are then compared to expected results." The final effectiveness outcome of every project is rated on a six-point scale ranging from highly unsatisfactory to highly satisfactory. Not all projects that meet the 50% criteria described above were completed and evaluated by 2009. For those that were completed and evaluated, we collected the outcome scores from ICRs that were available in the World Bank's online project database. In addition, we made note of projects that were either cancelled or subjected to the Inspection Panel process due to alleged violations of Bank policy. In the event that an ICR was not available for a particular project, we used either the Evaluation Summary or Project Performance Assessment Report rating given to the project by the Bank's Independent Evaluation Group, if available. This process yielded a sample of 146 environmental projects that were approved since 1994 and completed and evaluated by 2009.
We employ the standard proportional-odds logistic regression (i.e., "ordered logit") to link the hypothesized predictor variables to the project effectiveness rating. In order to reduce unnecessary noise in the dataset and increase estimation efficiency, we consolidate the ratings into three ordered categories of theoretical interest. The first category indicates that a particular project was a unsuccessful (19% of sample), and includes any project rated as highly unsatisfactory or unsatisfactory, and any project that was cancelled or subjected to the Inspection Panel process. The second category indicates projects that had marginal performance, with marginally unsatisfactory or marginally satisfactory ratings (17% of sample). The third category indicates that a particular project was successful (64% of sample), and includes any projects rated as satisfactory or highly satisfactory. For the sake of simplicity, we will refer to these three categories of projects as "successful," "marginal," and "unsuccessful."
Methodological Contributions:
The existing literature on aid effectiveness has mostly focused on identifying relationships between total aid flows - an aggregate measure that includes everything from funding for highway construction to peacekeeping operations to biodiversity protection - and highly aggregated development outcomes (e.g. GDP growth). Much of this research has yielded ambiguous results, fuelling general criticism of development assistance (Bourguignon and Sundberg, 2007; Doucouliagos & Paldam, 2009; Easterly, 2003, 2007; Roodman, 2007). Yet, a small group of scholars has begun to coalesce around the notion that future research should evaluate the impact that specific types of aid have on narrowly-defined outcomes (Asiedu & Nandwa, 2007; Dollar & Levin, 2005; Dreher, Nunnenkamp, & Thiele, 2008; Findley et al. 2009; Finkel, Pérez-Liñán, & Seligson 2007). A focus on specific outcomes is appropriate for environmental projects in particular, as successful implementation of environmental projects may have very little to do with increasing broad measures of economic output.
A key advantage of this study is that we use the performance of individual World Bank environmental projects as our dependent variable. By using a non-aggregated unit of observation, we are not only in a position to test propositions that have been central to research on environmental aid, but can also contribute to the growing literature on the ability of aid to achieve targeted outcomes. An additional advantage is that the risk of reverse causality is significantly diminished when using micro-data - a problem that has plagued the macro-aid effectiveness literature (Dollar & Levin, 2005; Isham et al., 1997, 1999). Furthermore, because individual projects are assessed, implemented, and evaluated independently of one another, many of the strong independence assumptions built into standard econometric techniques are sufficiently met.
Initial Empirical Results:
Only two predictor variables had a robust relationship with environmental project outcomes across our ordered logistic regression models. First, the World Bank Institute's (WBI) government effectiveness index showed a strong and positive relationship with the success of the projects in our sample. This has been a common finding in the aid effectiveness literature (Burnside & Dollar, 2000; Collier & Dollar, 2002; Dollar & Levin, 2005; Wane, 2004). The results of our models suggest that the strength and quality of government institutions and a strong policy environment are just as important to achieving success in the environment sector as they are to other sectors.
Second, the more a project focused on achieving global outcomes (e.g. reducing biodiversity loss or combating climate change), the less likely it was to be successful. This finding supports the hypothesis that borrowing countries are most likely to ensure successful implementation of projects that meet local priorities, such as those in the water and sanitation sector. These types of projects directly address the needs of local constituencies, which are likely in a better position to require accountability from local policy-makers and government agencies.
In order to show the substantive impact of these variables on the probability of a project falling within each success category, we plot the predicted probabilities over the range of data for each variable. In the government effectiveness plot, the value of the global proportion variable is held at its mean, and vice versa. In both cases, the variables are modeled to have a strong substantive impact, in both cases approximately doubling or halving the probability of success over the range of the data.
Predicted probabilities of project success
Policy Implications:
This research project has confirmed one of the most widely held assumption in development finance - that the success of aid projects is strongly influenced by the governance characteristics of the recipient country. Recipient governments that have a reasonably well-functioning policy-making process and the capacity to deliver general public services are more likely to successfully implement environmental aid projects. Like more traditional development projects, environmental projects depend on host institutions at many points during design and implementation. It would be preferable to have data on the quality of public agencies that play a role in environmental management, but such data are highly limited across both countries and time.
We have also confirmed that the more an environmental project aims to achieve global outcomes, such as reducing greenhouse gas emissions or protecting biodiversity, the less likely it is to be successful. This finding raises a number of practical questions about recent calls for a massive increase in global financing for climate adaptation and mitigation projects in developing countries. In order to ensure that many of these projects are as successful as possible, it may be necessary to find projects that meet both local development priorities and global climate goals. It may also be useful to seek out opportunities to work with actors that have different incentives than borrowing governments, such as the private sector (IEG 2008). Our study also suggests that the now popular notion that the World Bank has a "comparative advantage" in financing global public goods should be viewed with a measure of caution (Birdsall & Subramanian, 2007; Gilbert et al., 1999; IMF & World Bank, 2007; Jayaraman & Kanbur, 1999; Kremer, 2006;).
Outputs:
"When Do Environmentally-Focused Aid Projects Achieve their Objectives? Evidence from World Bank Post-Project Evaluations." With Mark Buntaine. Under review at Journal of Development Effectiveness.