Project evaluation and Project preparation.

EVALUATION AND PROJECT PREPARATION A. EVALUATION AND PROJECT DESIGN  This chapter explains how evaluation can improve project design and planning, and can set the stage for evaluation activities throughout the project cycle. It begins by reviewing the steps to ensure that the project is addressing the relevant development problem and that it has a clearly defined purpose, as these two attributes are important for enhancing project performance and facilitating evaluation activities. The chapter also describes the evaluation products that are generated at the design stage of the project. It must be emphasized that, at the project design stage, some of the more vital aspects are: Establishing a clear understanding of the development problem; Building into the project design lessons from previous operations; and Setting the stage within the project design for effective evaluation both during the monitoring and ex-post stages. The above considerations do not replace, and are only complementary of the Bank's economic, financial and technical analyses. 1. Evaluation Framework for Project Design The Logical Framework is an evaluation tool that may be used at the ex-ante or project design phase of the project evaluation process. This handbook cannot do justice to all the details of Logical Framework Analysis, but a brief synopsis is provided. For further details, see Annex 1 of this handbook. 2. Preparing for the Development of a Logical Framework A central part of any evaluator's work is to determine whether a project has been successful in addressing the development problem it was designed to solve. In too many past projects this has been difficult because the development problem was never well understood when the project was designed, nor was a link established to the solution provided by the project. 3. The Stakeholder or Interested Party Analysis This analysis clarifies which groups are directly or indirectly involved in a specific development problem, their respective interests in relation to it; their perceptions of the difficulties related to the development problem; the resources (political, legal, human, and financial) they may contribute toward resolving the development problem; their respective mandates with regard to the problem situation; their reactions to a possible project strategy; and the existing or potential conflicts among stakeholders. The stakeholder analysis is an important source of information for the evaluation of the project during its execution, and thus it is important to identify the stakeholders and understand their roles in the execution of the project. 4. The Problem Tree The problem tree is an important aid to understanding the development problem. It considers the negative conditions perceived by the stakeholders in connection with the development problem. It arranges the principal problems according to their cause-and-effect relationships, thereby clarifying upon which objectives the project should focus. The definition of the chain of problems allows for improvements in project design, a better monitoring of the "assumptions" of the project during its execution, and, upon project completion, facilitates the task of the evaluator, who must judge whether these problems have improved or worsened as a result of the project. A simplified example of a problem tree is presented on Figure 3. It addresses the issue of a city bus service and identifies the cause and effect relationships between the principal problems. 5. The Objectives Tree The development problems identified in the problem tree are transposed into project objectives as part of the initial stage of designing a solution. The objectives identified as the outputs of a project become the means for addressing the identified development problem and provide a means to assess performance. An example of an objectives tree is on Figure 4, using the problems identified in the problem tree. B. THE LOGICAL FRAMEWORK The logical framework is one of the principal tools used nowadays by agencies for project design and planning. Developed for the U.S. Agency for International Development (USAID) in the late 1970s, the logical framework provides a matrix within which the evaluator can examine aspects of project performance at all stages of the project. The strengths of such a framework are that it provides:  a clear means-ends analysis of project inputs leading to outputs for set purposes in support of a goal specification of inputs and costs for project activities; objectively verifiable indicators of performance and sources of verification; specification of the key assumptions or risks underlying the project; and a framework for introducing lessons learned to be incorporated in future projects The logical framework is a tool that helps project designers better understand the problems they are trying to solve. The framework is based on two basic principles: first, cause-effect relationships between different parts of a problem which correspond to the four levels (or rows) of the framework which relate to activities (or inputs), components (or outputs), the purpose and the goal as the set of hierarchical project objectives; second, the principle of correspondence, which links the four levels of objectives to the measurement of attainment (indicators and means of verification) and conditions which may affect attainment (or assumptions.) 1. Vertical Logic The logical framework helps systematize and apply a rational approach to the design, execution and evaluation of projects. In Tables 3 and 4 the vertical logic postulates that if we contribute certain inputs we will deliver certain outputs: thus, there is a necessary and sufficient relationship between inputs and their corresponding outputs, as long as the assumptions are confirmed in reality. At the next level of vertical logic of the framework we again make a causal inference. If the project delivers those outputs (or components), and the assumptions hold, the purpose (a hypothesis) will be achieved (i.e., the outputs are necessary and sufficient conditions). Continuing to the final step, if the purpose is achieved, and the assumptions at the purpose level hold, we will have contributed significantly to the attainment of the goal (i.e., the purpose is necessary but not sufficient). 2. Horizontal Logic In practical terms, the horizontal dimension is a description of how project managers, Country Office staff responsible for monitoring the project, and evaluators measure the attainment of results expected at each level of objectives. In the Tables mentioned above, the second column includes what are called "indicators". These are predetermined, quantitative and qualitative measures that indicate the status of input or output delivery, the achievement of the purpose (project impact) or the extent of contribution toward attainment of the goal. The third column explains how they will be measured, by specifying sources of information and methods to be employed. The fourth column describes the assumptions or risks that must hold in order to ensure the achievement of the activities or products of each level, and to proceed to the next level in the hierarchy of objectives. 3. Indicators Indicators provide the focus for evaluation data collection, so they need to be specified carefully and sparingly. The logical framework must include indicators for all the important objectives, but, at the same time, indicators should be chosen carefully to ensure their measurability. In setting indicators, the quantity, quality and timing required to achieve the next level of objectives must be considered. These indicators are very important because they provide an operational response to the issues being addressed by the project. Furthermore, they provide a focus for data collection at the preparation stage and a map to guide project monitoring and evaluation later on. The former function is presented later in this chapter; the latter are the subjects of chapters IV, V, and VI. A general introduction to the logframe concepts is presented on Table 4. It is followed by a sample logical framework that builds on the example used for the problem and objectives trees presented earlier in this chapter. 4. Assumptions As mentioned in previous sections, for the "vertical" relationships to hold, project managers rely on the verification or confirmation of the establiched critical assumptions which, as stated previously, in the logical framework matrix are outlined on the far right column for each level. These assumptions are the critical factors not controlled by the project managers or the executing agency, but which influence its implementation and chances for success. Table 4 : THE STRUCTURE OF THE LOGICAL FRAMEWORK Objectives Indicators Means of Verification Assumptions GOAL The Goal is a statement of how the project or program will contribute to the solution of the problem (or problems) of the sector. The indicators at Goal level describe how the overall impact of the project shall be measured. They are specific in terms of quantity, quality, and time (target group and location if relevant). The means of verification are the sources of information that can be used to verify that the targets were achieved. They can include published material, visual inspection, sample surveys, etc. The assumptions indicate the important events, conditions, or decisions necessary for the sustainability in the long run of the benefits generated by the project. PURPOSE The Purpose is the direct impact to be achieved as a result of the Outputs produced by the project. It is a hypothesis about the impact or benefit that the project attempts to achieve. The indicators at the Purpose level describe how the direct impact of the project shall be measured. They should include targets reflecting the end of project status (EOPS). They are specific in terms of quantity, quality, and time (target group and location if relevant). The means of verification are the sources to which that the executor and evaluator can refer to see if the targets are being achieved. They can indicate that there is a problem and suggest the need for changes in project Outputs. They can include published material, visual inspection, sample surveys, etc. The assumptions indicate the events, conditions, or decisions that must occur in order for the project to contribute significantly to the achievement of the Goal. OUTPUTS These are the goods, services, and training that the project executor is required by contract to complete. They should be expressed as work completed (systems installed, people trained, etc.). The indicators for Outputs are succinct, but clear, descriptions of each of the Outputs that has to be completed during execution. Each should specify quantity, quality and timing of the goods, services, etc. to be delivered. This cell tells where an evaluator can find the sources of information to verify that the products/services contracted have been delivered. Sources can include site inspection, auditor's reports, etc. The assumptions are the events, conditions, or decisions that have to occur in order for the Outputs will achieve the Purpose for which they were undertaken. ACTIVITIES Activities are the tasks that the executor must carry out in order to produce each of the Outputs of the project and that denote costs. Activities are listed in chronological order for each Output. This cell contains the budget for each Output produced by the project. This cell tells where an evaluator can obtain information on whether the budget was spent as planned. It is usually the accounting records of the executing unit. The assumptions are the events, conditions, or decisions that have to occur in order to complete the Outputs of the project. Table 5 (a): LOGFRAME FOR IDB PROJECT DESIGN AND EVALUATION Example: Transportation Project Case COUNTRY: PROJECT PLANNING TEAM __________: PROJECT: PROJECT NO.: ESTIMATED PROJECTG START DATE: ESTIMATED PROJECT COMPLETION DATE: NARRATIVE SUMMARY INDICATORS MEANS OF VERIFICATION ASSUMPTIONS PROJECT GOAL Use of the CBS by the population increases. l Number of passengers increases from X0, in base year, to X3, at the end of the year 3, X4 at the end of the year 4, X5, by December 2002 and X6, by December 2003. l The number of passengers complaints decreases from B0, in base year, to B3, at the end of year 3, B4, at the end of year 4, B5, by December 20002 and B6, by December 2003. l Audited City Bus Company statistics reported to the City Council. l Results of passenger surveys. PROJECT PURPOSE The service offered by the CBS is reliable. l The rate of accidents decreases from Y0, in base year, to Y1, at the end of year 1, Y2, at the end of year 2, Y3, at the end of year 3 and Y4, at the end of the project (December 2001). l The number of delays (+/.5 minutes) drops from Z0, in base year, to Z1, at the end of year 1, Z2, at the end of year 2, Z3, at the end of year 3 and Z4, at the end of the project (December 2001). l City Department of Highways statistics. l Statistics from the City Police Department. l Audited City Bus Company statistics reported to the City Council. (l The relative price of the gas is stable). PROJECT OUTPUTS 1. Drivers drive carefully. 2. Buses are in good condition.    3. Scheduling and use of buses is optimized. l Infractions of safety regulations decline from S0 in base year, to S1 at the end of year 1, S2 at the end of year 2, S3, at the end of year 3 and S4, at the end of the project (December 2001). (1) l Breakdowns decline from J0, in base year, to J1, at the end of year 1, J2, at the end of year 2, J3, at the end of year 3 and J4, at the end of the project (December 2001). (2) l City residents living within 1/2 mile of rush-hour bus stops increases from R0, in base year, to R1, at the end of year 1, R2, at the end of year 2, R3, at the end of year 3, and R4, at the end of the project (December 2001). (3) l Audited City Bus Company statistics reported to the City Council. l Audited City Bus Company statistics reported to the City Council. l Baseline data from the census; updated from the monthly current population survey carried out by the National Statistical Office. (l Special bus lanes are approved by the City Council.) PROJECT ACTIVITIES 1.1 Train bus drivers. 1.2 Introduce incentives to drive carefully. 1.3 Improve working conditions. 1.4 Introduce safety regulations and inspection system. 2.1 Store repair equipment and parts. 2.2 Improve the repair facility. 2.3 Establish schedule for replacement of buses. 3.1 Optimize the routes and scheduling. 3.2 Equip buses with radios for communication. 3.3 Establish a communication station at the central bus terminal. 3.4 Collect statistics regarding compliance with schedules and safety regulations. 4.1 Undertake continuing passenger survey program. B U D G E T BUDGET EXECUTING DOCUMENTS (l Adequate road maintenance and reconfiguration by the City Public Works Department.) (l The drivers union is in agreement with the project strategy.) (l Import duties on vehicle parts do not increase.) (l Revenues from bus fares are sufficient to allow replacement of buses.) (l The City Council approves the revised scheduling and routes.) C. EVALUATION AND PROJECT APPROVAL Everyone involved in project approval should ensure that each new project will be able to benefit from future evaluation processes. An evaluability assessment must be incorporated at the design stage, and project documents submitted for approval should include the Logical Framework and other elements that ensure "evaluability". "Evaluability" is a review undertaken by the project team to assess the extent to which the design described in project documents is able to support monitoring and evaluation activities. An evaluability assessment will:  support the design team in ensuring the project is of the highest quality ensure that logical framework standards are being adhered to ensure that the project plan provides adequate criteria for monitoring and evaluation assess the extent to which lessons and best practices have been incorporated The following guidelines should be used for this evaluability assessment. Table 6: Evaluability Guidelines Evaluability Requirements Paragraph No. Objectives Problem or need that the project is attempting to address has been identified and analyzed. Identification of whose problem or need it is. Causes of the problem or need have been identified and ranked. Expected objectives have been consistently defined. Lessons learned from previous operations and evaluations have been taken into account. Indicators Conditions (physical, institutional, economic, social) prior to the execution of the project have been described. Baseline data on the conditions prior to the execution of the project have been included. If no baseline data provided, project design includes data collection. Benchmarks, target figures, or other evidence to monitor progress and determine attainment of the objectives have been provided. Outputs Goods or services that the project will generate have been identified and described. Description of how and when the beneficiaries will use the goods and services generated by the project has been provided. Benefits to be derived from the use of the goods and services generated by the project have been identified. Assumptions Individuals, groups, institutions, and organizations that could positively or negatively affect the execution of the project have been identified. Events or elements that are outside the direct control of project management and that could affect project viability, outputs and objectives have been identified and described. Provisions have been made to review financial and economic feasibility analyses if the project experiences implementation delays which will negatively impact the indicators of project success. Table 7: PROJECT STRUCTURE: PERFORMANCE BENCHMARKS PROJECT CYCLE STAGE TYPES OF BENCHMARKS PROJECT ISSUES EXAMPLES OF INDICATORS MEDIOS DE VERIFICACIÓN CONDITION AT PROJECT START-UP SITUATION: to be changed as a result of the project and its objectives (ex-ante). (For Logframe users: refers to Stakeholder and Problem Analysis) BASELINE BENCHMARKS: or zero of the project indicator system. (a) problem or need identified and analyzed. (b) causes of the problem identified and analyzed. (c) data on initial conditions of the problem set (physical, economic, social, gender, financial, institutional, environmental, etc.) (d) identification of project stakeholders (i.e. population that will benefit from the project, interested public and private institutions, public and private institutions that could become impediments to the project) and listing of assumptions on behaviour of stakaholders and/or events that could affect project execution and/or development impact upon completion. (e) Lessons learned from previous operations. (a) high mortality of children between ages of 0 and 5 in rural areas, of which "x" % with skin infections,"y" % with gastrointestinal diseases, etc. (b) (1) sanitary conditions in hospitals; (2) sanitary conditions in the home; (3) children's malnourishment; (4) low level of education of mothers. (c) (1) "x" number of hospitals without basic sanitary modules in children's wards; (2) "x" % of nursing staff with only "y" number of years of training; (3) "x" % de mothers with only "y" number of years of primary schooling; (4) "x" % of children under five years with only "y" level of nutrition, etc. (d) (1) women's employment must increase in the region/country/city, etc; (2) Health Ministry functions will be decentralized; etc (e) (1) no new hospitals are needed -- rehabilitational improvement of existing is sufficient to attain project outcome; (2) no new physicians need to be trained -- training of nursing staff is sufficient; etc. Specific means of verification must be identified for all indicators (i.e. hospital and school records, social and demographic sources,etc). With the timeliness and periodicity required. If these sources do no exist the generation of validating information will have to be introduced as a project activity. PRODUCTION OF COMPONENTS AND PROJECT PERFORMANCE OUTPUTS: or products to be generated as a result of project activities during execution and at completion, or final disbursement (ex-dure). (For Logframe users: refers to Activity and Component levels) MONITORING BENCHMARKS: (a) measure the progress and efficiency of project execution (by adhering to expected schedules and products) at mid-term and completion stages, (b) assess the behavior of critical assumptions as they may affect project execution. (a) goods and services the project will deliver mid-term upon completion. (b) assumptions on behavior of stakeholders and events that are outside the direct control of project management. (a) Mid-term: (1) "x" number of hospitals rehabilitated at "y" water quality level by years "t1", "t2", "t3", etc; (2) "x" number of nursing staff trained in "y" quality techniques by years "t1"..., etc.; (a) Upon completion: (1) a total of "x" hospitals rehabilitated and capable of sustaining "y" quality standards of medical attention to the 0 to 5 year age grupo by year "z"; (2) a total of "x" nursing staff trained at "y" technical standard and capable of assisting in training of other nurses by year"z"; etc. (b) (1)"x" number of hospitals budgets are adjusted to "y" level by years "t1"...etc. to facilitate rehabilitation programs; (2) "x" number of schoold budgets and teaching programs are improved by "y" levels to facilitate training of mothers by years"t1" ...etc. PERFORMANCE DEVELOPMENT IMPACT OUTCOME: or desired situation after project completion as products are used by beneficiaries (ex-post) (For Logframe users: refers to Purpose and Goal levels) TARGET BENCHMARKS: measure the use and effectiveness of project products and assess the behavior of assumptions that could affect project development impact. (a) Use beneficiaries will make of the goods and services of the project and benefits they derive; (b) Solution, or contribution to the solution, of the problem identified at situation stage above; (c) List of assumptions on behavior of events in the hands of the borrower/beneficiaries, or beyond their control, after project completion (final disbursement). (a) (1) "x" % of hospital staff uses "y" % improved basic services in "z" number of hospitals; (2) "x" % of better trained nurses provide care to "y" level to "z" number of children within target age group per year; etc. (b) decrease of child mortality in rural areas: (1) rate of infections of children in target age cohort falls from "x" to "y" % between years "z" and "v"; (2) number of hospitalizations of children in target age group falls from "x" % to "y" % between years "z" and "v"; etc. ( c) (1) all assumptions of project completion stage continue to operate and project sustainability is attained by beneficiaries contributions to upgraded hospital, training and nutrition programs. D. BASELINE BENCHMARKS These benchmarks, and their corresponding indicators (see Table 8), are most important and are intended to provide a picture of the situation before project intervention. They describe the situation by quantifying the levels of selected indicators which can be re-visited later on. Changes in the levels are expected to have a plausible link to the effects of the project. Baseline benchmarks provide a basis for the measurement of change and test the reliability, validity and feasibility of certain types of information upon which monitoring and evaluation can be built. E. ESTABLISHING BASELINE BENCHMARKS Unless they already exist, baseline benchmarks may have to be collected for at least some of the indicators identified in the project logical framework. The following checklist can be used to identify necessary baseline data and to indicate when it should be updated. It often happens that when projects are evaluated, at mid-term and ex-post levels, appropriate data are difficult, impossible or too costly to acquire. Careful attention at the preparation stage can often do much to alleviate these problems. The first step in completing the checklist is to identify the status of data on each indicator: What data are available now? Should additional baseline data be collected before the project is implemented? Will data be required for this indicator for project monitoring activities? Will data for this indicator be required for mid-project, end-of-project, and/or impact evaluations? The following checklist has been completed for two of the indicators identified in the logical framework presented earlier in the Chapter. Table 8: Baseline Benchmark Checklist Indicators Baseline Data When will updates of data be required? Data Available Data to be Collected Monitoring Evaluation Mid BEP Impact Number of Passengers using CBS 2,317 fare-paying passengers, August 1994 (CBS statistics) N/A N/A ü ü Reliability of CBS service Require data on delays (+/- 5 minutes) for representative sample of rush-hour bus stops ü ü ü  SUMMARY POINTS Through its contribution to a sound logical framework and integration of lessons learned, evaluation is a fundamental part of project design. Evaluability assessment provides decision-makers with vital information for those in charge of project approval and those involved in the project's execution. Baseline data are essential for sound planning and subsequent evaluation.