Comprehensive impact assessment systems for NGO microenterprise development programs

WorldDevelopnzerzr,

Vol. 26, No. 2, pp. 261-276, 1998

0 I998 Elsevier Science Ltd All rights reserved. Printed in Great Britain 0305-750X/98 $19.00+0.00

Pergamon PII: SO305750X(97)10022-5

Impact Assessment Systems for NC0 Microenterprise Development Programs

Comprehensive

ERIC L. HYMAN

Appropriate

Technology International,

Washington, DC, U.S.A.

and KIRK DEARDEN”

JHPIEGO

Corporation and Johns Hopkins University, Baltimore, Maryland, U.S.A.

- This paper reviews the monitoring and evaluation systems of four nongovernment organizations (NGOs) providing business development services to microenterprises in developing

Summary.

countries. It compares their indicators. While the diversity indicators, the paper provides which such systems could be Kev nor& ~ monitoring. informal sector

approaches to data collection and analysis, uses, and types of impact of microenterprise programs makes it difficult to specify a standard set of examples of the questions that can be addressed and suggests ways in improved. 0 1998 Elsevier Science Ltd. All rights reserved

aid evaluation,

impact assessment,

I, INTRODUCTION

indicators,

NGOs, microenterprise,

This study is based on the survey responses by the representatives, follow-up phone organizations’ interviews and correspondence, and a review of documentation on the systems. The full survey findings, a companion review of loan tracking systems, and detailed descriptions of both types of impact monitoring systems are reported elsewhere (Hyman and Dearden, 1996). The four NGOs employed a wide variety of strategies to assist microenterprises and all had multiple developmental objectives. AT1 and Tech-

This study reviews comprehensive impact tracking systems of four nongovernmental organizations (NGOs) that support or implement business development programs beyond credit alone for microenterprises in developing countries. It was conducted for a project funded by the US Agency for International Development (USAID) on Assessing the Impact of Microenterprise Services (AIMS). The purpose of this study was to identify existing practices before development of new tools and indicators under AIMS. This analysis followed a survey of the 41 NGOs based in the United States or Canada belonging to the Small Enterprise Education and Promotion Network (SEEP) and consultations with AIMS project staff and USAID. The four organizations whose comprehensive impact assessment systems were reviewed are (a) Appropriate Technology International (ATI), (b) Freedom From Hunger (FFH), (c) Save the Children (SC), and (d) TechnoServe (TNS). The authors selected these systems as being representative, relatively well developed, or instructive for other organizations seeking to establish or improve their systems.

noServe

emphasized

business

planning,

technology

“This research was funded by the project on Assessing the Impact of Microenterprise Services (AIMS) of USAID’s Office of Microenterprise Development (PCE-0406-C-00503600) conducted by Management Systems International, in cooperation with the Harvard Institute for International Development, the University of Missouri, and the Small Enterprise Education and Promotion Network. The contents do not necessarily reflect the view of USAID or the AIMS Project. The authors would like to thank Carolyn Barnes, Elaine Edgcomb, and Monique Cohen for guidance. Helpful comments were provided by Janney Carpenter, Henry Oketch, Jennefer Sebstad, Victoria White, and two anonymous reviewers. Final revision accepted: August 18, 1997.

261

262

WORLD DEVELOPMENT

commercialization, organization of production and processing, and market development. Both provided or arranged for some loans together with other types of assistance, but credit was not their primary focus. Freedom From Hunger combined credit for women’s income-generating activities with education on nutrition and health. Save the Children offered loans and training for microenterprises with the objective of improving the nutrition, health, and education of children. Save the Children and TechnoServe also supported some community infrastructure development.

2. COMPARISON

OF THE SYSTEMS

Table 1 compares the characteristics of the four systems based on survey responses from the organizations and the authors’ subjective judgment. In mid-1996, Save the Children had just begun field testing its system and Freedom From Hunger and TechnoServe were in the process of finalizing changes to their systems. ATI had added several new impact indicators to its system earlier in the year. The four organizations had hybrid systems for monitoring outputs and each measured aspects of impact while reserving more extensive impact assessment for less frequent evaluations or special studies. AT1 and TechnoServe applied their systems to all major projects while Freedom from Hunger and Save the Children only tracked client-level impact in selected programs. All relied on both quantitative and qualitative data, although the mix of methods varied. The most common sources of data were program records and client surveys, but others included focus groups, loan applications and renewals, and key informant interviews. Differences in impact indicators across organizations reflected the diversity of their microenterprise programs and institutional mandates. All were concerned with financial and/or economic impact. ATI assessed environment and natural resource impact because of its efforts in sustainable ecodevelopment. Freedom From Hunger and Save the Children tracked nutrition and health impact because their programs combined microenterprise development with interventions in these areas. TechnoServe’s main focus was development of communitybased enterprises. All of the systems addressed impact on client enterprises and at least partially addressed impact on the households of owners and workers and other enterprises; but the number of beneficiaries cannot be easily compared across organizations because of differences in definitions. For example, ATI only counted participating producers themselves, while many other SEEP members surveyed counted all the

members of the producers’ households as beneficiaries (usually by incorporating a gross assumption of average household size). All four of the systems reviewed in detail considered broader impact within the local economy. but this was often limited to backward and forward linkages to related enterprises not directly assisted by the programs (those supplying equipment, raw materials, or other inputs to the assisted enterprises or processing or selling their products). Save the Children and TechnoServe were concerned with community-level impact in addition to impact on enterprises and households. All four organizations hired consultants to implement some data collection activities. In some cases, field staff also collected data, either by themselves or supervised by external consultants or headquarters staff. All four collected longitudinal data and used a standard set of impact indicators that had been adopted or was being tested. ATI and Freedom From Hunger, however, noted the necessity of tailoring client survey questionnaires to specific projects to gather this information, rather than applying the standard list of indicators directly. Except for pretesting, the NGOs did not routinely involve clients in evaluation design or data collection. Table 2 presents the types of indicators currently in use (or proposed) for the reviewed systems and reflects revisions based on the organizations’ prior experience with impact indicators. For example, ATI had dropped some indicators that proved burdensome for field staff or did not generate useful information, such as person-days of labor time generated by projects. Comprehensive impact assessment systems devoted more resources than simple loan tracking systems to estimating net enterprise income, total household income, and household expenditures. All monitored the gross income gains of assisted enterprises. ATI and TechnoServe focused on the value added to the national economy-the sum of net income gains to the enterprise and payments to workers and raw material suppliers (to the extent that imported inputs were not used). All devoted some attention to the net income gains of the enterprises, at least in evaluations or special studies if not in more frequent monitoring. The four systems frequently considered changes in enterprise assets, but some only counted major fixed capital assets because working capital and smaller fixed capital assets are harder to measure. Most did not measure household income separately from the income generated by the enterprises directly or indirectly assisted. Changes in household productive assets or housing and material goods consumed were not usually addressed. Three of the systems examined changes in

Annually

Timing: Frequency

Bretrdth: Client\ (assisted enterprises) Households of owners and workers of assisted enterprises Other enterprises (not directly assisted) Communities

U.\es ,for decision making: Changes in program design Resource allocation Strategic planning Design of new projects and programs Decisions about compensation or personnel Selection and change of partner organiLationa

Secondary

users

I

quarterly; irregularly year

Impact

Partially Partially

Partially Partially

Partially Partially

Partially

Fully Partially

No

Pat-tially

No

YCS

No

Fully Fully

No

NO

Yes

Fully Fully

No

Yes

Yes Yes Yes No

TechnoServe

proposed)

and field staff

2 years; Impact 4 years

(semiannually

Headquarters

Monitoring: assessment:

Quarterly

Fully Fully

Fully Partially

Yes No YCS Yes

Headquarters. regional, national, and local staff Donors and development professionals

Annually or more often (determined by the field) Still in testing stage

Fully Fully

systems’

Save the Children

impact assessment

No Ye\

Yes

Academicians

Field staff and partner donors

Monitoring: assessment: Less than

Fully Fully

Freedom From Hunger

of selected comprehensive

Yes Yes Yes Yes

Headquarters, field staff, and USAID Donors and development professionals

Principal users

Use W

3 years

Time system has been in use (as of mid-1996)

of updating

Fully Fully

Monitor programs Assess impact

Purposes:

Table I. Characte’stics

Appropriate Technology International

field staff and staff

Yes

Yes

Special studies only

Yes

No

Yes

yes; Impact: no

Monitoring:

Yes

and field staff

Yes Consultants

Yes

Yes

TechnoServe

Yes Yes

Special studies only

Yes Yes

and field staff

Save the Children

Consultants

Yes

Yes

Yes

No

Yes

Yes

Yes

yes; Impact: no

Monitoring: Yes

From Hunger

and field staff

Freedom

Consultants

Yes

Yes

Yes Consultants, headquarters Yes Yes

Yes

Yes

“Indicates no information was available Source: Hyman and Dearden (1996)

Population for datu collection: Clients Nonclients within the same communities Nonclients from nonprogram communities Social and demographic data

Longitudinal data collection Data disaggregate by gender

Rigor: Computerized system Type of data collectors

Types of tools Quantitative Qualitative

Appropriate Technology International

Table 1. Conrinued

COMPREHENSIVE

IMPACT ASSESSMENT

SYSTEMS

265

progres\

infrastructure

or

Indicator

resources;

Regulatory recognition of programs (SC): Changes in policies affecting microenterprises (SC, ATI) Change in access to electricity, water, sanitation services (SC, TNS)

Positive and negative effects on environmental/natural Value of impact (ATI)

Solidarity; Relations in community; Child care, Food security: Significant changes in communities: Whether borrower had to take a loan from another source to pay back program loan (FFH): Time allocations of participant and spouse; Household dcclsion making; Membership and leadership in community organizations; Aspiration\ for boys and girl\; Evidence of group \elf-managcmcnt (SC)

Potable water; Medical facilities (TNS); Self-reports on whether client has di?cussed/learned valuable health/nutrition information: Self-reports on health/nutrition practices clients tried or told others about; Health/nutrition knowledge; Diet; Food security (FFH): Actual nutritional status (FFH, SC) Estimated illiteracy mte in project area (TNS); Number and percent of childrenin school; Gap between male and female child enrolment rates (SC); School availability (TNS) Change in amount spent on food. clothing, and other basic needs (FFH, SC); Increased spending on purchases for self; Value of last item purchased cxclusivcly for children from cntcrprisc profits (SC); Uses of increased income (FFH) Electricity (TNS)

Number of consumer households saving money as a result of the project and the value of the savings (ATI)

“indicates no information was available Source: Hyman and Dearden (1996).

Community services

Policy impact

C. Community Level Environmental or natural resource impact

Housing or other material goods conturned Empowerment

Expenditures

Educational

Health (nutrition, water supply, sanitation)

B. Household Level Net income Consumer savings

Type of impact

used

to clcctricity

(TNS)

Quantity of environmental impact or natural resource use without the project compared to that with the project. valued in monetary terms whcrc possible (ATI)

Member5 dixuh\ing ahsoclation problems. taking actlon, or assisting one another with family emergencies or repayment difficulties; Compatibility of activities with child care (FFH); Questions about decision making (SC)

Accc\s

Reported

May have been merged with enterprise income Consumer savings may be due to lower prices for a good or service or incrcascd cfficicncy of use and do not represent total expenditures on the good or service (ATI) Adequate povahle water source in community (TNS): Greater medical facilitic\ in the community (TNS); Self-reporting of whcthcr client and family were able to eat more and better food; Health/nutrition practices borrowers have tried or told others about (FFH): Mid-upper arm circumference; Height, weight and age (FFH, SC) Adequacy of school in community (TNS)

How defiined/Que,tion

2

COMPREHENSIVE

IMPACT ASSESSMENT

employment in the assisted enterprises. Only ATI considered displacement of labor from other enterprises and that was just in selected cases. Three of the organizations tracked assisted enterprises’ access to financing from sources other than their own programs. Only Save the Children proposed monitoring changes in enterprise management practices. Three organizations planned to address sustainability of income gains in the future by continuing to monitor impact after project assistance ends. ATI also assessed the proportion of assisted enterprises of proven sustainability while the project was still underway (defined as enterprises operating profitably without any subsidized project assistance during the year). Two organizations examined impact on empowerment of households within the community and individuals within the household. Three incorporated health indicators (including nutrition, access to water, and sanitation supply) and two monitored education of client household members and development of community services. Three reported on policy impact when it could be readily observed. Most NGO microenterprise programs can only afford to do relatively small sample surveys due to the cost and time involved in data collection and analysis. As a result, evaluation surveys have mainly been confined to project participants and sample sizes commonly ranged from 2.5 to 100 respondents. ATI, Freedom From Hunger, and Save the Children have done some evaluation surveys with relatively large sample sizes (ranging from 100 to 500). SC has surveyed samples of 25.000 or more individuals in a few special studies, but these were exceptional cases. Even when stratified sample sizes were large enough to allow statistical testing, none of the organizations regularly tested the statistical significance of data. In some projects. AT1 and TechnoServe have established medium-scale processing enterprises that benefit farmers and other raw material producers. The processing enterprises then maintained business records and collected data on the raw material suppliers that were directly useful in monitoring and served as a sampling frame for randomly selecting suppliers to survey. None of the organizations regularly surveyed nonparticipants in program areas, but some did occasionally. For example, ATI surveyed some farmers in the project area who had not purchased treadle pumps to learn about reasons why they did not adopt the technology. None of the four organizations surveyed nonparticipants in nonprogram areas. In some special studies, FFH collected data from comparison groups using smaller sample sizes or collected data at several points in time. Some organizations stratified samples by the types

SYSTEMS

26-l

or amounts of assistance provided. For example, ATI administered different questionnaire modules for alpaca herders depending on the services they received: (a) technical assistance, credit, and marketing; (b) credit and marketing; or (c) marketing alone. All of the organizations collected a mix of quantitative and qualitative data. While AT1 generally included open-ended questions in surveys to gain information about perceptions, opportunities and constraints, and suggestions for improving project services. it mainly relied on quantitative measures of impact. FFH had emphasized qualitative data the most, but is moving toward more quantitative indicators. It has also used anthropometric measures of health. ATI has written guidelines for its impact tracking system in English and Spanish and trained consultants in various countries in survey techniques (Stosch and Hyman. 1997). Save the Children placed a high priority on strengthening institutional capacity by involving headquarters and field staff in hypothesis generation as well as selection of indicators and research methods. It also trained staff and local research institutes in data collection. All four organizations reported that impact findings often led to changes in the design of the evaluated projects. Three relied on impact information in strategic planning and design of new projects and programs. and two in resource allocation decisions. Only AT1 reported that impact information affected decisions on compensation and personnel issues or selection of partner organizations. All four organizations noted that headquarters and/or field staff and project partners were the principal users of impact information. Three listed development professionals or academicians as a secondary audience, and two mentioned donors. TechnoServe was an early leader in electronic data exchange among field offices. Save the Children plans to actively involve staff, clients, and local community members in discussing findings of special studies.

3. DETAILED DESCRIPTIONS THE SYSTEMS

OF

ATI first instituted an annual Impact Tracking System (ITS) in 1993 to supplement midterm and final evaluations of major projects with more timely information for decision making and allow comparison and aggregation of indicators across projects. In 1994, the ITS was substantially revised to focus on the most critical information needed, eliminate items

268

WORLD DEVELOPMENT

that were difficult to monitor annually, clarify definitions, and make the database form more user friendly. ATI added several new indicators in 1995 to address the interest of its major donor, USAID. It also grouped organizational performance indicators under four objectives: outreach and impact; sustainability of impact; funding diversification; and cost effectiveness. Some had associated targets while others were just tracked. Only minor formatting changes were made for 1996. An annual Program Report organized by program area described each projects objectives and targets. outputs, factors affecting implementation and impact, and lessons learned (Hyman and Stosch. 1996). Quantitative data on project inputs. activities, and impact were compiled in a computerized database that allowed analysis by project, geographic region, program area, and the organizational portfolio. An annual Impact and Learning Report analyzed the database information, addressed the organization’s performance indicators, contained a synthesis of the overall lessons learned, and noted any replications of project technologies or services outside the project area (Hyman, Stosch and Budinich, 1996). Data for the annual impact tracking came from new surveys, extrapolations from previous surveys, project records, and staff or consultant estimates. Some projects had donor-funded budgets that allow annual impact surveys, but most did surveys less frequently. In selected cases, ATI has continued to track impact after project funding ended. Impact was assessed through comparisons with baseline data from subsector studies, feasibility studies. business plans, and special surveys conducted early in project implementation. The database recorded project budgets and expenditures, donor and project partner involvement. and expected dates and completion dates for baseline data studies and evaluations (all monetary values were converted to US dollars at the end-of-year exchange rate). It also monitored enterprise financing from project and nonproject sources, including the annual and cumulative number and value of project loans, loan repayments due and repayments received, and cash available for lending. Where projects made equity investments in enterprises, the ITS tracked the number and value of the investments and the returns the project received from dividends and sale of ownership shares. The database also listed the main technology interventions, number of enterprises producing and using the technology, cumulative number in use, number of new units in use, number of units that went out of use, and unit cost. AT1 monitored outreach through the number of assisted enterprises, producer participants earning more income, and consumer households saving money from lower product prices or more efficient

product use. The ITS subdivided assisted enterprises into those newly assisted this year; that stopped operating this year: owned by individualsipartnerships versus groups; with IO or fewer workers versus I I or more; and classified in the informal sector. formal sector, or farming. AT1 only counted producer participants earning at least $20 per year in additional income as a result of the project. People receiving project training or credit were not counted unless they had income gains exceeding that amount. Producer participants included owners of individual or group enterprises and their workers and principal raw material suppliers. The number of producer participants in each category was disaggregated by gender. ATI only counted the producers themselves and not their whole families. The ITS defined economic participants to include producer participants and consumer beneficiaries, but only one person per household was counted as a consumer beneficiary (the main bill payer). The $20/ year rule did not apply to consumers because their savings is typically smaller, but consumer beneficiaries were also counted separately from producers. ATI’s measure of economic impact, total monetary benefits (TMB) equalled the sum of producer income gains (producer cost savings, incremental gross value of final goods and services sold, plus the value of final goods consumed by the producers’ own households). and the value of consumer savings resulting from the project. The producer income gains incorporate the incremental net income earned by the enterprise owners and the payments to workers and raw material suppliers. The ITS also tracked environment or natural resource impact in projects where that was anticipated. ATI relied on proxy indicators of sustainability of economic impact that could be applied while projects were still underway. It defined an enterprise of proven sustainability as one that continues to operate and recover the full direct costs of production and distribution. either without any support from ATI or its project partners in the current year or with full recovery of the operating costs of any project services received. Sustainable enterprises did not necessarily have to recover their fixed costs or sunk development costs and firms that only received project assistance in a prior year may qualify. ATI estimated the subset of economic participants and total monetary benefits associated with proven sustainable enterprises. It also used two other proxy indicators of sustainability: private sector investment in enterprises and cost recovery by projects or related service enterprises. The ITS monitored progress toward the organization’s funding diversification objective through four indicators: cumulative noncore donor commitments/ cumulative USAlD core commitments: noncore

COMPREHENSIVE

Table 3. ATI’s cost-effectiveness

IMPACT ASSESSMENT

indicators

Project-Level 1. Cumulative net project cost/enterprises assisted 2. Cumulative net project cost/producer participants 3. Cumulative TMB/cumulative net project cost 4. Net annual project cost/new enterprises assisted 5. Net annual project cost/change in producer participants 6. Net annual project cost/change in TMB.

analysis incorporated several scenarios with conservative assumptions that no further project expenditures would be made, no additional enterprises assisted, and some existing enterprises would close. The net present value of benefits was recalculated annually.

(b) Freedom

Porrfolio-Level 1. Cumulative 2. Cumulative

TMB/cumulative TMB/cumulative

Source: Hyman,

Stosch and Budinich

core funds received total donor funds spent (1996).

commitments from LJSAID; overhead cost recovery rate. and unrestricted income. ATI defined project cost as total expenditures from all funding sources (ATI’s own contributions,

donor funds supporting ATI or its project partners, and partner counterpart support). It defined net cost as project cost less cost recovery from clients (whether obtained by ATI, a project partner, or related service enterprises). Table 3 contains the cost effectiveness indicators used by ATI. Because enterprise development activities continue to generate benefits in future years, AT1 conducted a cost-benefit analysis annually projecting economic impact forward 10 years. The cost-benefit

269

SYSTEMS

From

Hunger

As of mid- 1996, Freedom From Hunger had done limited impact monitoring of all of its Credit With Education programs quarterly for two years and special impact studies of selected programs for four years. The special studies typically included small surveys of participants and nonparticipants, but some larger surveys, qualitative investigations, and reviews of specific programmatic issues were also done. FFH’s partner organizations gathered baseline data through surveys of women’s health and nutritional knowledge and practices and a credit assessment examining women’s economic activities and prior credit experience. Partner organizations were also responsible for financial monitoring of credit programs and the performance of each credit association. At the first meeting of each loan cycle, field agents collected information collected on the participating credit associations’ finances, loan

Table 4. indicators used by Freedom From Hunger Program Participants Number of members and borrowers Number of new members ~ Member dropout rate Number of members who are pregnant or lactating Number of members with children