by Lisa A. LaCava

Part 3 of a 3-Part Series

In this last article of our three-part series on evaluation as a management and learning tool, we focus on outcome evaluation. Management Zone readers, including foundations, want to know:

What is the CYD movement trying to achieve, generally? What difference is CYD making? How does my organization or program fit into the CYD outcome landscape? What does success look like?

Our readers describe common traps that programs fall into, which result in outcome evaluations that are far less useful than they could be. Given the day-to-day stress of work and the pressure to prove that programs work, programs are often prey to traps that result in outcome evaluations that are less useful than they could be. For instance:

• Delegating outcome evaluation, including defining outcomes, to a third-party evaluator
• Accepting funders' definitions of outcomes
• Being confused or intimidated by outcome evaluation terminology
• Collecting what can be easily measured, whether it is related to core outcomes or not

Based on lessons learned from research and experience, the following sections outline four useful recommendations for avoiding these common traps, and for making outcome evaluation as useful and usable a process as possible.

We should be on guard not to overestimate science and scientific methods when it is a question of human problems; and we should not assume that experts are the only ones who have a right to express themselves on questions affecting the organization of society.

-Albert Einstein

1. Be actively involved in defining outcomes.
Probably the most important way for CYD professionals to impact the outcome development and evaluation process is to be active players in defining and measuring outcomes. Ask yourself whose outcomes you are using to define success-your own definition, the participants' definition, an outside evaluator's definition, or the funders'? Make sure that all key stakeholders have had input into the outcome definition process.

2. Start with individual-level participant-centered outcomes . . .
When people think about outcomes, often they think about program goals. The problem is that program goals are usually stated in terms of service delivery or system goals (e.g., reducing the number of young women on welfare), rather than on clear outcome statements about how participants' lives will improve as a result of the program. What difference will this program or initiative make in the lives of those served?

There are multiple ways to reduce the number of young women on welfare (the stated outcome), but not all are equally beneficial to participants. The program might focus on quick-fix job placement for young women into low-skill, low-paying jobs. However, if what many participants need is a long-term program for skill building and support, this might not be the most appropriate or most beneficial program model. "Reducing the number of young women on welfare" is a program, system-focused outcome rather than an individual, participant-centered outcome.

A participant-centered outcome helps us focus on and measure what is truly important to improving the lives of young women on welfare. For example, the primary participant-centered outcome for this program might be: "Participants will gain life and job skills adequate to succeed in their chosen field," or "Participants will gain life and job skills necessary to be self-reliant and economically independent."

As you revisit your outcomes, ask yourself, "What are you really trying to achieve with the young people you work with? What changes do you hope to see in their lives as a result of your program?"

. . . But don't stop there.
A focus on participant-centered outcomes does not mean that other types of outcomes are not important to the CYD movement. In fact, the CYD movement is very much focused on making change at broader levels, including family- and community-level outcomes, program- and system-level outcomes, organizational-level outcomes, and policy outcomes-in short, social change.

Many CYD programs are aimed at impacting families, neighborhoods, and, in some cases, whole communities. These family- and community-level outcomes are inextricably linked to individual youth outcomes. Family outcomes might include improved communication, increased parent-child-school interactions, and keeping children safe from abuse. Community outcomes might include increased civic engagement and participation, decreased violence, shifts in authority and responsibility from traditional institutions to community-based agencies and community resident groups, or more intensive collaboration among community agencies and institutions.

In addition, program- and system-level outcomes speak to improved interagency partnerships and changes in the ways CBOs and other youth-related institutions operate and interact with youth. Policy outcomes address changes in youth policy at different levels of government, within the private sector, or within other partner and allied agencies. Organizational outcomes affect how well the program can achieve individual participant outcomes and organizational outcomes that will lead to improvements in program management and organizational effectiveness.


3. Demystify the jargon.
The technical language of outcome evaluation can often intimidate or confuse program staff and stakeholders. An important step in developing an effective outcome evaluation is to demystify this jargon. One example that quickly comes to mind is outcomes versus indicators versus performance benchmarks.

Outcomes and indicators are often confused as one and the same, when they are actually distinct concepts. "Indicators" are measurable approximations of the outcomes you are attempting to achieve. For example, self-esteem, in and of itself, is a difficult concept to measure. A score on the Coopersmith self-esteem test is an indicator of a person's self-esteem level. Yet, it is important to remember that the individual participant-centered outcome is not increased participants' scores for this test, but increased self-esteem. The Coopersmith test simply becomes one way to measure self-esteem. Similarly, a program may have constructed teacher assessments of a child's self-esteem to be administered quarterly. Here the indicator has changed from a standardized, norm-referenced test to a more open-ended, qualitative assessment of self-esteem; however, the outcome remains the same: increased self-esteem.

"Performance benchmarks" are targets that chart progress toward outcomes. Think of them as progress indicators that specify the level of outcome attainment you expect or hope for (e.g., the percentage of participants enrolled in post-secondary education; how many grade-level increases in reading ability). These benchmarks help clarify and provide specificity about where you are headed and whether you are succeeding.

4. Collect only the data you will use, and use all the data you collect.
What data you collect and how you collect it will depend on several variables:

• Your evaluation team's philosophical perspective regarding the most accurate measure of your stated outcomes.
• The resources available for data collection. (One example is time and labor intensity to administer and interpret student portfolios versus standardized achievement tests.)
• Privacy issues and how intrusive the data collection methods are.

Your team should also consider the current state of the measurement field, reviewing the indicators, if any, that currently exist for the specific outcomes you are attempting to measure. The key here is to collect the data that most accurately and effectively measures the outcomes you want to achieve.

Another common problem is that programs often underestimate what can be measured, and consequently attempt to achieve only those outcomes they know how to measure, or that are relatively easy to measure. Since the field of measurement of human functioning will never be able to provide an accurate and reliable measure for every outcome-in particular, complex human feelings-and since program staff and stakeholders have expertise in only a subset of existing indicators, this method is likely to exclude critical outcomes. Start with the overall goals and outcomes of the program, and then determine how to go about measuring these outcomes. From our perspective it is better to have meaningful outcomes that are difficult to measure, than to have easily measurable outcomes that are not related to the core of a program that will make a difference in the lives of those served.

No matter what data you choose to collect, ultimately, you want to ensure that the findings from your outcome evaluation process are useful. To this end, before you have even finalized data collection strategies, think through how you will use different outcome data and, depending on the findings, what specific actions you might take. This will increase the likelihood that you will focus on the critical outcomes, select the most accurate and meaningful indicators, collect the most appropriate data, and analyze and interpret the data in the most meaningful ways. In addition, it will increase the likelihood that you and your staff will act on what you find.

Author