by Lisa A. LaCava

Part 2 of a 3-Part Series

Recent questions from our readers reveal that many "Management Zone" readers are interested in how they can balance the call to "prove" their CYD approach is worthwhile with their need to continuously learn and improve their practice. Taken together, a composite question from the field looks like this:

I recognize both the strengths and shortcomings of many traditional approaches to evaluation currently being practiced. And although I understand the importance of being able to demonstrate the effectiveness of social programs and prove they work, I believe that too many promising CYD programs are never fully actualized or understood. It seems that not enough attention is paid to learning about how and why they work, or understanding the contextual conditions that support or hinder their growth and development.

What kind of evaluation designs/components should we consider for maximum community-wide learning? Will alternatives to the "experimental" all-numbers approach work in this politically charged environment that seems all about outcomes?

While it is certainly true that funders and other stakeholders are very focused on outcome accountability and "scientific methods," it is also true that there is often room for negotiation of outcomes and indicators as well as the means of measuring them over time. It is important to build our management capacity in the CYD movement so we can create local evaluations that not only help demonstrate the effectiveness of the project, but also help us improve and strengthen it. In our evaluation capacity-building work with the W.K. Kellogg Foundation, we identified three recommendations for a "better balance":

1. Learn about and reflect on alternative research paradigms and methods that are appropriate to our work.

2. Question the questions and value different perspectives.

3. Take action to deal with the effects of paradigms, politics, and values embedded in all research and evaluation work.

Alternative Paradigms
At least two alternative paradigms can be built into many evaluation strategies to maximize community-wide learning:

• Participatory evaluation
• Developing your theory of change/program logic model

Participatory evaluation. Evaluation tends to work best and provide the most useful and usable information when it is carried out in a participatory, collaborative mode. In this setting, staff work alongside "professional evaluators" to jointly determine questions to be answered, the most effective ways to answer them, and the meaning and implication of the information collected.

Participatory evaluation processes attempt to:

1. Create a more egalitarian process, where the evaluator's perspective is given no more priority than other stakeholders;
2. Make the evaluation process and its results relevant and useful to stakeholders for future decision making; and
3. Help improve program implementation and outcomes by actively engaging all stakeholders in the evaluation process.

A participatory evaluation approach also provides opportunities to supplement traditional evaluation strategies (which are often designed and implemented by a third-party evaluator, with little involvement of program staff) with a variety of evaluation tools for improving programs-for example, innovative uses of existing management information systems or participant feedback surveys.

In addition, by actively seeking, valuing, and incorporating young peoples' perspectives and opinions into the evaluation design, participatory approaches can strengthen the impact of CYD programs and activities on the young people themselves. As youth continue to struggle with feelings of disenfranchisement and powerlessness in a society that does not do enough to value its youth, the positive impact of becoming active, contributing members of the evaluation and program improvement process cannot be overestimated.


Developing your theory of change/program logic model. Most CYD programs and organizations are not single, standardized, one-shot program designs-instead, they are complex, comprehensive, integrated, community-based, long-term models that also require complex evaluation designs. As such, it is often useful to begin the evaluation process by developing a theory of change or program logic model that depicts how your program works in graphic (and narrative) form. This logic model would then provide the foundation for all evaluation activities.

A program logic model links outcomes (both short- and long-term) with program activities/processes and the theoretical assumptions/principles of the program. This model provides a roadmap of your program, highlighting how it is expected to work, what activities need to come before others, and how desired outcomes are achieved.
There are multiple benefits to the development and use of a program logic model:

• Program design benefits. By utilizing a program logic model as part of the evaluation process, staff will be able to stay better focused on outcomes; connect interim outcomes to long-term outcomes; link activities and processes to desired outcomes; and keep underlying program assumptions at the forefront of their minds. In short, the process of creating a program logic model will clarify your thinking about the program, how it was originally intended to work, and what adaptations may need to be made once the program is operational.

• A powerful base from which to conduct ongoing evaluation. Because the program logic model spells out how the program produces desired outcomes, you can systematically decide which pieces of the program to study in determining whether your assumptions were correct. A program logic model helps focus the evaluation to measure each set of events to determine what works, what doesn't work, and for whom. In this way, the evaluation team will be able to discover where the model breaks down or where it is failing to perform as originally conceptualized.

• Effective for evaluating complex initiatives with intangible or long-term outcomes. A program logic model lays out the interim outcomes and the more measurable outcomes on the way to long-term and intangible outcomes (such as community participation). As a result, it provides an effective way to chart the progress of more complex initiatives and make improvements along the way based on new information.

• A valuable process. The process of developing a logic model is an iterative one, which requires stakeholders to work together to clarify the underlying rationale for the program and the conditions under which success is most likely to be achieved. Gaps in activities, expected outcomes, and theoretical assumptions can be identified, resulting in changes being made based on consensus building rather than on personalities, politics, or ideology. The clarity of thinking that occurs from this process becomes an important part of the overall success of the program.

"Mindful practice is simply the ongoing discovery of the thread of interconnectedness."

-Jon Kabat-Zinn

Question the Questions
In our previous article, we reminded readers that "unanswered questions are far less dangerous than unquestioned answers." The key to developing the most productive evaluation design possible is to create an open environment where different perspectives are valued and encouraged. CYD managers need to encourage staff to think about which questions are not being addressed-and why. Perhaps the design is driven by a particular philosophy or a set of values that may or may not be shared by all stakeholders. Or perhaps there is a strong sense of political consequences. Whatever the reason, it should be on the table-and one way to ensure this is to continuously questions the questions that are driving the evaluation design.

Taking Action
Dealing with the influence of different paradigms, politics, and values among stakeholders is a part of every evaluation. Acknowledging and exploring differences can lead to a more informed evaluation design and an open, safe environment. The following are among the most popular tips for program directors and evaluation staff.

• Challenge your own assumptions. Look for evidence that you are wrong, or as Steven Covey says, "Seek first to understand, then to be understood."
• Remember, there may be multiple "right" answers.
• Make the evaluation and its findings useful and accessible to program staff, partners, and participants. Early feedback and a collaborative relationship lead to a greater willingness to disclose important and sensitive information.

"Evaluation is just good thinking."

-Ricardo Millett, Director of Evaluation, WKKF

In the next part of the "CYD Management Zone," the last in the three-part series, look for a focus on identifying and measuring CYD outcomes.

Author

CONTACT BOX For more information on evaluation design, methods, sample logic models, and case studies, contact the author or order the W.K. Kellogg Foundation's Evaluation Handbook: Ricardo A. Millett, Director of Evaluation W.K. Kellogg Foundation One Michigan Avenue East Battle Creek, MI 49017-4058 USA Phone: 800-819-9997 Web: www.wkkf.org For evaluation capacity building, contact: Susan P. Curnan and Lisa A. LaCava Center for Youth and Communities Heller Graduate School, Brandeis University Email: curnan@brandeis.edu or lacava@brandeis.edu