CYD Management Zone:
Evaluation as a Management and Learning Tool
 

by Susan P. Curnan, Lisa A. LaCava

Part 1 of a 3-part series

Someone once said, "charisma without a program is always ineffectual." It's easy to conjure up images of persuasive talkers who fall short in the "doing"-the dreamer without a plan, the politician without the capacity to govern and negotiate.

In contrast, the leaders of the CYD movement have charisma to spare as well as a rapidly growing set of programs. Without evaluation capacity, however, the movement may be destined to plateau long before it realizes its full potential. Typically, "evaluation" consists of taking a snapshot of "outcomes" at the end of the program to prove to a funder that "it" worked or failed. Too little, too late, these limited evaluation techniques often shortchange the CYD movement, and, in so doing, make no difference at all. We have to do better. Evaluation can, and should, be used as an ongoing management and learning tool to improve our effectiveness.

To gain greater recognition for the CYD movement, we must improve our organizational learning through widespread use of credible and innovative (or as the researchers say, rigorous and relevant) evaluation and research. We must increase capacity among CYD practitioners, funders, policymakers-and, yes, even evaluators stuck on single-method, problem-centered approaches.

Recently, a growing movement has emerged that broadens the way we think about evaluation. Rather than existing as a "standalone event," which occurs at the end of the program to judge merit or worth, evaluation is an ongoing process: one that provides leaders with useful and usable information for decision making. Reframing evaluation in this way allows mangers to collect and analyze important data-from assessing community needs prior to designing programs and services, to making connections between activities and intended outcomes, to changing program design when needed, to providing evidence to funders that yours is an effort worth supporting.

As it turns out, there is a groundswell of interest in this area, particularly around the link between evaluation and management. Though this link is vitally important for CYD professionals, capacity building and professional development opportunities are still few and far between. In the last issue of "The CYD Management Zone," we asked you to think about program evaluation as a management and learning tool by considering two specific questions:

  • How can we use evaluation information and innovations in evaluation methodology to improve CYD programs and shape policy?

  • What do managers and leaders in the CYD movement need to know and be able to do regarding evaluation?

In this issue, the first of a three-part series, we address a group of questions from the field that focus on preparing for evaluation. The next two issues will focus on determining your evaluation design and measuring the meaningful outcomes.

Preparing for Evaluation
-How do I select an evaluator or evaluation team that is right for my organization and that will lead to useful and usable information?
The sidebar that follows addresses the selection of evaluators in the form of a case study from a national organization serving women in the welfare and prison systems.

Case Study: An organization serving low-income women in three communities hired a consultant to evaluate the early stages of an innovative program for young women living in poverty. Not having a great deal of expertise with evaluation design, the director and key staff hired evaluators based primarily on their expertise in traditional evaluation methods.

For several reasons, the relationship failed and the evaluation was not useful to staff or other key stakeholders. For example:

  • Staff did not believe the evaluators understood how to work effectively with the women in the program.

  • Staff felt the evaluators had markedly different perspectives or values about how to define success or a positive outcome, and what was considered important enough to document. For example, the evaluators defined a positive outcome as securing employment where income taxes were withheld, while seeming to ignore outcomes the staff considered critical, such as increased sense of self, healthy relationships, coping and problem-solving skills, etc.

  • Staff did not consider themselves knowledgeable enough about the technical aspects of evaluation to effectively make the case for why the evaluation, as planned by the "expert" evaluators, would not yield useful information.

  • Staff neither understand their potential role in determining the purpose and goals of the evaluation, or the evaluators' roles. Both staff and evaluators assumed it was the evaluator's job to design the evaluation and define roles; thus, roles and expectations were never explicitly discussed.

  • Because the evaluators' expertise was in traditional survey research and impact studies, they assumed their job was to judge the merit or worth of the program through traditional paper-pencil surveys and follow-up telephone interviews. Without an evaluation team where staff and other key stakeholders were empowered to contribute and shape the evaluation, critical information was never uncovered or utilized. For example, many of the women produced false statements on the survey (e.g., noting they were not on welfare when, in fact, they were). Staff had been concerned that this would be a likely scenario on a paper-pencil survey, given the women's distrust of "forms" they did not understand, from people they did not know. Staff also knew that many of the women in the program did not have telephones in their homes. Thus, follow-up phone interviews-an important second phase of data collection-were not going to be an effective means to collect information. However, because the staff and evaluators were not working together on an evaluation team, none of this information was utilized or acted on during the evaluation.

  • Finally, staff, who were very dedicated to the women being served and sensitive to the multiple barriers and difficulties they had faced, began to resent the evaluators for their seeming lack of sensitivity to these women, and the fact that the evaluation might feel intrusive and impersonal. Consequently, the relationship quickly deteriorated and most staff members and participants were disappointed with the final report. One staff member put it this way: "I didn't even see our program in this report. Or the women. Or really what the women got out of the program. None of that was in there."

In the second year of the program, given a chance to hire a new evaluator, staff members reevaluated their hiring criteria. The most important characteristic to look for in an evaluator, they decided, was someone who had values and a philosophy that matched that of the organization. They also wanted to hire an evaluator who, like them, advocated for the cause of women in poverty. Finally, they wanted to ensure that this evaluator would treat the women served with the same high level of respect and care that each program staff person and volunteer did.


Ultimately, the moral of this story may be that "unanswered questions are far less dangerous than unquestioned answers." Despite recent changes in the way evaluation is perceived, funders, managers, practitioners, and researchers still rely on the traditional model of the external, third party evaluator, who conducts standalone evaluation with little participation or involvement of organization staff or program participants. And too often, this unquestioned model or "answer" leads to the same outcomes that this organization faced-an unrecognizable report, of little use, which does not impact decision making within the organization.



Key Questions for Evaluators
The following are several questions for CYD leaders to address each and every opportunity they engage in an evaluation, regardless of its purpose or goal.

-What type of evaluator do we need?


In addressing this question there are generally three options. (1) You can hire an external evaluator contracted from an outside organization. (2) You can assign the responsibility for evaluation to a person already on staff, or hire an internal evaluator to join your organization. (3) You can use a combination approach where an internal staff person takes primary responsibility for conducting the evaluation with the help of an external consultant who might assist with the technical aspects (such as survey analysis or gathering specialized information). The decision you make will depend on answers to questions such as these:

  • How important is an outside perspective for this evaluation?
  • Does our evaluation require specialized evaluation skills?
  • Is it politically important to engage a third party evaluator?
  • Is it critical that our evaluator is familiar with the program and our organization?
  • How important are opportunities for ongoing, day-to-day data gathering and informal feedback between the evaluator and key stakeholders?

In the end, though, the formation of an evaluation team consisting of key staff, relevant stakeholders, and evaluators is more important than who you hire. Through teamwork and ongoing dialogue, an evaluation team approach will ensure that a well-planned and appropriate evaluation emerges. A team approach will also ensure that many perspectives impact knowledge of the program, and that appropriate actions are taken based on this knowledge. Ultimately, the evaluation results will be more accurate, fully developed and useful, and will more likely to lead to action.

-What criteria should we use to select an evaluator?

-What does the evaluator need to know and be able to do?


Here again, organizations typically assume that the selection of an evaluator is based on criteria associated with specific evaluation skills or knowledge sets. However, as noted in our case study, other criteria may be important to consider, which should be explicitly defined at the start of the search. For example, you may require someone familiar with the specific community you are working in, with knowledge of the specific target population you are engaging, or with an understanding of your program strategy and approach.

The emerging view of evaluation-that evaluation should be a participatory process, where both the process and results are meaningful and useful to those ultimately responsible for improving and assessing the program-has resulted in a growing and changing set of skills required of evaluators. Methodological expertise is just one of many skills of an effective evaluator. Others include the ability to listen, negotiate, bring together multiple perspectives, analyze contextual factors, manage people, and assist in developing a design with an evaluation team that will lead to the most useful and important information. The fit with individual styles and your organizational culture is often an important factor to consider, as is the candidate's philosophical assumptions and values about evaluation. But the most important overall characteristics include the ability to remain flexible, strong problem-solving skills, and effective interpersonal skills.

-What role should the evaluator play?


Whether you decide on an external or internal evaluator, or some combination of both, it will be important to think through the evaluator's role. As the goals and practices in the field of evaluation have diversified, so too have evaluators' roles and relationships with the organizations and programs they evaluate. Yet, many organizations, like the one highlighted in the case study, do not realize that they can and should take an active role in determining the evaluator's role. Depending on the primary purpose of the evaluation and with whom the evaluator is working most closely, an evaluator might be considered a consultant for program improvement, a team member with evaluation expertise, a collaborator, an evaluation facilitator, an advocate for a cause, or a synthesizer of information.

The evaluator's role will influence the criteria you use to select the evaluator, and so the question of what selection criteria to use and what role the evaluator should play should be considered together. For example, if the role of the evaluator is to determine outcomes and the difference the program made, you might look for an evaluator with traditional methodological expertise and experience-particularly in the area of outcome evaluation. If the evaluator's role is to facilitate program improvements and changes, you might look for someone who has excellent communication skills, a good understanding of the program and people served, organizational development knowledge and skills, logical and inclusive thinking skills, and facilitative leadership abilities.

One role that is not often considered is in the realm of capacity building, so that evaluation activities and processes can continue once the final evaluation report is submitted. A part of every evaluator's job (internal or external) is to assist in building the skills, knowledge, and abilities of other staff and stakeholders. In our view, it is better to have an evaluator who has spent time working with staff-to integrate evaluation activities into day-to-day project management and delivery-than to have a perfectly constructed evaluation, with strong recommendations that no one uses, with no one able to continue the work.


Authors

Susan P. Curnan is director of the research and policy center on youth and communities and cochair of the MBA and MM program concentration in Children, Youth and Families at the Heller Graduate School, Brandeis University. She is also a director and founder of the new Institute on Sustainable Development. She can be reached at curnan@brandeis.edu.

Lisa A. LaCava is a management coach and senior program associate at the Heller Graduate School, Brandeis University. She writes and consults on evaluation design and strategy with foundations, community-based organizations, and CYD policymakers and practitioners in the USA. She can be reached at lacava@brandeis.edu.
 

CYD Journal © 2001