| Spring 1999, v15-2 |
-by Barry M. Kibel, Ph.D.
Outcome-Based Accountability
There has been a dramatic shift in the demands for accountability in recent years.
Previously, many agencies and programs could justify their worth in the process
zone, indicating number of services provided, number of people reached, and levels
of consumer satisfaction with services received. Today, most funders and other stakeholders
insist that outcome-contributors go further and document, quantify, and assess the
worth of the sustained behaviors and consequences that have resulted from these services.
In short, they want outcome-contributors to prove their worth in the outcome
zone.
This shift in demands is laudable. Agencies and programs ought to be held accountable
for outcomes and not simply for processes. However, the way that outcome-based accountability
is established requires some careful thought. A good place to start this thought
process is to define what a comprehensive, outcome-based accountability system ought
to accomplish. Three things stand out:
- Clear documentation of what the program
does to promote outcomes, as well as careful and accurate presentation and measurement
of these outcomes;
- Quantification of the amount of credit (attribution)
that the program can rightly claim for producing or, more likely, for contributing
to these various outcomes; and
- Timely and provocative feedback for
guiding the program toward enhancements and increased outcome production.
From an outcome-accountability perspective,
what counts are not the outcomes produced per se, but the credit for these outcomes
that can rightly be attributed to the outcome-contributor and assurances that the
contributor is intent on getting ever better at outcome production. To determine
this, we need to know precisely what the agency or program is doing to contribute,
so that fair and appropriate credit can be assigned and so that clues can be garnered
regarding adjustments or additions needed to increase outcome production. The conceptual
framework and associated methods and measures upon which to base an accountability
system must necessarily depend on the specific mix of strategies being used
by the outcome-contributor. For maximum utility, that framework must be as multi-faceted
as is the work of the agency or program to which it is being applied.
In short, a mix of tools keyed to the strategies being employed is needed for use
in (a) documenting and assigning credit to outcome-contributors fairly and consistently,
and (b) uncovering ways to enhance their work. In surveying the universe of possible
strategies associated with outcome production, I determined that six strategy types
appear to capture the entire set of possibilities. Hence, the ideal, all-purpose
toolbox, as I have conceived and designed it, consists of six generic tools that
are customized for each new application. These will be introduced in brief below.
Enter the Outcome Engineer
Engineering, as a discipline, is concerned with putting scientific knowledge to practical
uses. Outcome engineering focuses on knowledge related to outcome generation
and what it takes to increase the quantity and quality of outcomes in cost-effective
ways. The outcome engineer is the professional who applies the toolbox to document
links between services provided and resulting outcomes and then determine outcome
credit. The outcome engineer then uses these data to assist an agency or program
to improve its outcome productivity and meet its related, outcome-based accountability
challenges. The engineer may be a hired consultant or a staff person assigned these
responsibilities on a full or part time basis.
The role played by the outcome engineer is three-fold:
Role 1. Helping the outcome-contributor to chart the strategies it
currently uses or projects it will use to reach toward each of its designated outcomes;
Role 2. Customizing and then applying the appropriate tool from the
toolbox to gauge the success of each current strategy in contributing toward its
associated outcome; and
Role 3. Facilitating design sessions to devise new strategies for increased
outcome productivity (including initial designs for start-up ventures).
Associated with each of these roles is a recommended technique. These I have named
Outcome Pathway Analysis, Outcome Scorekeeping, and High-IMpact Planning. These techniques
are employed by the outcome engineer to assist agencies and programs to become more
outcome conscious and outcome productive.
Each of these techniques is next briefly described, in turn.
Outcome Pathway Analysis
An outcome is a sustained change in the behavior or status of some individual,
group, organization, or community that can be attributed in part to the efforts and
influence of an outcome-contributor. The contributor makes use of one or multiple
strategies to bridge between its efforts and each of its designated outcomes. Outcome
Pathway Analysis is used to locate these different strategies within a three-by-two
grid. Of critical importance from an outcome-accountability perspective, each of
the six cells of the grid represents a distinctly different type of relationship
that exists between the outcome-contributor using this strategy and those whom it
is attempting to influence.
|
|
| |
Plotting the strategy or strategy
mix of the outcome-contributor using this six-cell grid serves a few, important purposes:
First, it makes apparent the approach being used by the contributor
to tackle the particular outcome.
Second, it indicates the relative influence that the contributor is
likely to have on the individuals being targeted.
Third, it suggests the type of tools that ought to be used to track
the work of the outcome-contributor and gauge its impact on individuals being targeted.
Fourth, it helps pinpoint strategic gaps that potentially can be closed.
The first three strategy types in the grid (I-1, I-2, and I-3) involve a direct relationship
between the outcome-contributor and the individuals for whom outcomes are anticipated.
The remaining three strategy types (E-1, E-2, and E-3) are directed at the environments
of these individuals with expectations that the environmental changes will trigger
behavioral changes. Within each set of three, there is one strategy that is founded
on cause-and-effect principles (I-1 or E-1), a second based on the application of
persuasive power (I-2 or E-2), and a third that emphasizes the building of support
to underpin and bolster subsequent efforts (I-3 or E-3).
I-1 strategies focus on repairs, reducing or eliminating deficits, and solving
or preventing relatively straightforward problems. The outcome-contributor applies
some treatment that has previously been shown to be efficacious. The individuals
receiving the treatment tend to benefit almost immediately and these benefits tend
to be sustained. An example of an I-1 strategy is the provision of tetanus shots
to all members of a family. A second example is the provision of emergency assistance
to community members whose homes have been destroyed by a flood or other natural
disaster. For these I-1 strategies, the outputs produced by the contributor
are identical, or closely associated, with the outcomes desired. To illustrate, in
the first example, the output is the tetanus shot received; the outcome is ten years
of protection against this potentially fatal disease.
I-2 strategies are more persuasive than causal. The outcome-contributor prescribes,
directs, recommends, or instructs individuals with regard to options and behaviors
that the contributor deems worthy of their consideration. The individuals receiving
this output must then make personal choices relating to these options and
prescribed behaviors. For sustained effects, these choices often must be made multiple
times and may lead to other behavior changes and follow-through actions. Examples
of I-2 strategies are referrals by primary care providers to specialists, the use
of motivational speakers to promote healthy behaviors, and workshops providing training
in some new computer software. Note how the relationships between the outcome-contributor
and individuals being impacted differ for I-1 versus I-2 strategies. In the first
case, control for outcomes rests largely with the outcome-contributor. In the second
case, control for outcomes rests largely with the individuals. This has important
implications for appraising contributions toward outcomes, as will be discussed below.
I-3 strategies are more supportive than either causal or persuasive.
The outcome-contributor removes obstacles to health, safety, or growth; points individuals
toward positive options; then assists these individuals in gaining access to and
fully exploiting these options. Nestled within these strategies are I-1 and I-2 tactics;
however, the intent is broader and involvement by the outcome-contributor is more
frequent and sustained than under the earlier two strategies. Often, secondary
outcome-contributors are enlisted to support the work of the primary contributor.
Examples of I-3 strategies are the work of a mentor with a child or a case manager
with a family. As with I-2 strategies, ultimate responsibility for outcomes rests
with the individuals and not with the primary outcome-contributor.
E-1 strategies focus on the environments within which targeted individuals
carry out their work or life routines. The aim of the outcome-contributor is to manipulate
these environments through physical or policy changes so as to cause the individuals
to modify their routines in ways projected to produce desired consequences. E-1 strategies
may be restrictive or life-enriching. And again, as with I-1 strategies, the outcome-contributor
is largely in control of consequences. An example of an E-1 strategy is the placement
of speed bumps in residential areas to slow down drivers. A second example is limiting
driving privileges for youth under 18 to those who are attending or have finished
high school, or the equivalent. A third example is a government conservation project
to reclaim and protect the wilderness for future generations.
E-2 strategies implant the environments of targeted individuals with potentially
influential messages. As with I-2 strategies, the aim is to persuade and ultimate
success depends on subsequent choices being made by these individuals. Whereas E-1
strategies make use of penalties and rewards to promote compliance, E-2 strategies
depend for success on the readiness and willingness of the targeted populations to
take the messages being delivered to heart and act on them. An example is a stateís
governor and his wife sending congratulation cards to every parent of a newborn child
in the state along with an immunization reminder. A second example is the use of
billboard campaigns to promote non-smoking behaviors.
E-3 strategies aim to produce therapeutic communities or learning/action networks
within which its members are supported and support one another. Examples of E-3 strategies
include healthy workplace initiatives, researcher networks, churches and synagogues,
youth clubs, coalitions, private-public partnerships, community health collaboratives,
and sports groups. The role played by the outcome-contributor and its partners is
frequently that of a facilitator, although the contributor may well be one of the
community members.
Outcome Pathway Analysis is performed for each of the outcomes of interest to the
outcome-contributor. The outcome engineer works with the outcome-contributor and
perhaps others to define clearly these outcomes. For example, an outcome for a community
health initiative might read:
Outcome 3. Community residents who, when interviewed, will report that they
have "a voice in the community."
Note that the outcome is not quantified (what percent increase?) nor linked to a
specific time line (by what date?). At this stage of analysis, thinking and discussions
need to remain as open as possible and not constrained by target values.
For each outcome, the six strategy types are considered in turn. The questions asked
are "Is this strategy currently being employed and, if so, how?" and "Whether
being used or not, can this strategy be exploited to reach the outcome and, if so,
how might it be employed?" The results of this activity is a matrix that displays
each of the outcomes against the strategies currently being used, or projected for
possible use, to help produce that outcome. In the example above, the community health
initiative was currently relying exclusively on persuasive strategies (i.e., I-2
and E-2) but was able to project a far more ambitious effort to reach this outcome
based on the use of five strategy types.
|
|
| |
These are generic tools that need to be customized
prior to use. The customization process is interactive and involves participation
by the contributor, the outcome engineer, current evaluators, and others designated
by the contributor.
Output Maps (Used with I-1 Strategies): For this type of strategy, the outcome-contributor
ought to be applying a treatment that has been demonstrated previously to produce
a desired output for all or most persons treated. Since the connection between the
intervention and the outcome has been previously established, accurate counting of
the number of persons receiving the treatment (the outputs) provides a reasonable
surrogate for the outcomes produced. In addition, the contributor must remain on
the lookout for unexpected and possibly undesired side effects. The generic tool
for the I-1 strategy is a statistical summary form that is used to collect output
evidence. The form might be customized to capture relevant client information, service-linked
data (e.g., dosage levels), client satisfaction ratings, evidence of success, and
reports of side effects. Credit for the outcome-contributor would be linked directly
to the number of successful cases recorded that did not involve adverse side effects.
Mapped Sentences (Used with I-2 Strategies): Persuasive interventions directed
at individuals provide unique outcome measurement challenges. In some cases, a single
downstream outcome is all that is expected (e.g., the individual followed through
on the suggestion made by a program staffer). In other cases, sustained behavior
adjustments are anticipated (e.g., the individual attended support group meetings
or parenting classes on a regular basis). In still other cases, the program hopes
for stages of change (e.g., the individual studied the material gleaned from the
training during the first month post-training, began applying lessons learned during
the next few months, and had significantly modified practices by the end of the first
year post-training). The generic tool used for I-2 strategies is the mapping sentence,
a fill-in-the-blank form that is an expanded version of the one used for I-1 strategies.
That form might be customized, for example, by incorporating data fields to record
services provided at multiple time points, track sets of success markers that are
reached at different stages after the initial intervention, and capture the impacts
of follow-through activities conducted by the individual based on personal initiatives.
Different techniques can be used to define the success markers. For example, the
critical incident technique developed and used extensively at the American Institutes
for Research is ideally suited for this purpose. Once defined, individual progress
can be rated and these ratings used to gauge overall credit due to the outcome-contributor.
Spaces would also likely be provided on the customized form for rating the potency
of the services provided. These ratings are needed input for the credit quantification
process. In brief, a low rating would imply that the service provided was modest
and short-lived and very little credit can rightly be claimed for successes that
occur downstream. A high rating, in contrast, would imply that the service provided
was significant and likely made a major contribution to any short-term successes
reached and a major-to-modest contribution to longer-range changes. Again, as with
the success markers, different techniques can be used to devise a rating scheme.
The one I prefer employs a hierarchic classification system, called the Results Ladder,
to rate the levels of service provided.
Mapped Stories (Used with I-3 Strategies): When asked for outcome data, outcome-contributors
employing I-3 strategies are likely to respond, "Let me tell you a story!"
This is not an evasive maneuver. The story is often the most accurate form in which
to relate the work of a complex program. Numbers may not be accurate in such cases.
They are one-dimensional, whereas the program is three or four dimensional. However,
the difficulty with the story, as the unit of evidence, is that it is anecdotal.
A technique was needed to convert narrative, story data into "hard" data.
That technique is called Results Mapping. A set of 28 rules and conventions are used
to insure that independent mappers will document, code, and score story data in near
exact manner. In Results Mapping, outcome-contributors earn credit for services provided,
for networking with other agencies (i.e., collaborative models), for the services
these agencies provide to the program's clients, and also for actions by the clients
themselves for self-help or to benefit others with similar problems. These different
types of credit are quantified and can be aggregated or disaggregated so that funders
and programs can focus on those aspects of program performance and collaboration
of concern to them.
The advantage of story mapping is that it shows clearly what it really takes to bring
clients to levels of readiness and capacity to engage in acts of self-determination
associated with short and longer-range outcomes. Outcome counts alone can create
the illusion that the link from services to outcomes is relatively simple. The mapped
stories destroy the illusion by depicting the often complex set of steps needed to
attain each outcome. These stories, in combination with the points and outcome counts,
can be translated into performance measures that can feed into outcome-based funding
processes. Further, the richness of information and insights gained from full exposition
of the best work of a program or agency is often reward in itself. Rare is the outcome-contributor
that has taken time to fully study its own best work and then determine from these
data what it does well and might do better. Results Mapping provides a way to do
just that.
Impact Maps (Used with E-1 Strategies): In contrast with I-1 strategies that
involve a one-to-one relationship between contributor and client, the E-1 strategies
are based on a one-to-many relationship. The credit due to the outcome-contributor
is directly linked to the demonstrated power of the physical or policy intervention
it introduces to alter behaviors en masse and generate desired impacts. Since
the changes evoked are widespread and often difficult and costly to track, the outcome
engineer must gauge the impact of the intervention through an indirect method. The
generic tool used for E-1 strategies is the impact map. Such maps may range from
complex, computer-based simulations to simple diagrams drawn on a single sheet of
paper. The availability of user-friendly software packages has brought the capacity
for model construction within the reaches of many, allowing for more complex impact
mapping if budget and time permit.
When an E-1 strategy is considered, estimates are made of the types and numbers of
persons who likely will be impacted and the nature of the impact. Once the intervention
has been implemented, data are collected to determine if the intended results and
impacts have happened or are in process. I suggest that four types of data be collected:
narrative accounts of the intervention and its effects, statistical data (based on
interviews of sample populations) that array numbers and types of impact, graphical
data that depict the dimensions of the impacted area, and selected stories to capture
the human side of the impact. Again, if time and budget permit, these data should
be collected immediately following the introduction of the intervention (to gauge
"halo effects"), three to six months later (to gauge the "stabilized
effects"), and a year or more later (to gauge the "normalized effects").
As with I-1 strategies, attention should also be paid to gathering information regarding
anticipated and unanticipated side effects.
Influence Maps (Used with E-2 Strategies): Like E-1 strategies, E-2 strategies
are one-to-many. Through a single or small number of activities, they are intended
to impact many people and, as a consequence, health or social conditions overall.
And like I-2 strategies, people are approached through persuasive rather than causal
means. The pragmatic outcome-contributor recognizes from the outset that it cannot
expect to convert everyone in the society who is not currently conforming with its
viewpoint. Instead, it must target its interventions toward those most near the edge
of conversion and who will also relate most easily to the messenger. By doing this,
the contributor aims to activate a chain reaction of positive responses, starting
where prospects for success are highest and then relying on the early converts to
spread the message to others who the contributor is unlikely to reach on its own.
The prototype tool for E-2 strategies is the barrier busting study. Since successful
E-2 strategies are targeted to those at or near the edge of conversion, the outcome-contributor
should first identify these populations, then assess the relative strength of the
barriers that currently stand in the way of their conversion. The contributor next
should select the barrier or barriers that are strongest and launch a campaign aimed
at weakening their strength. The barriers to consider would include unawareness,
indifference, misinformation, peer pressure, counter tactics, inconvenience, monetary
cost, difficulty to do, impact on self, and impact on others. To illustrate, in the
early years of the HIV/AIDS epidemic, indifference ("This is a problem for the
gay community, not for us!") and misinformation ("You can contract AIDS
through the air!") were barriers that needed to be challenged. Many in the population
who were unknowingly engaging in risky behaviors were also willing converts once
they recognized their vulnerability and had the correct information upon which to
base decisions.
The outcome engineer can assist the outcome-contributor in conducting the analysis
leading to the selection of target populations, barriers to tackle, and strategies
for weakening these barriers. And post-implementation, the outcome engineer can help
with the surveys, focus-group sessions, or environmental impact investigations used
to determine if the strategy is working and, if so, to what extent. One approach
useful for determining the impact of barrier-busting interventions is to seek evidence
of changes in intermediate behaviors within the populations of interest linked to
the target outcome. For example, to determine the effects of the AIDS awareness campaign
referenced above, the outcome engineer or local evaluator might have surveyed local
drugstores for evidence of increased condom sales.
Adoption Maps (Used with E-3 Strategies): The outcome-contributor uses the
E-3 strategy to build and sustain a support network of members of like mind or challenged
in similar ways. That network might be formalized as an institution (e.g., a church
or sports club). Central to such networks are a set of guiding principles of behavior
that members are expected to follow and model. These principles may be explicit or
implicit. In the latter case, a key role for the outcome engineer is to help make
these principles explicit through interviews and group sessions with key members
of the network. Key principles that are likely to be found in any such network are
respect for fellow members, support for fellow members, and personal commitment to
the mission and ideals of the network (as first articulated by its founders and perhaps
later revised).
To gauge the success of this strategy, I suggest use of an "ideal emulation"
scorecard. Periodically (say, monthly or quarterly), the entire membership of the
network or a sample of members for a large network are asked to complete the scorecard.
For each of the ideals listed, the respondent indicates his or her behaviors or actions
during the reporting period that best reflected that ideal. The respondent then rates
this performance on a 0-5 scale (with 5 representing some unusual and highly exemplary
performance). Initially, hypothetical examples of the type of performance corresponding
to different levels of the scale are provided as illustrations. Over time, actual
examples from network members are substituted. Results for each reporting period
are tallied to present an overall profile for the network. Exemplary performances
by individual members may be posted, included in printed materials, and/or rewarded.
Tracking these scores over time provides a basis for reflection and continual improvement
of network activities.
High-IMpact Planning
High-IMpact Planning is a concentrated planning process that is used by the outcome-contributor
interested in expanding its range of strategies for addressing current outcome targets
or for tackling some new outcomes. The process works exceptionally well for small
groups (say, 15-20 or under), but has also proven successful with larger groups.
The group is guided by a facilitator through a series of structured exercises that
result, in turn, in a long-view challenge statement, a set of shorter-term outcome
targets, a cluster of action items that can be launched immediately to move toward
each of these outcomes, and a fully articulated plan of action with quantitative,
qualitative, and efficiency criteria for gauging success during implementation. Within
6-10 hours, depending on the complexity of the issues or opportunities being addressed,
planning talk ends and action can begin in earnest.
High-IMpact Planning is based on an iterative design process that promotes learning
by doing and getting things done. Five stages are involved. The first three stages
are completed by the group in a six to ten hour session (or series of sessions).
A highlight of the process is stage 2, where a range of creativity exercises are
used to promote new thinking. The implementation period (stage 4) typically lasts
three to four months and may include monthly meetings of the whole group or its representatives
to take stock of progress and make mid-course adjustments. The final stage of the
cycle (assessment and scaling up) is again a whole-group activity and takes around
two to four hours to complete. Thus, in total, the non-action stages require between
eight to fourteen hours per cycle and lead to and support three to four months of
productive action. For small, self-contained units, scaling up (the transition
to the next design cycle) implies intensifying the design process by introducing
new elements and activities into the action plans. For larger groups, particularly
those intent on community-wide or systemic change, scaling up implies intensification
but also broadening of group membership each cycle and, of equal importance, helping
other groups launch parallel design processes.
For certain planning challenges, such as community-wide planning initiatives, I have
found that stage 2 is best accomplished through a "request for proposals"
process. Here, the planning team's role is to devise and implement the process, while
leaving the idea generation to those who are attracted to the process and submit
proposals. The action plan is simply the sum of the proposals that are funded. When
this approach is taken, the planning team needs to carefully construct the request
process to insure that proposals address desired action areas. The team should also
establish conditions for funding (e.g., willingness of successful grantees to participate
in regular learning community sessions) to insure that the total set of proposals
builds toward desired outcomes and impacts.
Summation
Outcome engineering is a long-needed resource for outcome-contributors, particularly
for those engaged in complex, multi-strategy work with individuals and their environments.
Its use does not understate this complexity, but rather provides a framework and
a related set of tools for unmasking this complexity so that it makes sense to stakeholders.
Moreover, its use supports and enriches continual improvement practices that are
associated with increased outcome productivity.
Please address comments, questions,
or inquiries to: Barry M. Kibel, Results Mapping Laboratory, Pacific Institute for
Research and Evaluation, 121 West Rosemary St., Chapel Hill, N.C. 27516. Phone: (919)
967-8998x14, E-mail: kibel@pire.org.
Some of these tools are similar to those currently being used to conduct research
or evaluate social and health programs. These existing tools can be substituted for
the ones suggested here, as appropriate, or blended with those suggested to draw
on their respective strengths.
|
|
