| Spring 1999, v15-2 |
-byJoseph Kahne and Milbrey McLaughlin
How will we know if these programs are working?"
a banker asks us. "And how will we know why or why not?" His foundation
has a long history of working with youth programs, so he is not naive. He was not
asking us to determine whether a six-week summer program for tenth graders from inner-city
Chicago will change their rates of college attendance. But he did want us to assess
each of the eight programs being funded for their consequences for youth and to suggest
ways the programs might be more effective in the future (for a description of each
program see the sidebar on page 19).
This request for evaluation presented both challenge and opportunity. Traditionally,
in the evaluation of youth programs it has been difficult to:
- Understand program impact in the contexts
of youth's everyday lives.
- Separate out and quantify programs' shared
long-term individual-level goals from a broad array of short-term goals associated
with youth development.
- Locate causality in investigations without
approprate control groups.
- Attribute outcomes to particular interventions
or programs.
- Make appropriate adaptation to the varied
methods and priorities deployed by programs.
To help us face these challenges, we turned to theory-driven evaluation. We were
drawn to its capacity to balance the need for systematic study of programmatic impact
with the need for findings that can inform policymakers in varied contexts (Chen,
1990; Connell, Aber, & Walker, 1995; Weiss, 1995). Specifically, this approach
requires the evaluators to work with stakeholders to identify the theories and assumptions
embedded in a given policy initiative. Once identified, these orient data collection
and analysis.
Further, theory-driven studies represent an effort to move away from largely atheoretical,
impact-oriented evaluations which emphasize "knowing that" rather than
"knowing how or why." It is important for the reader to know that even
in best-case scenarios, with clear outcome measures and control groups, data regarding
a program's impact are often of limited value for policymakers because they are context
dependent. Knowing that a given curriculum, with a particular age group, done by
specific staff members, for a defined length of time, in a specific neighborhood,
had a certain impact does not let us confidently project the impact on participants
when a different organization tries a similar approach (McLaughlin, Irby, & Langman,
1994). It is not that outcomes do not matter--they do--but by unpacking the relationships
among theories, assumptions, outcomes, and specific contexts we can frame and interpret
outcomes in more useful ways.
In the case at hand, the Foundation held two assumptions about youth and community
service as well as two assumptions about action. First, they assumed that engagement
of youth as civic resources would foster their development and secondly, that community
service opportunities would help youth attain the knowledge, skills, and attitudes
necessary for civic leadership. In its theory of action, the Foundation assumed that
program staff were committed to systematically engaging youth in leadership development
and that collaboration among youth organizations, the city, and related partners
would enable larger and higher-quality programming.
We examined the four central assumptions and discussed the implications of our findings
for policy and practice. The assumptions again:
- Engaging youth as civic resources will foster
youth development.
- Community service opportunities will foster
the knowledge, skills, and attitudes necessary for civic leadership.
- Program staff were committed to systematically
engaging youth in leadership development.
- Collaboration among youth organizations,
the city, and related partners would enable larger and higher-quality programming.
Engaging youth as civic resources will foster youth development:
The assumption about the connection between community service and youth development
was well supported. Unlike standard classroom tasks, the service-learning curriculum
embodied respect for adolescents' capacity. The work expected of the youth mattered
in ways they could readily appreciate. This, in turn, provided youth with a sense
of competence and a recognition of the need for responsible behavior. When asked
for advice to give those planning youth programs, for example, one youth told us:
It is good to put a lot of responsibility on the childrenÖso they can do it themselves
and not just have an older person do everything and the children sit there and watch
and whenever they need the children, they jump in. I think it should be the other
way around. The children should do it and whenever the children need something the
person who is oldest jump in. Then they get whatever they needÖso the children can
do it and then that makes the children feel better because they can say, well I did
that, or I painted that, or I did that flower box.
When asked, "Do you think that you have always had that sense of responsibility?"
the student made explicit the power of the community service component:
No, 'cause I had jobs where it wasn't done for the community, and it's done for
just me and I do it and I would really not want to do it. But this is how you show
responsibility and for me I'm doing something for the community which everybody gets
to see and I just don't get to see it and I can say I did it. I can show people that
I'm doing it so that they can walk past and see me doing it. So that just builds
my self-esteem up.
Over one-half of the youth we interviewed offered similar assessments about the
value of community service for them. In addition, youth often related the relationships
and sense of belonging that developed with peers and supervisors when they described
the growth they experienced as part of the program.
Youth also articulated other ways in which participation in the youth programs contributed
positively to their development. For example, the youth reported learning to control
their emotions and about presentation of self, crucial components of youth development.
The projects where youth supervised young children seemed particularly instrumental
in this regard.
When I started working here, I was like all these kids are just bad. Get away
from me and all that. But now, like I said, I'm much nicer to the kids and I see
they may have to come here for the love of us. They like to lay on you and stuff
and I can't treat children any way because you could make a scar that could last
for a long time.
Youth saw their participation in these community service summer efforts as contributing
to their personal development in broad and fundamental ways. Concrete expression
of the value youth ascribed to community service may be found in their eagerness
to participate and willingness to stick with the effort. All programs quickly filled
their slots. Attrition rates in all but one of the programs were low, with over 90%
of the students staying with the program for the entire summer. Participants were
clearly attracted by the idea of performing a meaningful service. Indeed, 90% said
they would like to volunteer in a youth leadership corps during the school year.
Two-thirds of the participants "strongly agreed" that the summer programs
were enjoyable.
In addition to serving youth, the projects were often successful in delivering meaningful
community services. Though the designs of all programs had the potential to deliver
valuable services, only half of the projects were successful in doing so. Significantly,
the programs that failed to provide meaningful services were limited by inadequate
bureaucratic support or poor planning by the project coordinators, not by lack of
youth capacity.
In half of the programs youth also expanded their capacity as resources for their
community through significant skill development. Project Child Care and Project Build
were best-case examples. In each instance, students undertook rigorous study related
to their service projects and developed the appropriate specific skill. Other projects
included little or no skill development. There were, for example, few if any opportunities
for youth working on the mural to study artistic techniques, murals as an art form,
or murals as a means of social commentary.
The task of preparing youth to address community needs extends beyond skill development.
The Foundation's theory of change assumed that youth would integrate a vision of
and a commitment to community improvement into their identity. The programs appeared
largely effective in leading youth to affirm the value of community development,
but not in ex-panding or deepening their vision of ways they could be helpful. This
is well characterized by the youth who, when asked what he was learning, told us,
"It made me realize that the world is not just about me. You know, give, give,
give. Sometimes you just have to give back to the community by helping out picking
up trash, or like I said with the Peace March." At the same time that their
commitment seemed sincere, their vision about what civic responsibility would or
could entail was not well developed.
Community service opportunities will foster the knowledge, skills, and attitudes
necessary for civic leadership: Powerful support for this assumption comes
from Project Leadership--though, as we discuss below, not from other programs. Project
Leadership's curriculum advanced youth commitment to civic leadership in two ways.
One was a strategy of appointing "senior leaders." In the spring, four
senior leaders ages 15-21 took a four-day retreat with the project directors. Here
they learned about strategic planning and developed an extensive plan for the summer,
which included their neighborhood improvement goals, budgets, youth recruitment strategies,
and interview protocols. They decided, in their words, to "transform the vacant
lots in [the neighborhood] into beautiful and useful gardens and parks." They
identified 40 specific tasks associated with carrying out their plans and assigned
responsibilities. These tasks included writing letters to Aldermen and block club
leaders, purchasing supplies, constructing flower boxes, journal writing, and morning
devotions focusing on the story of Nehemiah.
Upon their return from their retreat, they interviewed and selected 15 "junior
leaders" from a pool of 30 applicants. Once the project began, the project director
was careful to give space for the senior leaders to lead, and worked constantly to
structure meaningful leadership opportunities.
The notion of collective response to community needs was explicitly connected to
each morning's Bible readings and prayer. Several students talked about how these
sessions defined a clear link between their work and their commitment to the community.
One said, for example,
Anything we do, it's always hooked up with the Bible. And that's good because
we [were reading about] NehemiahÖ And just like Nehemiah when he was building the
wall and people were attacking him he still got up there and did it. And that's how
it is with us. I think the devil was trying to attack us [when the project was moving
slowly] to see if we would stop then and leave it like that and I said we're going
to keep fighting for it.
Independent of connections to the Bible, the project supervisor consistently
pushed themes of community commitment.
I want to be a slum buster for this community five or ten years from now. [What
do you mean by that?] On my own, if I see a vacant lot damaged. Just bring my rake
and shovel and clear this lot and then go to the next. And once they see me doing
that, hands will come in together and we'll be united in this community.
While all programs were designed to foster civic leaders, only some actually
implemented leadership components. Implementation shortfalls occurred in two broad
areas, program curricula and collaboration with city agencies.
Program staff were committed to systematically engaging youth in leadership development:
Program proposals detailed ways youth would develop leadership skills such as
public speaking, strategic planning, facilitating, supervising, critical reflection,
conflict resolution, and visioning. Nonetheless, few of the programs included curriculum
that focused on developing these abilities. Instead, programs operated as though
youth would acquire these leadership skills indirectly through participation. The
program best suited on paper to advance this unfortunately suffered from a supervisor
with weak pedagogical skills. Youth working on the Youth Committee did not learn
about public speaking, budgets, or strategic planning. They were given large responsibilities
(to plan new programs, for example) without a structure to guide them. They accomplished
little and grew frustrated. This was the only project in which we saw youth disrespect
the coordinator or complain about their wages.
The fact that several of the programs where staff seemed most capable of implementing
complex curriculum did not emphasize leadership development resulted more from priorities
of coordinators than from any lack of capacity among the youth. In Project Build,
for example, the participants developed specialized construction skills as they built
a model home, but were not explicitly taught leadership skills as were Project Leadership
participants. Project Build staff retained the significant leadership tasks for themselves.
We also saw important distinctions in the different projects' conceptions of leadership
development. Project Leadership located youth leadership squarely in the community
and reinforced youth's sense of civic responsibility. In contrast, most of the programs
sought leadership development through promoting individual responsibility. Though
this emphasis on individual responsibility may be a sensible way to support youth
development (Feldman & Elliott, 1990), it is notably different from a focus on
civic leadership, which, as Barber (1984) and Boyte (1984) explain, requires a broader,
more collective focus on identifying and responding to community needs. In the process,
an important message is communicated about the need to change society, not only individuals.
Leadership, this evaluation reminds us, must be a social act.
Collaboration among youth organizations, the city, and related partners would
enable larger and higher-quality programming: Promised collaborations
that failed to materialize contributed to program shortfalls at both the project
and community levels. At the project level, the Foundation's theory of action assumed
that city funds and resources would enhance neighborhood efforts. In the four cases
where program activities were tied to city promises, significant problems arose because
the city did not come through as planned. These broken promises generated program
disruptions and delays that in each case diminished program activities and often
disappointed youth and reinforced their cynicism about "the system."
At the community level, expected collaboration among youth organizations failed to
occur. The Foundation assumed that youth organizations would be interested in working
together as a network on some aspects of professional development and perhaps some
collaborative endeavors with their youth. These expectations were not met. Two initiatives
undertaken by the Foundation failed.
There was also a general lack of interest in collaboration. When the idea of youth
organizations working together was raised during one forum, a program director said
that he would rather take youth to a museum or park where they could be exposed to
something enriching rather than have them meet with other youth from the neighborhood.
On a different note, but one that also underscores a hesitancy to collaborate, a
particularly effective youth worker told us,
If I could give us a grade on what we were doing with the kids, I'd probably say
a B+. Of course we want it to be an A. And so if we have, let's say, an extra night,
how are we going to spend that night? Is it going to be meeting with [other organizations]
or trying to sustain our marriages and our families or working with our kids? And
those two things are going to come before any thoughts of let's get together with
[other organizations]Ö I don't know how much I care to know about what they are doing
because I've become really obsessed with the things that are going on here.
Contrary to the Foundation's theory of change, even like-minded and similarly
motivated youth organizations were not moved to work together. Even when linked by
a shared youth-focused mission, concerns about individual program survival and program-specific
priorities eclipsed any investment in collaborative action.
Some Conclusions
The experiences of these projects dispute myths about teens, especially teens from
inner-city neighborhoods. Contrary to myth, we found young people:
Eager to join these programs.
When engaged in work seen as valued, apply themselves with energy and commitment.
Show a pride in themselves and their accomplishments not primarily linked to a paycheck,
but to the way they were recognized as helping others in the community.
Meaningful work enables youth development. Although the theory that motivated these
programs emphasized community service, it may be the provision of meaningful work,
not simply community service, that is most important to participating youth. The
study also highlights issues of concern for policy and practice. While it appears
that youth development arises relatively organically from engagement in community
service activities, the skills, knowledge, and attitudes needed for civic leadership
do not. It appears that curriculum aimed directly and systematically at leadership
development is necessary.
Program descriptions and culminating activities/ presentations or displays of projects
completed may often imply a greater degree of youth leadership and ownership
than actually took place. That youth built playgrounds or supervised younger children
did not guarantee that they learned much about construction or child supervision
and development.
Efforts to foster collaboration, while seemingly a rational response to Balkanized
community services and limited resources, add new demands to already overburdened
programs. They may lessen program coordinators' degree of control and undermine their
efforts to assure high-quality implementation.
Assigning youth genuine leadership roles requires both time from program staff and
willingness to risk failure (as parents often say, "It's easier to do it myself").
In the press of these summer programs, staff evidently decided not to expend the
time and effort required to provide meaningful leadership training and opportunity
for youth.
The most disheartening experiences associated with these summer programs resulted
from broken commitments from city agents and agencies. The consequences of these
failed commitments extended far beyond projects not completed or plans unfulfilled.
The bureaucratic or political snags that kept promised materials, funds, or expertise
from arriving as agreed sent yet another signal to these youth of their irrelevance
to mainstream society and impotence in "the system." These inner-city youth
and their community leaders understood these broken promises not as just another
bureaucratic snafu, but in terms of the low priority their African American, low-income
neighborhood appears to hold in city chambers and municipal offices.
The challenges associated with designing and implementing high-quality community
service initiatives are formidable. Fortunately, so are the potential payoffs.
References
1. Barber, B.R. (1984). Strong
Democracy: Partici- patory Politics for a New Age. Berkeley: University of California
Press.
2. Chen, H. (1990). Theory-Driven Evaluations. New-bury Park: Sage Publications.
3. Feldman, S. and Elliott, G.R. (1990). At the Thres-hold: The Developing Adolescent.
Cambridge: Harvard University Press.
4. Kahne, J. and Westheimer, J. (1996). In the service of what? The politics of service
learning. Phi Delta Kappan, 77(9), 592-599.
5. Langman, J., and McLaughlin, M.W. (1993). Collaborate or go it alone? In S.B.
Heath and M.W. McLaughlin, Identity and Inner-City Youth. New York: Teachers
College Press.
6. McLaughlin, M.W., Irby, M.A., and Langman, J. (1994). Urban Sanctuaries: Neighborhood
Organiza-tions in the Lives and Futures of Inner-City Youth. San Francisco: Jossey-Bass.
7. Weiss, C. (1995). Nothing as practical as good theory: Exploring theory-based
evaluation for comprehensive community initiatives for children and families. In
New Approaches to Evaluating Community Initiatives. Washington, DC: The Aspen
Institute.
8. Yates, M. (In press). Community service-learning in the social studies: Historical
roots, empirical evidence, critical issues. Theory and Research in Social Education.
Authors
Joseph Kahne is
an assistant professor in the College of Education at the University of Illinois
at Chicago. He is currently studying and actively involved with educational reform,
service-learning, and youth development efforts in schools and youth organizations
in Chicago and around the country. His book, Reframing Educational Policy: Democracy,
Community, and the Individual, was recently published by Teachers College Press.
Milbrey W. McLaughlin is
author of several educational books, including Urban Sanctuaries: Neighborhood Organizations
in the Lives and Futures of Inner-City Youth. Milbrey currently teaches at Stanford
University.
|
|
