3.
Theory: Explaining How Your Project Works
- In
the process of planning the evaluation of an after-school
technology project for middle school students, an
evaluator convened a focus group of project managers
and board members. It quickly emerged that the various
stakeholders had completely different visions of what
good the project was supposed to do for participants
and their families.
-
- One
saw it as computer skills training to prepare young
people for jobs in local business, while another believed
that it would provide underprivileged families with
access to information capital to help lift them out
of poverty. After the meeting, the project director
explained in an aside with the evaluator that she
believed the biggest benefit was that girls were provided
a place to go after school, heading off behaviors
that might result in pregnancy.
The
implications for evaluation planning in this actual
case should be clear: It could be entirely possible
for one decision maker to see the project as a complete
success, while another determines it to be an utter
failure - based on the same evaluation findings -
depending on what outcomes they expected the project
was supposed to achieve.
- Projects
implementing technology in school settings may be
particularly challenging to describe since the project
may play so many different roles in the work of teachers,
administrators, or students, that may be thought of
as acting on teaching and learning processes in a
huge variety of ways.
Logic
Maps
SERVE
espouses the use of logic maps (sometimes referred
to as "logic models," "causal maps,"
or "theory-of-action models") as powerful
tools to clarify project activities and desired outcomes,
and to build consensus among project managers, participants,
and stakeholders.
A
relatively simple introduction to logic mapping is provided in Logic
Maps for Planning and Evaluation (PDF), while a more comprehensive view can be found in Building
Capacity for a Technology Project Evaluation: The
Role of Logic Maps (Power Point) - the latter
being a presentation at the U.S. Department of Education's
Evaluation Institute (February, 2005) by SERVE staff.
Another
excellent resource is Learning
From Logic Models: An Example of Family/School Partnerships
Programs from the Harvard Family Research
Project, a brief that provides an excellent step-by-step
overview of how to develop a logic map.
- Developing
a Logic Map
It
might first be useful to examine some examples:
While
a logic map may vary depending on the key terms that
planners agree to apply, the example logic map format
illustrated here has the following features, defined
as key aspects of the CAPE formative evaluation framework.
- Goals (sometimes called ultimate or long term "Outcomes")
are shown at the far right of the map
- Objectives (or "Intermediate Outcomes") are one level
to the left
- Arrows
connect Objectives to Goals, illustrating how meeting
the Objectives is expected to result in achievement
of the Goals
- Strategies applied by the project are shown at the next level
to the left, connected again with arrows to the Objectives
that each is expected to further
- Finally, Inputs are illustrated and linked
to the Strategies on which they bear
It
is generally most useful to tackle creation of a logic
map using the same "backward mapping" approach
suggested in the description above.
- Start
by listing the Goals that the project should achieve.
To the extent possible, describe these as new skills,
knowledge, or attitudes for stakeholders or as changes
in conditions bearing on the project - NOT as the
completion of project activities.
- Identify
pertinent standards (if any) and align Goals with
them - No Child Left Behind (NCLB), Enhancing Education
Through Technology (EETT), state curriculum expectations,
or others.
- Define
and clarify the Objectives that must be met along
the way to achieving the Goals, making sure Objectives
are not simply the completion of activities undertaken
in the project. Redefine if necessary.
- Define
the Strategies - the broad approaches - that will
be implemented in an effort to meet project Objectives.
If a large number of Strategies result, it may be
that either (a) some are actually activities - actions that make up Strategies - and they need
to be clustered with other activities, or (b) the
project is too broad or over-reaching in its definition,
and should be narrowed.
- List
the Inputs that should be considered under each strategy.
These are the resources available and contextual issues
that bear on project implementation (e.g., budget
constraints, relevant policies, curriculum requirements,
or data describing stakeholders' needs).
- Consider
rearranging logic map elements vertically in their
columns, putting them in some order (e.g., chronological)
that makes sense to the evaluation planning team
- Finally,
work through the logic map columns to be sure that
all logical linkages among elements have been illustrated
A
tool designed to assist you in reviewing your logic
map is the Logic
Map Self Check (PDF)
It
might prove useful to start this process by using the CAPE
Logic Map Template (Inspiration 7.5), completing
the elements with content defined for your project.
Using
the Logic Map
Importantly,
a logic map is a means to an end, rather than an end
unto itself. It is a conceptual tool that serves as
the foundation for subsequent steps of the evaluation
planning and implementation processes.
As
the evaluation process unfolds, the logic map is a living
document that will likely change as it is used. Project
managers should regularly hold their implementation
up against the logic map, to assess the degree to which
project implementation aligns with it. It may be necessary
to adjust the logic map to reflect changes in consensus
understanding about underlying theories driving project
design and implementation. It is also useful to overlay
onto the logic map information about data collection
and analysis, to ascertain the degree to which a complete
evaluation picture is developing.
Most
immediately however, the logic map should facilitate
the accurate definition of project Goals and Objectives,
and of the Strategies applied in attempts to achieve
them.
Next > Outcomes: Goals and Objectives,
and Their Relationship to Strategies |
