Program Evaluation: Program Logic
Step four will help you develop a logical model for your program (learn more about the other steps in general program evaluations):
- What is a Logic Model?
- Benefits of Using Logic Modeling
- Pitfalls and How to Avoid Them
- Steps to Developing a Logic Model
Logic modeling is a thought process program evaluators have found to be useful for at least forty years and has become increasingly popular with program managers during the last decade. A logic model presents a plausible and sensible model of how the program will work under certain environmental conditions to solve identified problems. The logic model can be the basis for a convincing story of the program's expected performance – telling stakeholders and others the problem the program focuses on and how it is uniquely qualified to address it. The elements of the logic model are resources, activities, outputs, short, intermediate and longer-term outcomes. Some add the customers reached, as well as the relevant external contextual influences, present before a program begins or appearing as the program is implemented.
While logic models may take many different forms, including narrative and table form, a basic logic model diagram is shown in the figure below.
Benefits of developing and using logic models are:
- The model helps communicate the program to people outside the program in a concise and compelling way
- The model helps program staff to gain a common understanding of how the program works and their responsibilities to make it work
- Choosing a small set of performance indicators based upon a logic model:
- Keeps attention on all aspects of performance, balances the perturbations that measurement puts in the system.
- Informs the timing of indepth evaluations (e.g., there is no reason to look for outcomes if resources haven't arrived)
- Attribution of outcomes to the program is partially demonstrated by showing the related program activities and outputs.
Logic models can be used to design new programs or to confirm that an existing program design is still reasonable under current circumstances. Similarly, a logic model can confirm that the set of existing performance measures covers key aspects all along the performance spectrum.
Here is a logic model with the types of evaluations and relevant evaluation questions overlaid on it. You can see, for example, that process evaluations are more about inputs to outputs and impact evaluations are more about the sequence of outcomes.
For an example of a generic logic model that encompasses most EERE programs, and how this model helps choose performance measures, see this presentation.
There are three primary areas of criticism of logic models. These are listed in the table below along with some of the ways to avoid these pitfalls (McLaughlin and Jordan, Exhibit 1.1 in Handbook of Practical Program Evaluation, 2004). The first is that people take so much time and resources doing logic modeling that there aren't enough resources left to complete whatever task the logic model was to facilitate. A second criticism is that logic models are too linear and not only is little in today's world linear, or people tend to assume a temporal sequence that may not be the case. A third area of criticism is that the models themselves or the use of the models is rigid rather than dynamic and thus doesn't capture the change inherent in the program and its circumstances.
|Time and Resource Sink||Avoid trying for perfection. If time and resources are limited, construct only a high level of logic model (absent detailed representation of pathways.)
Plan cost and schedule to include downstream activities such as choosing performance measures or planning next steps.
Include all benefits in cost-benefit analysis, including team building, benefits to stakeholders as well as evaluator.
Actual total costs might be lower if the effort avoids costly premature impact evaluation or costly program design or implementation flaw.
|Too linear||Recognize that the linear view can often be a helpful simplification. Acknowledge the feedback by showing feedback but without attempting to highlight every possible feedback link.
Use the "Z" form to show concurrent or sequential logics with shared responsibility for impact.
Color code to show aspects such as timing.
Use a tree-type diagram with roots, trunk, branches, leaves and fruit.
Use spirals or 3 dimensional drawings.
|Rigid use, not dynamic, Not responsive to new information||Develop with program staff and stakeholders. Indicate length of time for components of the logic model. Also indicate if a component of the logic is associated with a critical program decision stage.
Use and revisit at least annually, check assumptions, since program operations often shift as staff measure and respond to performance.
Keep an eye on developments in the program context that influence performance.
Here are the steps a program can take to develop and use a logic model, (McLaughlin and Jordan, 1999 and 2004.)
First: Collecting the relevant information
Whether designing a new program or describing an existing program, it is essential that the evaluator or work group collect information relevant to the program from multiple sources.
The information will come in the form of program documentation such as strategic plans, previous program evaluations, pertinent legislation and regulations and from interviews with key stakeholders both internal and external to the program.
Conducting a literature review to gain insights on what others have done to solve similar problems, and on social science theories about the R,D&D process show how others have described the logic.
2nd: Describing the problem the program will solve and its context
Clearly defining the need for the program is the basis for all that follows in the development of the logic model. The program should be grounded in an understanding of the problem that drives the need for the program.
This understanding includes understanding the problems: who is involved, and what factors "cause" the problems. It is these factors that the program will address to achieve the longer-term goal of solving the problem. In other words, start with the big picture and the program goals and strategies.
3rd: Defining the Elements of the Logic model
Building a logic model usually begins with categorizing the information collected into "bins", or columns in a table. Each column is an element of the logic: resource, activity, output, short-term outcome, intermediate outcome, long-term outcome or external factor.
Not every program detail has to be identified and cataloged, just those that are key to stakeholder understanding of how the program works.
Check the accuracy and completeness of the information contained in the table by asking "How" and "Why" questions. For any table entry, ask the question, "How did we get here?" For example, for a particular short-term outcome, is there an output statement that leads to this outcome? This begins to organize the elements in the table into chains of activities, outputs, and a sequence of outcomes.
4th: Describing the Logic is Diagram and Text
Using the program elements in the table, a logic model diagram can show more complex logical relationships than the one-to-one linear relationships in a table. A diagram enables the logic model audience to understand and evaluate the hypothesized linkages.
The notion is to pare down concepts and words so the essence of a program's logic can be displayed on a single page. Such brevity cannot stand alone, however, and thus accompanying text should be developed to guide people through the logic and provide the detail or examples needed to understand the logic.
Logic models can also be developed for activities or sets of activities, or various targeted customer groups to provide detail program staff or particular stakeholders need to understand their portion of the logic model.
5th: Verifying the Logic model with Stakeholders
The work group responsible for producing the Model should continuously evaluate the Model with respect to its goal of representing the program logic -- how the program works under what conditions to achieve its short, intermediate, and long-term aims.
The verification process followed in Step 3 should be done with a broader group of appropriate stakeholders engaged in the review process. During this time, the work group also can address what critical information they need about performance, setting the stage for evaluation and measurement plans.
6th: Using the Logic Model in Monitoring and Evaluation
The idea is to first describe the logical framework for the program and then determine what to assess, both ongoing monitoring and program evaluations. Performance indicators are defined by the boxes, such as the quality and quantity of outputs or whether or not an outcome has occurred. Evaluation questions are the arrows between boxes, such as "did these activities cause the outcomes and why?"