Introduction
What are the basic questions an evaluation can answer?
What is involved in conducting an evaluation?

Introduction

Program managers and staff frequently informally assess their program's effectiveness: Are participants benefiting from the program? Are there sufficient numbers of participants? Are the strategies for recruiting participants working? Are participants satisfied with the services or training? Do staff have the necessary skills to provide the services or training? These are all questions that program managers and staff ask and answer on a routine basis.

Evaluation addresses these same questions, but uses a systematic method for collecting, analyzing, and using information to answer basic questions about a program - and to ensure that those answers are supported by evidence. This does not mean that conducting an evaluation requires no technical knowledge or experience - but it also does not mean that evaluation is beyond the understanding of program managers and staff.

What are the basic questions an evaluation can answer?

There are many different types of program evaluations, many different terms to describe them, and many questions that they can answer. You may have heard the terms formative evaluation, summative evaluation, process evaluation, outcome evaluation, cost-effectiveness evaluation, and cost-benefit evaluation. Definitions of these terms and others and selected resources for more information on various types of program evaluations are provided in the appendix.

You may have also heard the terms "qualitative" and "quantitative" used to describe an evaluation. However, these terms, which are defined in the glossary, refer to the types of information or data that are collected during the evaluation and not to the type of evaluation itself. For example, an outcome evaluation may involve collecting both quantitative and qualitative information about participant outcomes.

There are two types of program objectives - program implementation objectives and participant outcome objectives. Program implementation objectives refer to what you plan to do in your program, how you plan to do it, and who you want to reach. They include the services or training you plan to implement, the characteristics of the participant population, the number of people you plan to reach, the staffing arrangements and staff training, and the strategies for recruiting participants. Evaluating program implementation objectives is often referred to as a process evaluation. However, because there are many types of process evaluations, this manual will use the term implementation evaluation.

Participant outcome objectives describe what you expect to happen to your participants as a result of your program, with the term "participants" referring to agencies, communities, and organizations as well as individuals. Your expectations about how your program will change participants' knowledge, attitudes, behaviors, or awareness are your participant outcome objectives. Evaluating a program's success in attaining its expectations for participants is often called an outcome evaluation.

An evaluation can be used to determine whether you have been successful in attaining both types of objectives, by answering the following questions:

• Has the program been successful in attaining the anticipated implementation objectives? (Are you implementing the services or training that you initially planned to implement? Are you reaching the intended target population? Are you reaching the intended number of participants? Are you developing the planned collaborative relationships?)
  
  
• Has the program been successful in attaining the anticipated participant outcome objectives? (Are participants exhibiting the expected changes in knowledge, attitudes, behaviors, or awareness?)

A comprehensive evaluation must answer both key questions. You may be successful in attaining your implementation objectives, but if you do not have information about participant outcomes, you will not know whether your program is worthwhile. Similarly, you may be successful in changing participants' knowledge, attitudes, or behaviors; but if you do not have information about your program's implementation, you will be unable to identify the parts of your program that contribute to these changes.

These evaluation questions should be answered while a program is in operation, not after the program is over. This approach will allow you and your staff to identify problems and make necessary changes while the program is still operational. It will also ensure that program participants are available to provide information for the evaluation.

What is involved in conducting an evaluation?

The term "systematic" in the definition of evaluation indicates that it requires a structured and consistent method of collecting and analyzing information about your program. You can ensure that your evaluation is conducted in a systematic manner by following a few basic steps.

Step 1: Assemble an evaluation team. Planning and executing an evaluation should be a team effort. Even if you hire an outside evaluator or consultant to help, you and members of your staff must be full partners in the evaluation effort. Chapter 3 discusses various evaluation team options. If you plan to hire an outside evaluator or an evaluation consultant, Chapter 4 provides information on hiring procedures and managing an evaluation that involves an outside professional.

Step 2: Prepare for the evaluation. Before you begin, you will need to build a strong foundation. This planning phase includes deciding what to evaluate, building a program model, stating your objectives in measurable terms, and identifying the context for the evaluation. The more attention you give to planning the evaluation, the more effective it will be. Chapter 5 will help you prepare for your evaluation.

Step 3: Develop an evaluation plan. An evaluation plan is a blueprint or a map for an evaluation. It details the design and the methods that will be used to conduct the evaluation and analyze the findings. You should not implement an evaluation until you have completed an evaluation plan. Information on what to include in a plan is provided in Chapter 6.

Step 4: Collect evaluation information. Once you complete an evaluation plan, you are ready to begin collecting information. This task will require selecting or developing information collection procedures and instruments. This process is discussed in Chapter 7.

Step 5: Analyze your evaluation information. After evaluation information is collected, it must be organized in a way that allows you to analyze it. Information analysis should be conducted at various times during the course of the evaluation to allow you and your staff to obtain ongoing feedback about the program. This feedback will either validate what you are doing or identify areas where changes may be needed. Chapter 8 discusses the analysis process.

Step 6: Prepare the evaluation report. The evaluation report should be a comprehensive document that describes the program and provides the results of the information analysis. The report should also include an interpretation of the results for understanding program effectiveness. Chapter 9 is designed to assist you in preparing an evaluation report.