There are many levels of evaluation that a program administrator may choose, depending on the target audience and the nature of the program. For those who are reporting outcomes to funding sources or seeking to find grant funds for continuation, a higher level of rigor is required. For small programs operated by agencies with limited budgets and/or where program improvement is the primary need, a less formal evaluation design may be acceptable.

Therefore, it is important to consider these factors before beginning to design an evaluation plan. For formal evaluation designs, the plan should be devised and finalized BEFORE the program begins. Usually collaboration with a trained evaluation consultant or group is advised. The consultant should participate in the design and then carry out the data collection and analysis, resulting in a final report. For less formal "post-only" designs, the program staff can usually plan and conduct the evaluation, using the types of tools referenced here. In all cases, more than one person should be involved in the evaluation team.

Once the target audience(s) for the evaluation results has been established, the next step is to define the evaluation questions. These must be specific questions to which the proposed audience needs to know the answers in order to decide if the program is being effective. Generally these questions will relate to the program goals, assuming that those goals have been formalized as part of the program design. If not, goals should be developed as part of the evaluation plan. What is it that the program was designed to accomplish in the first place?

Evaluation questions may fall into the following categories:

  • Program Implementation
    e.g., Extent to which requests for mentoring are met.
  • Mentee Perceptions
    e.g., Extent to which mentees had positive feelings about the mentoring experience.
  • Mentee Outcomes
    e.g., Whether mentees improved their school grades.
  • Mentor Perceptions
    e.g., Extent to which mentors found the experience to be positive.
  • Mentor Outcomes
    e.g., whether mentors improved their relationship skills.
  • Community Outcomes
    e.g., Extent to which school dropouts decreased
For some evaluations, a careful analysis of the relationship between program goals, activities and outcomes is expected. Logic models showing the causal nature of the relationships must be developed. The mentoring programs funded under the Juvenile Mentoring Program (JUMP) usually have this requirement.

Even for the less formal evaluation designs, a distinction should be clearly made between "descriptive" information and "change measurement" information. The former usually can be gathered at a single point in time to describe characteristics of the program participants. Change measurements, on the other hand, require at least two measurements, one before intervention and one after, to determine whether change has taken place. Dealing with these issues will also help the team to decide on an evaluation design that will collect the data necessary to answer the specified questions to the satisfaction of the defined audience(s). Experimental or quasi-experimental designs are described in several of the references listed here. The use of a pre-post versus a post-only design is often determined by the questions that have been formulated earlier in the process.

Data collection instruments must then be identified or developed specifically for this evaluation. Of course, the statistical manipulation of the data, once it has been gathered, requires further deliberation and decision making by the evaluation team. In addition, the team must carefully think out the process of analyzing the results and determining how best to disseminate the information.

In order to assist program managers with all of these tasks, an annotated set of resources has been gathered and appears below. Please consider your situation and determine which of these references might be most helpful to your team. Some of the resources are available online, while others must be ordered in print form. Your feedback on the use of this list will be appreciated. Use the Contact page of this site to send feedback.

RESOURCES FOR PROGRAM EVALUATION

  • E. Mertinko, L. C. Novotney, T. K. Baker, and J. Lang. (2000). Evaluating Your Program: A Beginner's Self-Evaluation Workbook for Mentoring Programs. Washington, DC: Office of Juvenile Justice and Delinquency Prevention.
This workbook is designed to walk an evaluation team through each of the various steps of planning and conducting a comprehensive program evaluation. The workbook grew out of the work Information Technology International is currently doing to evaluate the Juvenile Mentoring Programs (JUMP), supported by the federal Office of Juvenile Justice and Delinquency Prevention. The manual stresses, however, that it is designed to be used with JUMP and non-JUMP programs alike. The loose-leaf manual covers all types of evaluation designs and provides a chapter-by-chapter example of a fictitious organization conducting an evaluation. Many worksheets and data collection devices are include, some of which have samples already filled out for the fictitious organization.

Program managers wishing to take a look at a comprehensive, yet not overly technical, evaluation-planning process should consider using this workbook. The references at the end of each chapter may help lead the program manager to other sources of guidance.

This workbook is currently available, free of charge, to any youth program interested in designing and conducting a local evaluation. You may receive a copy of this publication by contacting ITI at 301-765-0060, or by sending an e-mail message to ITI@ITIincorporated.com Find out more by checking the ITI website at http://www.itiincorporated.com

  • R. Saito. (2001) What's Working: Tools for Evaluating Your Mentoring Program. Minneapolis, MN. Search Institute.
This workbook helps the program evaluator to think through the evaluation process and conduct a thorough evaluation, using the data collection forms, analysis sheets and reporting formats included in the manual. The process is limited, however, to "post only" evaluation designs, thus limiting the usefulness of the tools. The printable mentor and mentee surveys and the data collection forms will prove very useful to those conducting a limited analysis of their programs.

The $24.95 workbook can be ordered from the Search Institute at http://search-institute.org/catalog or by calling 877-240-7251.

  • D. Gottfredson, M. Harmon, G. Gottfredson, E. Jones, and J. Celestin. (1999) Outcome Measures for Prevention Programs-A Handbook of Instruments for ATOD Program Evaluation. Ellicott City, MD. Gottfredson Associates.
A compendium of information about a variety of evaluation instruments, this handbook organizes information according to the problem behavior that a program manager may wish to measure and then reviews those evaluation instruments that are considered "promising" for measuring that type of behavior. It is really not a "handbook of instruments" but a handbook of information about instruments.

This manual is available for $35 from Gottfredson Associates, 3239 B Corporate Court, Ellicott City, MD 21042. The website address is www.gottfredson.com

  • National Mentoring Center. (2001) Strengthening Mentoring Programs Training Curriculum; Module 5, Measuring Outcomes. Portland, OR. Northwest Regional Educational Laboratory.

This is part of the national training curriculum for mentoring programs. Module 5 deals with measuring outcomes. This is public domain information, allowing program staff to copy or use the material for their own programs, as long as the publisher is acknowledged. This module addresses the following objectives:

      Know how to define concrete outcomes for your program.
      Understand data collection strategies.
      Know how to measure the quality of relationships between mentors and youth.
      Understand the uses one can make of outcome findings.
A six-step plan for planning and carrying out measurement of outcomes is included.

This curriculum is available from the National Mentoring Center, Northwest Regional Educational Laboratory, 101 Southwest Main, Suite 500, Portland, Oregon 97202. The web address is http://www.nwrel.org/mentoring

  • J. B. Grossman and A. Johnson. (1999). "Assessing the Effectiveness of Mentoring Programs." In Contemporary Issues in Mentoring. Edited by J. B. Grossman. Philadelphia, PA. Public/Private Ventures.
"Grossman and Johnson discuss and provide three types of measures that mentoring programs can use to assess their own effectiveness: changes in participant outcomes, measures of effective relationships and descriptions of participant characteristics. Directly measuring change in specific outcomes is what many think of as the only way to demonstrate program effectiveness. However, youth behaviors and attitudes change over time as a result of maturation, not just program effectiveness. In order to interpret changes in outcomes correctly, operators need to have examples of typical changes against which they can compare the changes they measure. Chapter 2 provides examples of these typical changes. In addition, the chapter provides program operators with other techniques with which to evaluate their programs, in particular, benchmarks of programmatic quality-which is often easier to measure-yet are empirically linked to impacts on academic behavior, initiation of drug and alcohol use, and self-esteem. These benchmarks include length of relationship, frequency of contact, and various measures of the quality of the relationship (as perceived by the youth and program staff)." [from website description.]

The document is available for $15 from Public/Private Ventures, One Commerce Square, 2005 Market Street, Suite 900, Philadelphia, PA 19103. The web address is www.ppv.org

All of the above documents are available for loan from the Council on Drug and Alcohol Abuse, 630 Janet Avenue, Lancaster, PA 17601. Use the Contacts page of this site to find out more information.