Introduction
Setting Goals
Mentoring Goals
Improving and Building Skills
Using Mentoring to Support Program Quality
Quality Improvement
Developing an Implementation Plan
Evaluating the Process and Impact of Mentoring
Outcome Evaluation
Putting It All Together: Planning and Evaluation

Regarding program planning, the Head Start Program Performance Standards say...

1. Grantee and delegate agencies must develop and implement a systematic, ongoing process of program planning...1304.51(a)(1)

Note: Planning for a mentoring program is part of the program planning process.

Take Stock! If a mentoring advisory committee is formed, who can serve on it to help our agency develop or strengthen our mentoring program? What contributions would they bring?

Take Stock! Where can our agency get information to help us identify our goals for mentoring? How can these sources help us?

Mentoring, like any new initiative, will not occur overnight. Planning is key to successfully launching a mentoring program. Planning and process evaluation together form a continuous improvement cycle that strengthens mentoring. Outcome evaluation is also an inherent part of the planning process, since outcome evaluations measure attainment of goals (see ...[the "Evaluating the Process and Impact of Mentoring" section below] for a full definition and description of these types of evaluations). Including both kinds of evaluation in the planning process ensures that agencies will identify and collect relevant data from the outset.

The importance of the planning process is emphasized in the Head Start Program Performance Standards and Program Guidance. In elaborating on the standards, the Guidance [(45 CFR 1304.51(a)(1)(i)-(iii))] describes program planning as a "continuous cycle, involving key members of the Head Start community. Planning, therefore, is critical for setting clear program goals and for defining an organized approach to program services driven by the specific priorities of the community."

The cycle of program planning, implementation, and evaluation is a dynamic process, during which ideas are shared and discussed. To ensure that the organization as a whole moves forward to reach the mentoring goals, agencies may want to form a mentoring advisory committee with representation from all key stakeholders or use an existing education advisory committee. This committee can meet regularly to guide the mentoring process.

This chapter focuses on the key elements of a planning and evaluation process:

• Setting goals
• Developing an implementation plan
• Evaluating how well the mentoring initiative is working and whether it is having an effect on program quality.

Setting Goals

Identifying long-term and short-term goals is an integral part of the planning process. The goal-setting process used for the mentoring program will mirror the process used for overall program planning as described in the Head Start Program Performance Standards. The following chart describes how goal-setting principles could apply to mentor program planning.

Goal-Setting Principles

Goals stem from a thorough assessment of the agency's program, including:

• Community assessment
• Federal monitoring review
• Ongoing monitoring
• Self-assessment
• Assessment of protégés in the context of specific children and families
• Parent and partner agency feedback
• Staff evaluations.

How this principle applies to a mentoring program
Agencies look at how specific program needs may be met by the kind of professional development and quality improvement that mentoring has to offer. Setting meaningful goals requires a clear understanding of both organizational needs and the purpose of the mentoring initiative.

Goal-Setting Principles

Goal setting is an ongoing, dynamic process.

How this principle applies to a mentoring program

Goals are based on needs, but goal setting is not a one-time event. As programs grow and develop, goals may change. Goals are revised on the basis of ongoing evaluation of the mentoring program.

Goal-Setting Principles

Goal setting helps establish agreed-on priorities.

How this principle applies to a mentoring program

"Agreed on" priorities help ensure buy-in from all stakeholders. Stakeholders include the agency's management staff, Policy Council representatives, mentors, and protégés. Community representatives are included as appropriate.

As noted, assessment is the first step in the goal-setting process. Allocate ample time for assessment, and gather information from multiple sources.

Mentoring Goals

Take Stock! From our assessment, what have we learned about our agency's goals for mentoring? How will we define our mentoring goals?

Mentors can be helpful during orientation because they are -

• Available on an ongoing basis to answer questions, help solve problems, and provide encouragement and support
• Nonjudgmental, because their role is to support teachers to do the best that they can do
• Models of best practices in early childhood education. How better to learn than by watching a master firsthand!

Take Stock! How can mentoring enhance our program quality?

Staff who feel supported and rewarded by an agency are more likely to remain with the agency.

The primary goal of Head Start mentoring is to improve outcomes for children by enhancing staff skills and program quality. Mentoring can improve program quality by

• Helping new staff learn about program policies and procedures during orientation
• Improving and building the skills of current staff
• Increasing the retention of qualified staff
• Assisting programs seeking accreditation.

Orientation

One way to improve program quality is to provide teachers and home visitors from the outset with information they need to perform effectively. In addition to overall information about Head Start, new staff need to learn how the Head Start Program Performance Standards are implemented within a particular agency. Mentoring can be an effective strategy for improving the orientation of new teachers and home visitors.

Improving and Building Skills

The most common reason to start a mentoring program is to enhance staff skills and knowledge that will improve the quality of services for children. Professional development provided through mentoring can improve the quality of teaching and enhance child outcomes. Taking classes, attending workshops, or going to summer institutes often has little impact on teachers' and home visitors' practices because these types of training are not connected to the environment in which they work. They offer few opportunities to learn by doing or to reflect on practice with their colleagues. Because mentoring is rooted in experience, it can profoundly change what staff actually do.

Retention

Mentoring can influence teacher retention in several ways. First, as both new and more experienced teachers improve their skills through mentoring, they are likely to feel more competent and more content as teachers. Both mentors and protégés reap rewards during the mentoring process. Some of these rewards may be tangible, such as salary increases or certificates of achievement. Teachers experience individualized, quality mentoring attention, and they derive intrinsic rewards from learning in a nonevaluative environment. Mentors derive satisfaction from nurturing a teacher's professional growth. The recognition of one's own potential to serve as a mentor also can have a positive impact on teacher retention.

Accreditation

Another quality improvement goal is to have Head Start centers become accredited by the National Association for the Education of Young Children and by other accrediting bodies, such as the National Family Child Care Accreditation Program. Mentoring can help teachers understand accreditation criteria and assess whether these criteria are being met in their center-based and family child care settings. Mentoring, especially from someone who is knowledgeable about and has undergone the accreditation process, can provide important support.

No matter what goals are set for the Head Start program's mentoring approach, it is important to make sure that the choices made serve the desired outcomes. For example, if mentoring is used to orient new teachers, the mentors need to be experienced, expert professionals who are available during the crucial first few days on the job. If the mentor's role is to work with experienced teachers, mentors must understand how to give feedback to teachers who may feel that they "know" all about best practices in early childhood education. The chart on page 11 highlights mentoring goals that some programs have set.

Using Mentoring to Support Program Quality

Program Goal

Orientation

Description

Mentors are available to new teachers just starting out to provide advice, assistance, and support. Mentors serve as models of best practices and help new teachers learn how best to do their jobs.

Orientation

Sample Programs That Have Addressed This Goal

At the Upper Des Moines Opportunity, Inc. Head Start in Iowa, mentors at the same job level pair with new teachers from their first day. The new person shadows the mentor for up to a week, and then the mentor becomes a resource for the new teacher.

The PACE Head Start in Bedford, Massachusetts, offers a "Big Sister" mentoring program as part of its orientation and supervision of new teachers. During the new teacher's 90-day probationary period, veteran lead teachers from other classrooms serve as mentors. The mentoring program assists management staff in making appropriate decisions about the evaluation and retention of new teachers.

In the Lee County Pre-K Head Start program, the Florida-wide mentoring program for all new teachers is extended to the early childhood education environment. Mentors are trained in collegial coaching techniques. As mentors, they help develop new teachers' professional skills in such areas as room arrangement, classroom discipline, and curriculum work. Mentors also shepherd new teachers through routine paperwork requirements and the like.

Quality Improvement

Description

Mentors not only provide professional support but also spread the word about best and effective practices in early childhood education. Good mentors help teachers apply theory to practice. Mentoring for quality improvement means working with new and experienced teachers and individualizing the learning to meet their needs.

Quality Improvement

Sample Programs That Have Addressed This Goal

The Miami Valley Child Development Centers in Ohio use mentoring to improve teaching quality by setting up opportunities for peer-to-peer mentoring. Teachers in this Head Start program are represented on a teacher steering committee, which organizes workshops of interest to the staff. At these workshops, education staff have the opportunity to share ideas and learn from one another. Through this peer-mentoring process, teachers can troubleshoot, problem solve, and support each other in implementing their solutions. This contact improves their teaching, which means better outcomes for the children.

The Northwest Arkansas Family Child Care Association launched a mentoring program to help family child care teachers become accredited. The Homes Uniquely Giving Support (H.U.G.S.) program provided accreditation training to all protégés and mentors on one Saturday morning each month over a six-month period. The program trainer and the project coordinator provided the training, and mentors served as facilitators for the protégés they mentored. The more experienced family child care teachers mentored the less experienced family child care teachers in areas identified as needing improvement for accreditation.

Developing an Implementation Plan

Head Start agencies are required to engage in a continuous improvement cycle through the ongoing monitoring and self-assessment systems they must establish in accordance with Section 45 CFR 1304.51(i)(1) of the Head Start Program Performance Standards.

In evaluating the mentoring program, consider collecting information from some or all of the following sources:

• Records (staff evaluation forms, program plans for child development services, weekly or monthly work plans, periodic assessments, etc.)
• Observational tools, such as the Early Childhood Environment Rating Scale (ECERSR; 1997), Infant/Toddler Environment Rating Scale (ITERS), The Classroom, Family Child Care or Socialization Observation Instruments and the Home Visit Observation Instrument from the Head Start Program Instrument for Systems Monitoring, or PRISM
• Protégés, mentors, supervisors, and parents
• Mentor program coordinator, service area manager, Head Start Director, and Regional Head Start Quality Improvement Center
• The program's PRISM results and ongoing monitoring and self-assessment processes
• An outside consultant, if applicable.

Remember, multiple sources of information lead to a more accurate picture.

Once the agency has defined its mentoring goals, it is time to write an implementation plan that describes the agency's approach to mentoring. Effective implementation plans include strategies and timelines and identify the person(s) responsible for carrying out each step. The agency's mentoring advisory committee can use this plan on a regular basis to make sure that implementation is on targetmeeting projected timelinesand that personnel have completed their assigned roles and responsibilities. As the planning and evaluation cycle continues, this plan is continually monitored and updated to reflect any changes in implementation strategies, timelines, or responsibilities.

Evaluating the Process and Impact of Mentoring

Evaluation is a crucial component of any mentoring program. A good evaluation lets an agency know that it is running the program effectively and that the program is doing what it was designed to do. Having data to show that mentoring works also puts an agency in a better position to seek and win external funding for this and other initiatives.

There are essentially two kinds of evaluations. One of these is called a process (or formative) evaluation. This kind of evaluation looks at how a program is being implemented. It provides information that is useful for improving the program. The other kind of evaluation is called an outcome (or summative) evaluation. This kind of evaluation looks at the impact the program is having.Whatever the goals of the evaluation, the time to decide on how best to measure the success of a program is at the beginning. One simple strategy is to collect information from protégés, mentors, program administrators, and the mentoring advisory committee at the beginning, middle, and end of a program year strategy because

• Information collected at the beginning of the year sets baselines to show where the programs started.
• Mid-year evaluations show whether the program is on the right track and allow any necessary adjustments to be made.
• End-of-year evaluations demonstrate whether the mentoring program has had any effect and can help in determining whether to continue the program, with or without additional adjustments.

Because the kinds of information that process and outcome evaluations collect are different, each evaluation is discussed separately below.

Process Evaluation

Take Stock! How can our agency evaluate the mentoring process? Who should be involved?

An evaluation, even an informal one, is integral to an agency's continuous improvement process and can lead to important changes in programs.

As a result of its first-year evaluation, the Community Action Program of Evansville (CAPE) Head Start in Indiana found that some parts of the program weren't working. The program then made the following changes:

• Changing from no extra pay for mentors (only a trip to the National Head Start Association conference) to a yearly stipend
• Making Mentor Teacher a step on its five-step career ladder for teachers
• Changing from having the mentor and protégé in different classrooms and at different sites to placing them in the same classroom
• Adding extra training for mentors and even sending them to out-of-town workshops.

The Associated Day Care program in Massachusetts conducted an evaluation with program directors, mentors, and protégés. As a result of the evaluation, the program hired substitutes to release mentors for two full days of mentoring each month.

A process evaluation provides information that is useful in improving the program. It also can show how the supports that agencies have implemented for mentoring are being used. For example, a process evaluation might ask the following:

• Are mentors actually meeting with their protégés as planned?
• If not, what obstacles are keeping them from doing this?
• How can the obstacles be overcome?

Process evaluations can also answer questions such as these:

• Do the teachers need substitute coverage or release time?
• Are mentors obtaining the kind of follow-up support they need? If not, why not?
• Are attendees satisfied with the workshops related to mentoring that they are attending?

The way to uncover this kind of information is, quite simply, to ask the people involved. Information can be collected through informal discussions, written questionnaires, structured interviews, and focus groups. Using well-established questionnaires or observation tools is a good idea for at least two reasons: (1) it is easier there is no use reinventing the wheel by making up a questionnaire and (2) the quality of the questionnaire (its reliability and validity in technical terms) is usually already established.

Ideally, a neutral party should collect and confidentially summarize the evaluation information. In this way, people participating in the program can be honest and not worry about hurting anyone's feelings or reputation.

Agencies can use the instruments that Federal monitoring review teams use as models of how to conduct a process evaluation of the mentoring initiative. The Federal monitoring review process stresses multiple modes of inquiry as suggested above. The process gathers information through group and individual interviews, observation, and document reviews. Two Core Questions in the Federal monitoring review instrument are devoted to evaluation. One looks at ongoing monitoring and the other deals with self-assessment. The two Core Questions focus on measuring progress and effectiveness in meeting goals and objectives, as well as on collaborative participation in the evaluation process.

Consider the following outcomes:

Program outcomes

• Improved organizational communication
• Increased organizational commitment to teachers
• Improved morale
• Cost-effective professional development
• Staff retention

Mentor outcomes

• Professional advancement
• Revitalized interest and commitment to work
• Personal fulfillment

Protégé outcomes

• A feeling of being supported and acknowledged at work
• More effective interactions with children
• Engages in reading activities to develop early literacy

Child outcomes

• Better verbal skills
• More cooperative behavior
• Fewer problem behaviors
• Broader vocabulary
• Improved literacy or pre-literacy skills
• Improved numeracy or other mathematical skills

Observation gives you a richly detailed picture of?

• How teachers are implementing a given curriculum
• How they manage children's behavior
• How they accomplish transitions
• How they make decisions.

Take Stock! How can our agency determine whether we have met our goals for mentoring?

Outcome Evaluation

An outcome evaluation focuses on whether the program is achieving its goals. To start, look at the issues and needs identified at the outset of the planning process. What are the goals of the program? If the mentoring initiative was designed to retain skilled, effective teachers, for example, the outcome evaluation looks at whether being a mentor (or a protégé) improves job satisfaction and retention.

One principle to keep in mind in designing an outcome evaluation is that no one piece of information can prove that mentoring has made a difference. Protégés alone, for example, can't tell the whole story. Nor can mentors alone. To get the overall picture, it is necessary to look at mentors, protégés, and other teachers; program administrators; and children and perhaps families.

Because people's responses are affected in different ways by the way a question is asked, a well-conceived evaluation not only asks multiple people to tell their stories, but also collects information from many different sources, using a variety of collection methods. For example, although a questionnaire might contain appropriate questions, people might be rushed when filling it out and might not take the time to write all they know. An interview can be a way to let people provide information about mentoring outcomes in a relaxed, open way.

The kinds of questions asked in an outcome evaluation depend on the goals of the mentoring program. Questions may include the following:

• Are teachers interacting more sensitively with children?
• Are teachers more confident in their professional roles?
• Are teachers using space appropriately?
• Are children showing progress on their ongoing assessments?
• Are children acquiring the social and emotional skills that they will need for school success?

Another effective way to assess mentoring outcomes is through observation. Observing teachers provides different information than does asking teachers about what they are doing. A number of agencies use the Early Childhood Environment Rating Scale-Revised, the Infant/Toddler Environment Rating Scale, the Early Childhood Observation Instrument, or the Assessment Profile for Early Childhood Programs to assess pre- and post-mentoring practices.

Observations should occur at the beginning of the year to obtain baseline data. Then midcourse and final observations can be compared with the initial observation to see what has changed. Observers need to be trained before using any of these observation instruments. Information collected through observations during the programs annual self-assessment process is also useful.

The ongoing assessment of children's skill development also provides valuable data on mentoring outcomes. For example, such assessments provide valuable information about children's progress in a variety of developmental areas. Parents also are good sources of information about their children's attitudes toward Head Start and Early Head Start and about what their children are accomplishing in the program. Parents may also be able to provide information on changes they have noticed in their child's classroom as a result of mentoring.

Doing a formal evaluation is not easy. It takes time, energy, and creativity. The results pay off, though, when an agency not only can say that it has done something well but can prove it.

One part of doing an evaluation really well is finding something to compare findings against. Some ideas include the following:

• Network! Find a Head Start administrator or manager who isn't doing a mentoring program and who might let a staff member drop by to observe teachers who are not mentoring or being mentored.
• Ask! Because Head Start is a National Laboratory for early childhood education and development, in many places, people are collecting information about comparable programs.
• Search! Graduate students are often looking for research projects. Head Start programs can provide a research laboratory for these students.
• Wait! If outside information isn't available, use before-and-after data to make internal comparisons. Take measurements weekly or monthly for a while, then start the mentoring program. Compare "after" data with the "baseline" measurements.

When possible, a comparison group can be a powerful evaluation tool. Comparison groups show whether any improvements achieved are in fact due to the mentoring program and not to something else, like time, attention, or natural development. Here are two examples of how some programs have used comparison groups:

Program: California Early Childhood Mentor Teacher Program

How they used comparison groups Evaluators compared their protégés with two different comparison groups: community-based child care teachers and student teachers in a college lab school practicum. Because they included these groups, the evaluators were able to show that mentors' classrooms were indeed of higher quality than were those in the general community and that protégés were more effective teachers than the community comparisons.

Program: Montana Beginning Teacher Support Program

How they used comparison groups This mentoring program for new teachers in public schools included a comparison group of 21 unmentored teachers who were otherwise similar to the protégés. Program administrators were able to show better retention among protégés than among members of the comparison group (91% of protégés stayed with their teaching careers, compared with 73% of the comparison group members).

There is no question that evaluation is seen by some as an "added burden" or as "gravy" for any program initiative. However, evaluation establishes accountability. Accountability doesn't have to mean that the system is punitive. It means that the agency

• Is responsible for what it delivers
• Wants to correct problems early
• Is interested in continuous quality improvement
• Is ready to stand by what it does.

Evaluation is an effective way of supporting accountability.

Putting It All Together: Planning and Evaluation

Quality mentoring is a result of thoughtful planning and continuous improvement. Planning ensures that the program starts off on the right foot. Evaluation is the way to "learn by doing," to measure progress against goals, and to determine whether the program is effective. Evaluation feeds back into the planning cycle. Evaluation results serve as the basis for changes that make the program more effective. Involving staff, mentors, protégés, Policy Council members, parents, and outside consultants, as appropriate, ensures that the "best thinking" has gone into the design, implementation, and evaluation of the program and that all relevant stakeholders buy into the program.