Consulting with Experts and Practitioners
Reviewing Measures and Assessment Tools
Field Testing a Pilot NRS in Diverse Head Start Programs
SINCE THE ANNOUNCEMENT of the National Reporting System (NRS) in the President's
Early Childhood Initiative— Good Start, Grow Smart— in April 2002,
the Head Start Bureau has been working to develop the best possible design
for this important new effort.
The goal of this article is to explain the
consultation, planning, and field test strategy which led to the NRS design
for implementing the President's mandate and for strengthening the effectiveness
and credibility of Head Start programs.
Consulting with Experts and Practitioners
THE BUREAU CONVENED AND PARTICIPATED in an
ongoing series of focus groups, workshops, and discussion sessions with leaders
from local Head Start programs, early childhood researchers, and experts on
assessment strategies for young children. These activities broadened our awareness
of the strengths and limitations of currently available assessment tools,
including those most commonly used in Head Start programs for ongoing assessment
of children, for indicators of school readiness, and for research on key program
components. Issues and concerns around development of the NRS were shared.
The major events included:
May 2-3, 2002. Focus Group on Child Outcomes with 17 Head Start managers,
national assessment experts, and Head Start Bureau staff.
June 17-18, 2002. Early Childhood Education
and School Readiness Workshop: Conceptual Models, Constructs and Measures.
This session was convened in collaboration with the National Institute of
Child Health and Human Development (NICHD) and the Office of the Assistant
Secretary for Planning and Evaluation (HHS). Forty-eight experts, including
representatives from several Head Start grantees involved in research partnership
efforts, attended.
July 9, 2002. Discussion Session on the National Reporting System
with 11 national experts and Federal staff from NICHD, ACF, and the Head Start
Bureau.
December 16, 2002 and February 12, 2003.
Technical Work Group meetings of 16 experts, co-chaired by Dr. Craig Ramey,
Georgetown University, and Dr. Clancy Blair, Pennsylvania State University.
The Technical Work Group provides ongoing advice and support to the Bureau
as we complete the development and implementation of the NRS.
January 16, 2003. Associate Commissioner Windy M. Hill convened an
invitational conference for Head Start and Early Head Start Directors to provide
a comprehensive briefing on the status of planning and development of the
NRS.
January 16 and 23, 2003. Focus Groups with 49 Head Start Directors
and Managers discussed the NRS and its relationship to their local system
for ongoing assessment, staff training, and local computer capability.
March 5, 2003. Language Diversity Group
Meeting with six experts discussed strategies for assessment of non-English-speaking
children in the NRS.
May 2, 2003. The Technical Work Group reviewed reports of the field
testing and other work related to the NRS.
Reviewing Measures and Assessment Tools
THE HEAD START BUREAU is utilizing a variety
of important criteria in selecting the child assessment measures for the NRS
including:
- Focusing on skills and indicators that are important for predicting school
readiness and success.
- Determing the technical quality of the instruments in terms of accuracy,
validity, and reliability for measuring the progress of 4-and 5-year-old
Head Start children.
- Determining the feasibility of implementing the assessment with approximately
500,000 children in various program options.
- Limiting the total time required to administer the assessment so that
children and staff are not overly burdened.
Our first step in exploring child assessment
options involved analyzing Program Information Report (PIR) data from 2001-2
on assessment tools currently in use. This effort revealed that programs are
using 22 different assessment instruments and that more than 480 agencies
are using locally developed tools or methods. Given the many assessment tools
in use, it was determined that we could not create an NRS strategy based on
data from current, ongoing assessments. Accordingly, a single uniform package
of currently available assessments is being assembled to use in the NRS. This
approach will allow the NRS information to be used effectively and for the
intended purposes of designing and planning training and technical assistance
efforts, augmenting existing assessment information used in decision-making
about local curriculum and teaching strategies, and expanding the evidence
of program effectiveness considered in Federal program monitoring reviews.
The Head Start Bureau's commitment is to design and implement the NRS to
benefit Head Start programs and the children and families we serve.
Next, based on the President's charge to the
Bureau in the Good Start, Grow Smart initiative and the bipartisan
Congressional mandates in the 1998 reauthorization of the Head Start Act,
the Bureau commissioned an analysis of the technical quality of existing assessment
tools in the mandated areas of early literacy, language development and numeracy.
This review included tools currently in use in major studies of Head Start
children and programs (i. e., the Family and Child Experiences Survey (FACES),
the Head Start Impact Study, and the Head Start Quality Research Center Consortium).
In its review of assessment materials, the Bureau also considered input from
the various consultation sessions and from the Technical Work Group.
This process resulted in an initial set of
measures to use in the NRS field test, including measures of the following
five learning indicators as mandated by Congress in the Head Start Act in
1998:
- Understanding and using language to communicate for various purposes
- Using increasingly complex and varied vocabulary
- In the case of children whose native language is other than English,
progressing toward acquisition of the English language
- Identifying at least 10 letters of the alphabet
- Numeracy awareness
While we recognize the value of collecting
and reporting on a more comprehensive set of domains and indicators, limiting
the scope of the initial version of the NRS is a prudent and manageable way
to begin.
Field Testing a Pilot NRS in Diverse Head Start Programs
DURING APRIL AND MAY 2003, a sample NRS was
implemented in a diverse group of 36 local Head Start programs, including
two Migrant and two Indian Head Start agencies. This effort is expected to
provide valuable information on how the NRS design, materials, and reporting
system work in the programs. Representatives from each program participated
in a training-of-trainers session on the child assessment and reporting system
procedures. These participants, in turn, trained teachers and/ or other local
staff to administer the child assessment and to record and enter the assessment
information in the field test.
Staff from our NRS support contractors, Westat,
Inc. and Xtria, Inc., observed and gathered feedback on the implementation
efforts in each program. For a limited sample of children, contract staff
carried out a second assessment as a check on how well the training design
worked to prepare staff to carry out the assessment procedures in an accurate,
consistent way. Information from the field test and feedback from staff and
managers in field test sites is a crucial source of input for final decisions
on the NRS design, policy decisions, program guidance, and implementation
strategies.
In summary, we have successfully completed
initial steps in developing the NRS, including several key decisions on the
scope of the initial version of the system, the training procedures, and the
reporting mechanisms. We appreciate the contributions from Head Start, early
childhood, and research leaders in this unprecedented development of a system
to collect and use child outcomes information as a new Head Start management
and accountability tool.
In closing, the National Reporting System is a work-in-progress. We continue
to consult with experts, Head Start leaders, program staff, and parents as
we design, analyze, understand, and use the NRS. We will evaluate pilot efforts
to utilize the first year of NRS information along with other sources of information
about the quality and outcomes of Head Start programs. As we build upon the
initial experiences with the NRS efforts, we will broaden the measures of
child outcomes and accomplishments and develop procedures in other languages
spoken by Head Start children.
The Head Start Bureau's commitment is to design
and implement the NRS to benefit Head Start programs and the children and
families we serve.
PROGRAMS PARTICIPATING IN THE FIELD TEST FOR TRAINING AND IMPLEMENTATION
OF THE NATIONAL REPORTING SYSTEM (NRS) Thirty-six Head Start programs
were field-test sites for the NRS; 72 centers and 144 classrooms participated.
Approximately 1,440 children, 4 years and older, were assessed. In 17 of the
programs, at least one-fifth (20%) of the children were English language learners.
CITY, STATE
Akron, OH
Albuquerque, NM
Aliquippa, PA
Allentown, PA
Altadena, CA
Anchorage, AK
Atlanta, GA
Berlin, NH
Bridgeport, CT
Caguas, PR
Chicago, IL
Coshocton, OH
Detroit, MI
El Centro, CA
Elizabethton, TN
Fargo, ND
Fort Washakie, WY
Ft. Lauderdale, FL
Gillespie, IL
Grand Junction, CO
Hagerstown, MD
Hughesville, MD
Laredo, TX
Lubbock, TX
Nacogdoches, TX
Norfolk, VA
Plaquemine, LA
Richardson, TX
Roma, TX
Russellville, AR
San Antonio, TX
San Juan, PR
Seattle, WA
Sharon, PA
Springfield, MO
Window Rock, AZ
Tom Schultz is Special Assistant to the Associate Commissioner, Head Start
Bureau.
