In an attempt to address the critical needs of policymakers and educators for information on how to design appropriate assessment systems, SERVE convened an Institute on School Readiness Assessment Systems: Making Them Work for All Children in October 2001. There is increasing pressure for school systems and early childhood programs to demonstrate outcomes. Early childhood assessment, and how to include children with special circumstances, such as disabilities or limited English proficiency, have become increasingly important issues. Although there are no clear answers, the Institute provided participants the opportunity to grapple with the hard questions facing educators who are responsible for providing data on child outcomes and protecting the welfare of young children and the adults who are for them. We have included the notes that resulted from this two-day Institute on our policy page for this month.

Institute for Readiness Assessment Systems: Making Them Work for All Children
October 22, 2001  
Morning Session  
Establishing Purpose for Assessment Systems

Catherine Scott-Little from SERVEing Young Children opened the Institute by welcoming participants and presenters.

Background for the Institute
The state of North Carolina formed a Goal Team to develop recommendations for how North Carolina should define and measure school readiness. SERVE, the Regional Educational Laboratory for the Southeast, and the National Center for Early Development and Learning (NCECL) conducted a study to see how other states were defining and assessing school readiness. The two organizations worked together to survey all 50 states to ask their definition of “readiness” and how they assessed whether or not children and/or schools were ready. Results from the survey indicated that while many states in the union were exploring the area, only one state had a definition of readiness, and many had identified readiness as a specific chronological age a child should reach before entering kindergarten. (The final report, Readiness for School: A Survey of the States by G. Saluja, C. Scott-Little, and R. Clifford, can be found online at http://ericeece.org/pubs/.)

The North Carolina Goal Team wanted to know what assessment instruments were available in order to determine if one or more of the instruments might be appropriate for measuring school readiness. SERVE conducted an extensive review of assessment instruments. A compendium entitled Assessing Kindergarten Children: A Compendium of Assessment Instruments was created, listing roughly 40 instruments and basic information regarding the instrument. It can be obtained by calling SERVE at (800) 352-6001.

SERVE then sponsored a National Summit on School Readiness Assessment. Top researchers from the field gathered in Research Triangle Park, North Carolina, to discuss issues related to readiness assessment. Several states were invited to present their assessment system. The Summit provided useful information to participants, but children who did not fit the “norm” or typical developmental trajectory were not being addressed. Summit participants indicated that this was an area in which they needed assistance. This Institute is an effort to get participants to start discussing children with special circumstances and how to include them into assessment systems.

Readiness is a complicated issue, and we may not have all of the answers. This Institute is designed to facilitate productive discussions about important issues and may not provide concrete answers to all the issues.

Understanding Readiness Assessment
Catherine Scott-Little set the context for the discussion by reviewing assessment, difficulties encountered when creating a system, and guiding principals for assessing young children.

Why is readiness assessment so important?

The National Education Goals Panel’s Ready to Learn Goal established readiness as an important topic. Assessment is necessary to know whether our country is making progress toward the goal. Another factor that has increased the interest in readiness assessment is the growing demand for accountability from policymakers, early childhood programs, and schools. There is a greater need for data on children and schools. There is a higher occurrence for data-driven decision-making.

Why is this so difficult?

The nature of children’s development makes assessing children of this age difficult. Development is complex and multifaceted; all areas are interrelated and can affect the child’s success in school. Their development is rapid and uneven, so their abilities change quickly, making it difficult to capture in a moment what a child can actually do. Moreover, children are typically poor test takers. They have short attention spans and poor writing abilities. It is easier for them to “show” us what they can do.

No two children have the same experiences before entering school. Children are malleable and are impacted by their environment. Are our instruments assessing their abilities or where they have been? Are we assessing their previous opportunities or their potential abilities? Assessment instruments and procedures are often limited and cannot capture each child’s abilities and different experiences.

Position statements on assessing young children have been released from many early childhood organizations (see sidebar).

What principals should guide early childhood assessment?

There are three main principals for readiness assessment systems: (1) they should benefit all children, (2) the purpose for the assessment should be clear, and (3) the assessment should be sensitive to cultural and language differences.

Base the assessment on the purpose for the information. There are three purposes identified by the Institute for assessment: (1) screening, (2) supporting instruction, and (3) evaluation/accountability/policy decisions. It is rare that one single assessment instrument can fulfill more than one purpose very well. In fact, the recent report by the National Research Council entitled Eager to Learn states, “Each of these purposes represents an important opportunity for test or assessment data to inform judgment—if the tests or assessments are used carefully and well. No single type of assessment can serve all of the purposes; the intended purpose will determine what sort of assessment is most appropriate.”

Other guiding principals for assessment include:

1. The instrument should be age appropriate. There should be a three-year range from the age you are targeting. It is best to catch a full range of abilities.

2. Assess a range of indicators.

3. Use multiple sources and types of information.

4. Use naturalistic or authentic methods. Children do not “perform” well for strange people, in strange settings, etc.

5. The instrument should be reliable and valid.

6. Have a realistic data-collection process. Can teachers efficiently capture the information required?

7. The assessment should be appropriately sensitive to cultural issues (racial, economic, home life, etc.).

8. Consider appropriate ability to accommodate language differences.

9. Have appropriate ability to accommodate differing abilities.

What is readiness assessment?

Our definition: Readiness assessment is information needed to promote children’s success. It is more about the learning environment being ready for children than about the child being “ready.” Finally, it should be well grounded in its purpose.

Researchers spoke briefly on identified purposes, including good practices, challenges, and uses of assessment that appropriately include children from special circumstances.

Dina Castro, Frank Porter Graham Child Development Center
Considerations for English Language Learners (ELL)

Accountability Policy Challenges

• There are variables among states on policies to include or exclude ELL in large-scale assessments.

• States differ on type and use of accommodations.

• If ELL children have low assessment scores, there is no way to know whether it is due to limited language skills or due to low content knowledge or both.

• ELL test results may be inappropriately used (i.e., to sort or retain students).

• Once we have labeled a child, it is difficult to get them out of the system—especially with special education.

Methodological challenges

• Lack of appropriate norms—tests standardized in groups of middle class, English-speaking children

• Use of verbal standardized tests that are not valid and reliable for ELLs—these tests can be culturally biased, and translated versions often do not account for dialect variations.

• Lack of use of nonverbal qualitative assessment methods—we rely on scores, and systems are set up so that decisions are made based on scores.

• Language skills are assessed in English only. The child’s language skills in the home language are not assessed.

Accommodation strategies
Modifications of the test include: Assessment in native language, text changes in vocabulary, provide additional example items/tasks, and reducing language complexity of test directions.

Modifications of the test procedure include: extra time, oral directions in native language, administration in several sessions, small-group administration, and directions read aloud or explained in English.

Accommodation issues

• Validity—Is the construct of the assessment altered? (You are out of the standardized assessment and must use the results carefully.)

• Effectiveness

• Differential impact—We need information on what works for what children. One size does not fit all.

• Feasibility—How much will it cost? Do we have the resources for this?

Research tells us

1. Language proficiency strongly relates to test performance.

2. Translating test items does not significantly improve performance when the language of instruction is not the child’s native language.

3. In addition to language proficiency, other factors influence ELL performance, such as the length of time in the U.S. (exposure as well such as classroom time), overall grades, and child mobility.

4. Many accommodations require additional administrative time that may increase costs.

5. Collect child’s background information, including time in the U.S., etc.

Carolyn Weiner, Syndactics

Children in Poverty  
Children in poverty are the children that traditionally perform poorly on standardized assessments.

Carolyn Weiner opened her discussion by directing participants to their Institute packets. Eager to Learn: Educating our Preschoolers, page 254, lists “six fundamental development skills that lay the foundation for all learning and underlie all advanced thinking, problem solving, and coping.” They are listed below.

1. The dual ability to take an interest in the sights, sounds, and sensations of the world and to calm oneself down
2. The ability to engage in relationships with other people
3. The ability to engage in two-way communication
4. The ability to create complex gestures, to string together a series of actions into an elaborate and deliberate problem-solving sequence
5. The ability to create ideas
6. The ability to build bridges between ideas to make them reality-based and logical

Weiner stated that assessment should get as close to these six skills as possible. Moreover, children have learned many skills that should be credited. Therefore, assessment should look at a range of skills while looking at the domains separately.

Weiner also cited the Child Trends Research Brief, “School Readiness: Helping Communities Get Children Ready for Schools and Schools Ready for Children” (October 2001), as a good resource.

Health is an issue for children in poverty. They often suffer from chronic problems such as ear infections and poor dental health. According to the chapter from Eager to Learn (page 256), a study of public school kindergarten teachers indicating the importance of certain factors to readiness lists good health as an essential factor to school readiness.

Factors listed as "essential" by teachers are printed below.

• Physically healthy, rested, nourished; enthusiastic and curious in approaching new activities; and communicates needs, wants, and thoughts verbally in child’s primary language (>75% of teachers)
• Takes turns and shares; not disruptive of class; sensitive to other children’s feelings; and can follow directions (51-75%)
• Finishes tasks; knows the English language; and sits still and pays attention (26-50%)
• Counts to 20; has good problem-solving skills; can use pencil or paintbrush; and knows letters of the alphabet (<26%)

Weiner pointed out that often what is assessed is not what teachers claim is important. Assessment should include information teachers say children need.

Assessments should have nonverbal, problem-solving items with some that do not require a verbal response. As some of the earlier speakers pointed out, not all children will verbally respond in a strange situation. Furthermore, English Language Learners have skills that are difficult to show an evaluator in English. Assessments should credit the nature of young children; they have their own set of behaviors. They are children, and they need intrinsic motivation to complete a task. Therefore, look for items that delight children.

Evaluators must be concrete in their requests. In one example, an evaluator asked the child to count to ten. She could see the child was thinking and moving his head, but he was not saying anything. Finally, the evaluator thought to say, “Count to ten—aloud.”

Assessment should be timely. There should be enough time to do something on the child’s behalf. If assessment is tied into available community resources, such as the dentist or family practitioners, it is easier to connect the child to services. Including parents in the data information is important; it provides comprehensive information.

Finally, know your purpose for the assessment, and have a clear idea of where data is going. Do you have to collect this information? How will it be utilized? The better assessment design, the better it will be.

Judy Niemeyer, University of North Carolina Greensboro, Department of Specialized Education
Children with Disabilities

Issues and challenges for children with disabilities
Many assessments do not show what children with disabilities can do. We often look at it from a deficient approach. Instead, look at strengths. Parents know what their children cannot do, and it is so nice to hear what they can do. It is a more positive experience for them.

Niemeyer feels that the major question is, “What is the major purpose for the assessment?” For example, identifying a disability is different than building a child’s program.

Families are often left out of assessment, but they are with the child the most and are a valuable resource. Parents know their children’s potential, history, abilities, and disabilities. Programs should see parents as partners, either by getting information ahead of time, in the assessment, in the interpretation process, or doing assessment (if possible).

Multiple assessment periods are beneficial. Children have short attention spans. Therefore, needs assessment should be conducted over several periods of time and in different contexts with several different materials.

Environment is important. Most assessments for children with potential disabilities are conducted in clinics. If the child is comfortable, the evaluations will more likely show her true potential. Niemeyer suggests the evaluation be conducted at home or a familiar childcare setting.

Assessment should be age appropriate—what is the real age (developmental, chronological, or both)? What is the type of disability—sight vs. hearing vs. physical? These factors affect assessment.

Is the assessment appropriate for children with disabilities? Are we grading them at a lower scale? Who does the assessment? Someone they have never seen before? Modifications and adaptations, such as the ones mentioned before by Dina Castro and Carolyn Weiner are very helpful.

Consider the following issues:

• What is the look of “screening” vs. assessment?

• How does data collection work for program evaluation?

• How should data be collect?

• What can be done with the data?

• What can be said about the data?

Niemeyer emphasized again to go back to purpose. Assessment is never done outside of a context, which is affected by the purpose, feasibility and resources, the audience and their knowledge of early childhood development, and how the collected data will be used.

Assessment is a program for the child. Do not look at numbers, but look at the items and how the child answered to develop a classroom plan. Also, look at how the assessment was conducted and for what purpose. Finally, the instrument should be reliable and valid.

Assessment for Screening, Dr. Dina Castro

When Dina Castro spoke at the SERVEing Young Children Assessment Institute in October, she emphasized the individuality of children. “Special needs children can fall into more than one category,” she said. “They may live in poverty, be English language learners, and have a disability condition. So we need to look at each child carefully.” Screening is one way to do this. Dr. Castro identified the main purpose of screening as identifying children at risk of school failure, but she said that screening can also have other uses, including

• Determining eligibility for Title I
• Identifying gifted children
• Identifying physical conditions that may interfere with learning
• Identifying needed services

There is a danger in using screening results for other purposes, such as diagnosing disabilities or physical conditions.

Although these are all valid uses for screening, most programs will screen for those services their program can provide. For example, a program that provides interventions for speech and hearing disabilities would screen for speech and hearing problems. They might not screen for or recognize other problem areas, such as vision problems, or poor nutrition, or needs in other areas. This is one reason it is desirable for agencies to come together to do a cooperative screening and to share information.

A question that concerns many educators who are involved with screening is whether to include children with special circumstances in screening efforts. According to Dr. Castro, all children should be included, but it is important to make provisions for children with special circumstances. The Individuals with Disabilities Education Act (IDEA) requires the following:

• No single procedure alone may be used to determine placement.
• Testing and administration procedures must not to be racially and culturally discriminatory.
• Tests must be administered in the child’s primary language.
• Correspondence to parents must be in their primary language.

Although it is not always possible to meet these conditions fully, it is important to meet them to the extent possible. The process used for screening and the instruments used are particularly important. If the provisions for special population children are not made, it is possible to get inaccurate results from screenings that can result in inappropriate placements. Dr. Castro cited, as an example, a child who was four when he came to this country. He spoke no English, but he had to be enrolled in kindergarten. He was screened in English; his mother was excluded from the screening. As a result of this screening, he was put into a remedial pullout program. Four months later, when he had learned enough English to understand what was asked of him, it was decided that he did not need to be in a remedial program. The child eventually was placed in a gifted education program.

Dr. Castro discussed the importance of determining the child’s proficiency in both her native language and English before deciding on the assessment procedures and the language to be used in the assessment. The child’s language proficiency should also be taken into account when determining if she needs remediation or help. If programs wait until a child is proficient in English to include them in assessments and thus, to provide help, it could result in missed opportunities.

Screening is often a superficial process. Several important factors in the screening process should be taken into account before a decision is made about assessment.

• What is the child’s level of language proficiency in her native language and English?
• What is the parents’ level of language proficiency in her native language and English?
• Who will do the screening? School psychologists are good resources for local assessments. If an interpreter is used, it is helpful to have one with knowledge of subject matter as well as language.
• What kind of English does the child/parent speak? Conversational English often comes before formal English that is needed for academic settings. In some circumstances, this can be misread as a disability.

Screeners must look at these as well as other factors (e.g., demographic characteristics of the child and family, preschool experience) in assessing a child and interpreting assessment results.

When asked to identify essential issues in designing a screening system, Dr. Castro felt there were two important steps that should be addressed:

1. Are there program methodologies in place within the district/state that are inclusive of all children? An example would be a one-on-one assessment for assessing a child with potential disabilities.
2. Assuming there is at least one assessment tool identified by the district/state, there should be a process of cross-culturally validating the instrument. Be aware of limitations of the instrument. If a district/state directly translates an instrument, be sure the items are linguistically parallel/equivalent. She gave the example of a monosyllabic word in English, such as “kite”, and compared it with its Spanish translation “papalote”. Both have the same meaning but are not phonologically equivalent.

Dr. Castro considered it would be ideal to have sufficient time for one-on-one assessment, observations, and parent discussions.

Screening instruments must be chosen carefully to accommodate special populations. Forms of assessment other than tests must be included in the screening process to form a picture of the whole child. Children should be observed in natural environments (home and/or school settings). Screeners should gather information from parents. Assessment should be done one-on-one. Cultural and language factors should be taken into account. As Dr. Castro stated, this is a controversial field in which professionals and researchers are still seeking answers.

Assessment for Purpose of Curriculum Planning, Dr. Carolyn Weiner

Dr. Carolyn Weiner, with Syndactics, emphasized a different aspect of assessment. She told the audience that assessment should be part of the curriculum. Through assessment, educators focus on what they want to know about each child. Assessment enables teachers to collect information and plan curriculum to begin where the child is and move forward. For this to be effective, assessment must be dynamic and cyclical, moving from assessment into curriculum and back over a time period. Dr. Weiner stated, “Assessment and curriculum are two faces of a single educational philosophy.”

According to Dr. Weiner, teachers must have their curriculum in mind, based on what they want children to learn, before they look at assessment. Children go through two phases of learning up to kindergarten age. One is developmental, and the other is academic. The developmental stage is very important to children’s overall learning. Most children complete this stage before kindergarten, but children from special populations often have not. Statistically, 80-90% of special population children are still in this stage when they begin kindergarten. Sixty percent of children from impoverished backgrounds are still in this stage, and 100% of LEP children are still in this stage as it relates to their English proficiency.

Dr. Weiner identified the process of evaluation as it fits into curriculum as important. “Assessment for curriculum planning is what you choose to make it, after you have your curriculum in place.” Once the process is in place, the product will follow.

Catherine Scott-Little summed up the discussions by reminding listeners that broad principles related to what is appropriate for children are the starting point for assessment design, and that having a goal in mind from the beginning is essential. Educators who are involved with assessing children should keep these principles in mind and seek as much input as possible.

Establishing and Implementing Assessment Systems that Include Children  
from Special Circumstances: Experiences from the Field

NOTES

Presiding:       Richard Clifford, Frank Porter Graham Child Development Center
Ohio:              Dawn Denno, Ohio Department of Education
Missouri:        Ruth Flynn, Missouri Department of Elementary and Secondary Education
California:       Jan Brown, University of California Los Angeles

With increasing pressure for accountability and the increasing tendency to relate funding decisions to children’s performance on assessments, we sometimes find that assessment systems that have been put into place do not reflect what we know is best practice in the field. Often there are unintended consequences associated with these systems. Persons from three states that have done extensive work to develop an assessment system—California, Missouri, and Ohio—presented on their state’s system and issues they have dealt with as they designed and implemented the system.

California
California’s preschool program is serving approximately 417,000 children. Many of the programs are co-located with Head Start programs, and approximately half of the children enrolled are minorities. The Desired Results program was initiated five years ago. The emphasis in the initiative is on improving the quality of the programs. The initiative couples environmental ratings (using the ECRS) with child assessments and a self-study process to provide data needed to improve the quality of early childhood programs. The initiative has four goals for children and two for families.

The Desired Results system established a continuum of development, with a set of indicators and measures for birth through 14 years. Measures were developed for each age level. One challenge of the system has been to help teachers learn how to improve their observation skills. Many of the teachers were not accustomed to formally observing children, and a good deal of professional development has been provided to help teachers develop the observation skills they need to implement the Desired Results measures. Often teachers turn the tool into a checklist and try to check off each of the skills. They may miss opportunities to observe what children are doing because they are so focused on the checklist of skills they are supposed to observe. The professional development for the teachers has focused on helping them develop antidotal observations.

Another challenge has been the tendency for teachers to use the assessment as a basis for the curriculum, or teach to the assessment. The goal of professional development sessions they have provided is to help teachers learn how to implement quality programming for children and use observations to record examples of the skills included on the Desired Results measures.

A special project—Desired Results Access—has been implemented to look at how the Desired Results assessments can be adapted for children with disabilities. The Desired Results Access staff have been highly involved with the development of the instrument and have been very helpful in the decision-making process. For example, the Desired Results Access project staff have pointed out the need to construct a measure that can capture even very small changes in children’s development in order to document progress for children with disabilities. Analysis of results from the assessment indicated that a number of items were highly correlated, and there was some discussion of eliminating some of the highly correlated items. Based on feedback from the Desired Results Access staff, the decision was made to keep all the items—children with disabilities would have more opportunities to demonstrate progress with more items.

They have also worked to appropriately incorporate children who are English language learners in the assessment system. The assessment materials have been translated into languages other than English. However, there are a number of issues that still need to be addressed. For instance, often the teacher who is trained to administer the instrument does not speak the language of the child. The Aid may speak the child’s language but is not trained to complete the assessment. They are continuing to address these and other issues related to Limited English Proficient children. This part of the planning process has a ways to go, but they have involved a variety of persons who are knowledgeable about language and assessment issues.

The system is also working through a number of technical issues related to the instrument. There has been considerable discussion around the question of whether the instrument should be normed. There are a number of potential down sides to norming the instrument and a number of unintended consequences that could result. Nonetheless, it does appear that California is planning to collect data and develop norms for the Desired Results instruments.

Another issue relates to the use of the data. Original plans for the project called for data from local programs to be collected and analyzed very soon. However, there have been a number of complications that caused decision makers to re-examine when they should begin to use the data. Providers have been slow to “catch on” to the process—the purpose, how to collect the data, and what to do with the data. Therefore, the state has decided to delay collecting data from local programs. They also will need to develop a data management system so that when data does start flowing from the local programs, they will have the capacity to analyze the information.

Training for teachers has been accomplished through a “train the trainer” model. Funds for training on the instruments were limited, so the state is training administrators who then train their staff to use the instrument. Additional training is being provided for special education personnel. The system has to work out details on how children with disabilities will be assessed—the special education teacher will have more training than the regular teacher, but the regular teacher spends more time with the child than the special education teacher.

Missouri
Missouri has a long history of assessing children and has built upon that history to develop the Show Me Results project. Children with disabilities have always been included in their assessments. They are just now beginning to see an increase in the number of Limited English Proficient children in the state and can tell that this increase will require some flexibility in terms of program requirements as well as the assessment system.

In 1978, Missouri developed the KIDS—Kindergarten Inventory of Developmental Skills—assessment. It was a screening designed to identify children with possible disabilities who need further evaluations. Beginning in 1984, every school district in the state was required to provide Parents As Teachers (PAT) for families. They are currently serving approximately 50% of the families in the state. With this program came additional requirements for child assessments. Project Construct began in 1986. In 1992, the Missouri Department of Elementary and Secondary Education conducted a Kindergarten Readiness Survey. The survey contained questions for kindergarten teachers about the children in their class. The teachers checked only children who were deemed “not ready” in a particular area of development or skill. The Department summarized the results but did not do a lot with the data.

The Governor pulled together a Task Force in 1996 to look at what characteristics were related to children’s success in school. This group used a grant from the Kauffman Foundation to assemble an interagency team of Education, Special Education, and Health specialists to develop goals and objectives for the project. Based on this work, an assessment system was developed. The development process has lasted three years. Teachers are asked to complete an observational assessment to see what children know. Teachers received one day of training on the observational process during August and then had to complete the assessment in the first nine weeks of school. They wanted to collect the data after the children had had time to become comfortable with the classroom but not so far into the school year that the children’s skills were greatly impacted by the kindergarten experience. The data from the Teacher observations is complimented by data from a Parent Questionnaire. Teachers reported that it was a great deal of work to collect the data for the assessment, but that they were very prepared for the teacher-parent conferences that take place just after the first nine weeks of school.

Ohio
Ohio has learned that creating an assessment system is hard work and takes a long time. The state is spending $500 million on early childhood services, and there are great pressures to hold the programs accountable for how the funds are spent and for measurable outcomes. Despite this pressure for accountability, it is essential that the assessment system be designed slowly and carefully.

The process of developing an assessment system for Ohio began several years ago. When leaders first sensed that significant funding was going to become available for early childhood services, they began a planning process that involved all major stakeholders. They convened groups around the state to collect feedback on what stakeholders thought that early childhood programs should be accountable for. They tried to involve every possible stakeholder. One decision that was made was that the assessment system would be designed to hold programs accountable, not children. The data would not be used to keep children out of kindergarten—age is the state’s only criteria for kindergarten entry. The early childhood system builds on the state’s Head Start programs, expanding Head Start services to additional children. Therefore, all the children included in the program, and in the assessment system, are considered to be vulnerable or at-risk. This made the decision makers even more aware that they had to plan carefully and think about the possible consequences of collecting this data for the children involved in the programs.

The Galileo system is used to collect data on children. This is a computer-based system for collecting multiple types of data from multiple data sources. They are in their third year of implementation and have found that it took this long to work out implementation issues. They anticipate that it will take another three years to work through data aggregation and utilization issues. There are many issues that still have to be worked out so that the data will be usable to make decisions and changes in programs. An evaluation component was built into the assessment design effort to provide information on how the assessment was received in the field and issues that need to be addressed to improve the assessment system.

The assessment planners worked very hard to ensure that the assessment was aligned with the standards. They conducted numerous meetings, collected opinions on surveys, and collected feedback on a number of drafts before the assessment was finalized. Then they field-tested the instrument and conducted surveys again to collect feedback from users. They also involved about 45 people over the course of one and one-half years to decide how to incorporate children with disabilities in the assessment system. The end result was an assessment system that includes a continuum of skills and abilities from birth through age nine. The system was revised to lower the floor for children with disabilities and provide specific accommodation strategies. The teacher makes a decision about which level she will use to assess a particular child and works to accommodate the child.

The computerized system also is a tool to support efforts to include children with disabilities. Children with disabilities may not demonstrate progress on particular items. A pull-down option on the computer provides additional indicators to help the teacher gauge any indication of progress that the child may have made.

The instrument is designed to fulfill two purposes—to provide data that can be used to assess the impact that programs are making in children’s lives and to provide data that teachers can use to improve their curriculum activities. Trying to design an instrument that is useful for two purposes is particularly challenging. They have also made the system compatible with the Head Start requirements for assessment so that programs do not have multiple assessment systems.

Now that they have data from the pilot years, they are entering a phase of trying to decide how the data can be used. It is difficult to know what constitutes “adequate” progress during the year for a child, especially for children with disabilities. It is possible that the child’s IEP is a better gauge of progress than this instrument. The feedback they have received indicates that large aggregated data sets from assessment may be especially difficult to use for policy decisions. It is important to include children with disabilities but difficult to know how to use the data collected on these children. The question of what skills are expected of children at entry and exit are yet to be determined. Their first report to the legislature is due in January, so they are studying the data and trying to determine what can and cannot be concluded from the information they are collected. Recognizing the limitations of the data collected through this system, they have decided to include data from other sources, such as children’s hematocrit results, to provide others objective sources of data to compliment the teacher observations.

They have just begun to work toward including Limited English Proficient children appropriately.

Training teachers to use the system was particularly challenging. Because the system is computerized, training teachers to use the technology was the first challenge. A lot of training was required simply to get teachers to the point where they could navigate the data collection system. Training was also required to help teachers develop the observation skills needed to collect the data on children. Finally, they are working to train teachers on how to use the information they have collected. The main complaint they hear from teachers is that it takes too much time to collect the information. If they can help teachers begin to use the data in meaningful ways, then the concern about the amount of time it takes to collect the information may diminish.

Ohio seems to have learned several lessons from their assessment development process. First and foremost, it is essential to get input from a wide range of stakeholders. This is important in designing the system and then in interpreting the results of the data-collection process. “Never sit in a room by yourself and decide what data means—talk to stakeholders about what they think is going on to help you understand.”

They also have found it beneficial to train teachers on good principles of assessment and quality programming for young children rather than the assessment instrument itself. This decreases the likelihood that teachers will “teach to the test” and increases the likelihood of reaching their ultimate goal, which is to improve the quality of the programs. They also have a group of consultants in the field to provide “hands-on” assistance to teachers and to help teachers translate what they learn in training into their everyday practices with young children.

General Principles for Early Childhood Assessment

• Eligibility for programs should be separate from attempts to assess “readiness”—age should be the criteria for determining whether children start kindergarten
• Assessments should include a broad range of domains
• Children vary tremendously in their abilities and characteristics, and that is OK—it should be expected; it is probably not a good idea to have a single score upon which a child is deemed “ready” or “not ready”
• Do no harm—there are times when attempts to be accountable can have negative consequences
• Represent all children in the assessment system
• Assess both pieces of the school readiness puzzle—children and the schools where they will be enrolled; it is important to look at “ready for what?” The fit between the child and the school’s expectations is what is most important. Families and communities are also important elements of school readiness.
• Different purposes require different assessment strategies—using an assessment for a purpose other than that which it was designed carries certain risks. For example, using a screening instrument as a pre-post test may not give good data for comparison purposes, especially if an independent rater did the first rating and the teacher did the second rating and also if accountability judgments are going to be made based on the scores.

Best Thing and Biggest Challenge in Developing Assessment Systems
Each of the three states has seen positive consequences from the process of developing an assessment system. The primary benefits seem to be in stimulating discussions about what early childhood programs are supposed to do and what outcomes we expect. This has lead to serious examination of program quality issues and how programs can be improved. Other benefits include an increased understanding of early childhood assessment issues and hopefully improved assessment practices in the classroom.

The challenges include changing the mindset of traditional early childhood professionals who have avoided assessment and a more structured approach to teaching children academic-related subjects. Balancing the need for increased accountability with the goal of maintaining quality programs that reflect the best practices for young children is also difficult. Another challenge is how to help people understand and use the data they have collected. Finally, providing the resources—time, staff, and training—that are needed to collect, analyze, and properly disseminate the information is challenging.

Negative or Unintended Consequences
Assessment systems often can have negative or unintended consequences. California has worked hard to avoid teachers “teaching to the test.” They did not want teachers to set up artificial learning situations based on the assessment. This is the heart of the current debate about whether the Desired Results assessment should be normed. If the system is normed, there is a greater potential for misuse and teaching to the test.

Ohio saw potential negative consequences on several different levels. With children, there was great concern about “drill and kill” efforts to get children to learn exactly what is on the assessment instrument. They wrote rules for the use of the assessment data to decrease the likelihood of this happening. On the program level, they will ultimately begin to make comparisons across programs, but in many cases comparisons are not legitimate—programs have vast differences in quality and teacher qualifications, so it is difficult to use the data to make program comparisons. Within the broader field of early childhood, there is some concern that the success of the assessment will lead people to judge teachers based on the results. The instrument was not designed to be an evaluation tool for looking at individual teachers’ performance.

Strategies to Improve the Quality of Data Collection
In later grades, wide-scale assessments typically are independent of teachers. Students take standardized tests, and teachers are not directly involved in the process. With early childhood, we typically rely heavily on teachers to collect the data. All three states presenting today—Ohio, Missouri, and California—collect assessment data through teacher observations. Helping teachers collect data accurately and consistently is a challenge.

One important strategy to increase the likelihood that the data will be collected appropriately is to avoid having the data they are collecting be used to evaluate teacher performance. If teachers are being judged based on children’s performance on assessments and they are the data collectors, there is a natural tendency for them to have some bias in the way they assess children.

Providing quality training and manuals with explicit instructions is another strategy to improve the consistency of teacher-conducted assessments. However, this is tricky because typically the states have tried to avoid “prescriptions” for how to conduct the observations or training specifically on the assessment so that teachers are not as likely to “teach to the test.” Nonetheless, training and explicit instructions are key to improving the quality of teacher-conducted assessments.

Support from administrators is also important. Missouri has conducted a summit for superintendents to give them information on the assessment system and involve them in supporting teachers who are conducting the assessments. Missouri also requires that administrators must attend their Parents As Teachers training with their teachers so they are fully informed about the program. California is also providing training for administrators. Educating administrators on the assessment process is essential so that they understand the logistics of the assessment process and also understand that the results are not to be used to evaluate teachers.

Efforts to establish inter-rater reliability are an important part of improving the quality of assessment conducted by teachers. Ohio has conducted a concurrent study and found differences between independent observers and the teachers. The independent observers tended to rate children’s abilities higher than the teachers. One possible explanation for this is that the teachers do not have time to observe children as thoroughly as independent observers. Ohio has collected some data where two teachers in the same room rated the same children. In these cases, there was good agreement between the two observers.

None of the three states have specific requirements for how teachers document their observations. They encourage, but do not require, portfolio-based systems and good antidotal observations. It was suggested that we must do all we can to increase consistency in the observations but always keep in mind that the data is not totally reliable. It is probably “good enough” for policy decisions but is not perfect. It is important to take this into account when using the data.

Keynote Speaker – Mary Jean LeTendre

In her keynote address at the annual SERVE Forum, Mary Jean LeTendre, former director of federal programs for the U.S. Department of Education, maintained that too many of our nation’s neediest students continue to be left behind. “Poor children still are not getting the support they need,” she said, “Access is still inequitable.” She stated that disparate and unequal schooling of diverse groups has been a problem throughout most of U.S. history, and it continues to be a problem today. She affirmed the role of the federal government as an entity that must shape sound legislation and policies to promote the health, well-being, and learning of young children and their families.

LeTendre encouraged schools and districts that operate Title I schoolwide programs to think creatively about how they can combine federal funds with local and state funds to improve the quality of the entire school. She pointed out that Title I money does not have to be used solely for programs conducted on school grounds. Funds can be used to support transportation, as well as related programs, such as family literacy. She further noted the fact that many schools/programs do not place great emphasis on early childhood services. “Currently, only 2% of schools/ programs in Title I use the money to fund pre-k services,” she stated.

Title I Assessment
According to LeTendre, there is growing attention to the key component of Title I legislation regarding student assessment. Each state is required to implement an assessment system that will monitor progress toward achievement goals for accountability and will stimulate and support teacher instruction. While this requirement has heretofore been applied to children in grades three and above, it has not included children pre-k to grade two. Traditionally, paper/pencil assessments have been deemed inappropriate for children pre-k to grade two, and they have been assessed primarily through teacher judgment, interviews, and parent report. LeTendre suggests that this practice may change with the re-authorization of Elementary and Secondary Education Act (ESEA), and we may begin to see very young children included in this process.

LeTendre pointed out that children do not have to be assessed to determine eligibility if the child has been in Head Start, Even Start, or a state program serving at-risk, homeless or neglected children. Children enrolling in a pre-kindergarten program where the school is a schoolwide site also do not have to be assessed to determine eligibility.

Reauthorization of ESEA
LeTendre reported that Congress continues to debate the re-authorization of the ESEA. While both the House and Senate have passed versions of this bill, they continue to wrangle over compromise language in conference. Unresolved issues include assessment/measuring student progress, accountability standards, and flexibility in the use of funds.

What Can We Do?
Rhetorically asking, what can be done to effectively ensure that “no child is left behind”? LeTendre underscored the importance of education’s leaders. Today’s leaders must have four essential qualities:

• Unclouded vision

• The understanding that urgent doesn’t mean haste, but purposeful action

• Tenacity and persistence

• Knowledge confirmed by experience—wisdom

As a final note, LeTendre asserted that educational leaders must also have the strength to say, “What we are doing is just not good enough. We have an obligation to educate to the best of our ability.”

View a list of Selected Readings From the Presenters