In this section we offer some basic advice to help instructional planners and decision-makers judge the quality of experimental research, the type of study most useful in determining the impact of programs. It was written to help you judge the quality of research studies being cited as evidence of effectiveness for products and practices you consider for your schools. It is a very simple description to help you understand the importance of quality and the characteristics of studies that indicate quality.

Since the topic is inherently technical, the advice may prove most helpful if used to guide your work with qualified research advisors from a university, state department of education, or regional laboratory. The advice is grouped into several major topics, and consists of a series of high-level questions that should be answered satisfactorily for each study you consider.

Quality of conception

• What is the theoretical framework?

  Generally, good research is grounded in a theory. Having a more general sense of the meaning of research, helps to guide the development of practice and implementation.

• What are the research questions?

  These questions guide all the details of a study—the methods, settings, research design, measurements, data analysis, and conclusions.

• How clearly are the research questions articulated?

  Since the questions guide all the details of the study, it is important that they be clearly written and explained.

• Can the research questions be investigated using quantitative measures?

  The questions must be asked in a way that allows the researcher to measure the outcomes.

• How well did the research investigate and answer these questions?

The research base

• Is there a research base for the intervention under consideration?

  We often assume that the advice of experts is based on the research that they and others have conducted. Very often, though, no data are involved; the authors are simply giving their opinions and positions, or citing other experts, in a kind of endless loop, giving the appearance that there is a substantial body of empirical findings.

• What is the quality of the studies making up the research base?

  Even though many studies may have investigated interventions like the one that interests you, the quality of the evidence is still open to review and questioning. Poorly designed studies can result in unjustified claims of effectiveness that might not stand up to a more rigorous research method.

• How appropriate is the design?

The design of a study refers to built-in comparisons among intervention conditions. In experimental and quasi-experimental studies design refers to comparison of outcomes (e.g. achievement scores) of experimental and one or more control groups. Without the appropriate design a study cannot answer causal questions.

• Are intervention conditions clearly defined and documented?

Completeness of description is important for several reasons. (1) A detailed understanding of an intervention (as well as control conditions!), can help you to form your own judgment about the meaning of research findings. (2) It can tell you whether the intervention is a viable, practical option for the schools you are concerned with. (3) In order for other researchers to replicate a study a detailed description is required.

Next>>

Top of Page