printable version of page Printer-friendly page

Focus On Basics

Volume 1, Issue A :: February 1997

Understanding Quantitative Research about Adult Literacy

by Thomas Valentine
Adult educators make hundreds of small decisions every hour they are in the classroom - decisions about what to say, how to spend time, what materials to use. When they are new to the job, educators find these decisions difficult, but, through trial and error, they build up personal, experience-based knowledge about what works and what doesn't work. Gradually, they begin to recognize cause and effect relationships between what they do and how students respond. They develop and continually refine personal theories of how education works, and they use those theories to guide their decision-making when they approach a novel situation or a new adult learner.

Quantitative researchers engage in much the same task, but in a far more formal way. They attempt to identify and describe patterns of behavior that are clear enough and regular enough to guide educational action. Researchers try to clarify the seeming chaos of activity that surrounds educators by discovering patterns that naturally occur, and they trust that educators will be able to use this information to improve practice.

Although many working adult educators find quantitative research too esoteric to be understood fully, its apparent complexity is offset by the clear and highly patterned logic on which it is based. Educators need surprisingly little knowledge to get the gist of the articles they read. In this article, I will provide working educators with a few basic tools that will strengthen their ability to make sense of quantitative research. Instead of presenting the type of detailed, technical information that appears in statistics books, I'll attempt to provide information that will enable working educators to critically evaluate the quality and logic of quantitative studies that might have a bearing on how they do their jobs.

The Three Most Common Purposes of Quantitative Research on Adult Literacy

Most quantitative research studies on adult literacy attempt to accomplish one of three broad purposes: description, theory testing, and theory generating. I'll deal with them one by one.

Although quantitative studies can have markedly different purposes, they all use the same basic "tools." The following sections will explore the basic concepts and common analyses necessary for an understanding of quantitative research.

Basic Underlying Concepts

A basic tenet of perceptual psychology is that human perception is based on variation. If you were to look at a pure white wall that had no texture or irregularities, you would see nothing at all. If, in that wall, there was even a tiny crack, your eyes would be drawn immediately to it. In making sense of what you were looking at, your mind would automatically create a concept called "crackedness." You could then talk about any section of the wall in terms of its crackedness, with some sections having crackedness and some not.

All research builds on variations, and in statistical research, it is called variance. Things that vary, like crackedness in the above example, are called variables. Variance is the concept that underpins all statistical research.

The variance contained in variables can be described statistically in many different ways, some of which are quite familiar. Frequencies, expressed as numbers or percentages, are readily understood, because they amount to a simple tallying of the values of a variable. When a group of students is described as 55 percent women and 45 percent male, the variable is gender, the values are women and men, and the frequencies are the numbers themselves.

Means, or averages, are another common statistical expression that everyone more or less understands. The mean of a group of scores (or comparable measures) is commonly used as a way of talking about the group with a single number. However, the mean by itself can be a rather poor description of a group, particularly when the scores, taken together, do not arrange themselves into a predictable pattern. Consequently, you will rarely encounter a mean in statistical reports that is not accompanied by a standard deviation. The standard deviation indicates how spread out the scores are for that group, and it is a direct indication of variance.

Variance itself is rarely the primary focus of statistical research. Most statistical research focuses, instead, on some form of covariation, on whether two or more variables systematically vary together. For example, one would expect hours spent in instruction and learning progress to co-vary quite well, while height and learning progress would not meaningfully co-vary.

Although they appear very different on the surface, most of the statistical tests commonly encountered in adult education research reports represent attempts to establish covariation among variables. In all of these common statistical tests, if the numbers suggest that there is in fact a relationship that can't be attributed to chance, the researcher will conclude that the co-variation is statistically significant. Statistical significance indicates that there is a relationship between variables, but it doesn't necessarily mean that the relationship is strong enough to be important to working educators. Once significance is established, readers must use their non-statistical judgment to decide whether that relationship is strong enough to be considered substantively meaningful. For example, a statistically significant but weak relationship between years of schooling and learner motivation might be considered unimportant for program planning.

The statistical test actually used in any given study depends, to a great extent, on the types of variables being used. There are two distinct types of variables commonly used in statistical research about adult literacy. The first type of variable is called a categorical variable. Categorical variables vary in type or nature, but not in degree; they can't be rank ordered in any meaningful way. Gender and race are common categorical variables. The second common type of variable is called a continuous variable. Continuous variables vary in degrees, and can be expressed as a numerical scale. Test scores, satisfaction, and income all are continuous variables.

The final important concept underpinning statistical research is sampling. In most cases, researchers are attempting to identify patterns of behavior, cognition, or attitudes that apply to large numbers of people, but they only have access to a much smaller number. This small number of people is called a sample, and the sample is supposed to be a representative subsection of the larger group, or population. When the findings based on a sample are applied to a population, it is called statistical inference or generalization, and there are strict rules that allow researchers to generalize with confidence. Most of these rules require that the sample be randomly drawn from the population of interest.

Unfortunately, adult education researchers find it nearly impossible to follow the rules of pure statistics. Drawing true random samples from the population of interest is usually prohibitively expensive, so researchers often rely on convenience samples. In conducting experimental research, researchers quickly find that adult learners are not malleable enough to be randomly assigned to various "treatment" conditions, so researchers attempt to "match" treatment groups on selected variables. Despite these patchwork remedies, more often than not the compromises are severe enough to preclude any legitimate statistical inference whatsoever.

However, it's possible to glean useful information even from studies using highly compromised statistical procedures. By carefully studying the sample used in a study, educators can determine the extent to which that sample lines up with the people with whom they work. If the findings are clear enough, the sample reasonably large, and the characteristics of the sample similar to the people in their educational setting, educators can use logical inference to predict the probable implications of the findings for their own work. Consequently, work done with a nonrandom sample in Boston might have very real implications for educators working in Baltimore but none at all for educators working in rural South Dakota.

Common Statistical Procedures

In preparing this article, I looked over the articles that were recently published in journals of interest to adult literacy educators and found that surprisingly few statistical procedures were used with any frequency. In fact, if readers can understand the logic and statistics of eight basic procedures, they can understand the methodology of more than 90 percent of the quantitative pieces they encounter. I'll attempt to give a quick conceptual overview of these eight procedures, and will briefly discuss the actual meaning of the more important statistics they employ.


Closing Comments

Statistical research is not as formidable as it appears, but it requires a special type of reasoning. Statistical reasoning involves a tight, detailed, and codified logic that can be especially difficult for people who would rather deal in broad strokes and big ideas than with the making of fine distinctions about extremely well focused concepts.

Some people view statistics with a sense of moral indignation at the fact that statistics reduces things of human importance to numbers, and they relate statistics to the power that statistics could give to a "big brother" type of government or to a scorn of bean-counting bureaucrats. In reality, of course, statistical research reduces an object of study no more than a camera reduces the object of a photograph. Statistical reasoning simply represents a highly patterned and highly public way of looking at the world, and, because its details can be readily scrutinized and evaluated, it is often preferred by funding agencies and program evaluators over more subjective and less public ways of reasoning. Like all research methods, it can be used for good or bad purposes.

Statistics are a part of the everyday life of adult educators. We use them to report attendance, to evaluate our programs, and to learn about the demographic trends in the broader society that affect our work. It is in everyone's best interest that working educators learn how to be critical consumers of quantitative research. Even the best quantitative research on adult education is ultimately meaningless unless teachers and administrators put the findings to work.


When Reading Quantitative Research

Ask Yourself:

Updated 7/27/07 :: Copyright © 2005 NCSALL