There is, however, a specific stimulus for this post. In an effort to go paperless in my office, I’m re-reading old journal articles and deciding which ones I want to scan into my online notebook. To that end, I re-read a 1982 article in the Reading Research Quarterly (V.18 #1 p.23) in which the authors described their research on informal reading inventories and commented that using passages with different levels didn't seem to affect students' performance. In other words, as the grade level of the text went up or down based on the readability formulas used, students’ comprehension scores didn't go up or down with them.
The authors then went on to write: “Although readability formulae reflect word difficulty and sentence complexity, they fail to account for one's familiarity with a text Presumably, this failure to control for students' familiarity with reading material diminishes the validity of both commercial and curriculum-based inventories developed with readability formulae."
Despite the resurgence of readability formulas since that article was written (in the 1980s, they were roundly and repeatedly criticized and both the IRA and NCTE discouraged their use in the creation of written materials) I think the authors' suggestion that readability formulas are not fully adequate to the task of revealing how well students might or might not understand what they read is still timely. Actually, as many discussions of readability formulas and their history point out—See, for just one instance, the Plain Language Association website—the formulas were meant to measure ease of reading, not comprehension.
If you find that distinction confusing, you're not the only one. But after pondering it a bit, I think it means a passage coming in at a low grade level, based on a readability formula, could be easily read if you define "reading" as knowing and pronouncing the words. However, that ranking can't tell you whether or not readers can readily grasp the concepts or relationships expressed in the passage. That is, a passage given a low grade level by a readability formula is not necessarily easy to understand. By the same token, passages that earn high grade levels aren’t necessarily hard to read.
Readability formulas don't measure comprehension because they do not take into account key comprehension factors such as the syntactic complexity of the language, familiarity of the vocabulary, reader’s background knowledge, and the text’s conceptual difficulty. Instead they rely mainly on length of sentences and numbers of syllables with some formulas including elements like passive constructions and prepositions.
For instance, a readability formula would treat ennui and boredom as equals because both words have two syllables. Yet the truth is most students would immediately know the meaning of boredom and be dumbfounded by the word ennui. Similarly, inconceivable is shorter than unimaginable, but that certainly doesn't make it easier for student readers to interpret.
Readability formulas also rely heavily on length of sentences to identify ease of reading. Sentence length, though, doesn't tell the whole story. As the web usability guru Jakob Nielsen points out in his discussion of writing for the web, these two sentences are the exact same length but conceptually, they are far from equal:
He waved his hands.
He waived his rights.
As Nielsen correctly says, "everybody understands what the first sentence describes,[however] you might need a law degree to fully comprehend the implications of the second sentence."
Given what I see as the limitations of readability formulas, it's always disconcerting to be asked what readability formula I use to write my textbooks because, in all honesty, I have to say “none.” I use the Flesch Kincaid readability feature of Word strictly as a predictor of potential difficulty. If a passage comes out higher than I think it should be for the book’s audience, I check it for syntactical and linguistic features, known to cause problems, i.e. distance of pronouns from references, embedded clauses, passive constructions, etc. (The Purdue Online Writing Lab has a list of five principles for readability that I find invaluable, available as a PDF or PPT series). And if I really want to cover a topic that consistently comes out with a high, grade level, for instance, passages on the brain with all those pesky references to the multisyllabic word hemispheres, I will classroom test to see how students do with the passages in question.
While I could wax even longer on how readability formulas should be used with extreme caution, I’ll end with a quotation from a study put out by the University of Illinois at Urbana, available on the web at the Eric Clearing House and titled “Conceptual and Empirical Bases of Readability Formulas”:
Problems arise when difficult words and long sentences are treated as the direct cause of difficulty in comprehension and are used in readability formulas to predict the readers' comprehension. Readability formulas are not the most appropriate measure and cannot reliably predict how well individual readers will comprehend particular texts. Far more important are text and reader properties which formulas cannot measure. Neither can any formula be a reliable guide for editing a text to reduce its difficulty.
They study was published in 1986, but to my mind, the sentiments are not the slightest bit dated.