Why Students and Teachers Don’t Reason Well

Abstract

Paul begins this essay by developing the notion that all human action presupposes the use of humanly created logical systems that model, abridge, and summarize the features of the world about us, and that abstract inferential systems, and the reasoning they make possible, are as natural to us as a species as swimming is to a dolphin or flying is to a bird. As Paul puts it, we are continually “making inferences within a system we have created — about what is going on in our lives.”

Unfortunately, according to Paul, to reason well we must do more than simply engage in it. We must become aware of that engagement and use our knowledge of the nature of that engagement to improve it. Paul compares the good reasoner to the good ballet dancer, the good chess and tennis players. All three must explicitly study the principles and practice the moves involved (with explicit standards of performance in mind).

Having suggested what good reasoning requires, Paul presents evidence to show that most students are not good at it. What is more, he presents evidence to suggest that most teachers are not good at it either — at least not at assessing it when students are called upon to use it in their work. One of the major reasons, combining with ignorance of what reasoning requires, is a systematic confusion between intelligent subjectivity (wit, articulateness, cleverness without substance), and reasoned objectivity (careful, disciplined, reasoning about an issue), between subjective opinion (however “bright”), and reasoned judgment (however mundane).

Paul documents this problem with an analysis of a major mistake in a California Department of Education statewide assessment of reasoned evaluation in writing. He follows up this documentation of a mistake on the part of testing experts with the same mistake made by teachers. He then briefly explicates a model for the analysis and assessment of reasoning (based on the logic of the question at issue) complete with a series of samples of student reasoning, all duly analyzed for the reader.

Paul concludes the paper with a brief argument to the effect that “the logical structures implicit in an educated person’s mind are highly systematized.” In contrast he argues:

“When the logical structures by which a mind figures out the world are confused, a jumble, a hodgepodge, a mere conglomeration, then that figuring out is radically defective.... Then the mind begins it knows not where, takes things for granted without analysis or questioning, leaps to conclusions without sufficient evidence... meanders without a consciousness of its point of view.... Then the mind wanders into its own prejudices and biases, its own egocentricity and socio-centricity. Then the mind is not able to discipline itself by a close analysis of the question at issue and ignores the demands that the logic of that question puts on it and us as rational, logic-creating, logic-using animals.”

The Ability to Reason: A Defining Feature of Humans

Your capacity to reason is at the heart of all disciplined thinking. It explains how we alone of all the creatures of the earth have been able to develop full-fledged academic disciplines: biology, physics, botany, zoology, chemistry, geography, history, psychology, sociology, etc. We can go beyond immediate, instinctive reactions to reflective, reasoned responses precisely because we are able to develop small-scale and large-scale systems in which to intellectually operate and act. These systems enable us to mentally manipulate our possible responses to situations — to formulate them explicitly, to hold them at intellectual arm’s length, to analyze and critique them, and to decide what their implications are for us. Let me explain.

We understand the various particulars of everyday life by constructing abstract models or systems that abridge and summarize their features. In simplest form, we call these models or systems ideas. For example, our abstract concept of a bird is a model or system for thinking about actual birds in order to make sense of their behavior — in contrast to the behavior, say, of cats, dogs, turtles, beetles, and people. As we construct these abstract systems or models, we are enabled to use the reasoning power of our minds to go beyond a bare unconceptualized noticing of things to the making of inward interpretations of them, and hence derivations from them. In short, our concepts provide our minds with systems in which to experience and think; our minds operate (reason) within them to invest the world we experience with meanings rich in implications and consequences. Much of this is done, of course, quite automatically and subconsciously.

I can reason to any number of conclusions as the result of my having one simple model for a thing. For example, if I recognize a creature to be a dog, I can quickly infer it will:

1. bark rather than meow or chirp
2. wag its tail when pleased
3. growl when irritated
4. be unable to fly
5. have no feathers
6. be unable to live under water
7. be carnivorous
8. need oxygen
9. have teeth
10. have paws rather than feet, etc.

This word (‘dog’) is part of a much larger logical map upon which our minds can move in virtue of our capacity to reason. As we act bodily in the world, we act intellectually in our minds. These intellectual moves guide our actions in the world. Without these maps and the capacity to locate particulars on them, we would either thrash about aimlessly or be paralyzed by the bewildering mystery of things and events before us. In every situation in our lives we “construct” a response that results from how we are modeling the situation in our minds.

Hence, put us in any situation and we start to give it meaning, to figure it out with the logical structures we have at our disposal. So quickly and automatically do we make inferences — as the result of the way we are modeling the situation in our minds — that we do not typically notice those inferences.

For example, we see dark clouds and infer rain. We hear the door slam and infer someone has arrived. We see a frowning face and infer the person is angry. Our friend is late and we infer she is being inconsiderate. We meet a tall boy and infer he is good at basketball, an Asian and infer he will be good at math. We read a book, and infer what the various sentences and paragraphs, indeed what the whole book, is saying. We listen to what people say, and make a continual series of inferences as to what they mean. As we write we make inferences as to what others will make of what we are writing. We make inferences as to the clarity of what we are saying, as to what needs further explanation, as to what needs exemplification or illustration. We could not do this without “logical structures” by means of which to draw our inferences.

Many of our inferences are justified and reasonable. But, of course, many are not. One of the most important critical thinking skills is the skill of noticing and reconstructing the inferences we make, so that the various ways in which we inferentially shape our experiences become more and more apparent to us. This skill, this sensitivity or ability, enables us to separate our experiences into analyzed parts. We learn to distinguish the raw data of our experience from our interpretations of those data (in other words, from the inferences we are making about them).

Eventually we realize that the inferences we make are heavily influenced by our point of view and the assumptions we have made. This puts us in the position of being able to broaden the scope of our outlook, to see situations from more than one point of view, to become more open-minded. This requires that we recognize our point of view as a “logical system” that guides our inferences, a system that we can exchange for another (an alternative point of view), depending on our assumptions.

Often, then, different people make different inferences because they bring to situations a different point of view. They see the data differently. Or, to put it another way, they have different assumptions about what they see. For example, if two people see a man lying in a gutter, one might infer, “There’s a drunken bum.” The other might infer, “There’s a man in need of help.” These inferences are based on different assumptions about the conditions under which people end up in gutters and these assumptions are connected to the point of view about people that each has formed. The first person assumes: “Only drunks are to be found in gutters.” The second person assumes: “People lying in the gutter are in need of help.” The first person may have developed the point of view that people are fundamentally responsible for what happens to them and ought to be able to take care of themselves. The second may have developed the point of view that the problems people have are often caused by forces and events beyond their control. The two are modeling the situation differently. They are using a different system for experiencing it.

In any case, if we want our students to become good reasoners, we must become concerned to help them begin to notice the inferences they are making, the assumptions they are basing those inferences on, and the point of view about the world they are taking — hence the systems in which they are thinking. To help our students do this, we need to give them clear examples of simple cases, and lots and lots of practice analyzing and reconstructing them. For example, we could display the above inferences in the following way:

Person One:
Situation: “A man is lying in the gutter.”
Assumption: “Only bums lie in gutters.”
Inference: “That man’s a bum.”

Person Two:
Situation: “A man is lying in the gutter.”
Assumption: “Anyone lying in the gutter is in need of help.”
Inference: “That man is in need of help.”

Our goal of sensitizing students to the inferences they make and to the assumptions that underlie their thinking enables them to begin to gain command over their thinking (the way they are using logical structures to model the world). Of course, it may seem odd to put any effort into making explicit such obvious examples. In the harder instances, however, the value of the explication becomes more evident. In any case, because all human thinking is inferential in nature, and all inferences are embedded in a system, we cannot gain command of our thinking unless we can recognize, one way or another, the inferences embedded in it and the assumptions that underlie it.

Consider the way in which we plan and think our way through everyday events. We think of ourselves as washing up, eating our breakfast, getting ready for work, arriving on time, sitting down at our desks, making plans for lunch, paying bills, engaging in small talk, etc. Another way to put this is to say that we are continually interpreting our actions, giving them meanings — making inferences within a system we have created — about what is going on in our lives.

And this is to say that we must choose among a variety of possible systems for thinking about things. Again, consider some simple cases. As I am sitting in my easy chair, am I “relaxing” or “wasting time”? Am I being “determined” or “stubborn”, or worse, “pig-headed”? Did I “join” the conversation or “butt in”? Is Jack “laughing with me” or “laughing at me”? Am I “helping him” or “being taken advantage of”? Every time I interpret my actions within one of these systems that each word in the language represents, every time I give them a meaning, I make one or more inferences on the basis of one or more assumptions within some point of view.

As humans we continually make assumptions about ourselves, our jobs, our mates, our children, about the world in general. We take some things for granted, simply because we can’t always be questioning everything. Sometimes we take the wrong things for granted. For example, I run off to the store (assuming that I have enough money with me) and arrive to find that I have left my money at home. I assume that I have enough gas in the car only to find that I have run out. I assume that an item marked down in price is a good buy only to find that it was “marked up” before it was “marked down”. I assume that it will not, or that it will, rain. I assume that my car will start when I turn the key and press the starter. I assume that I mean well in my dealings with others. We make hundreds of assumptions, use hundreds of concepts, make hundreds of inferences, without noticing that we are doing so. Most of them are quite sound and justifiable. Some however are not.

The question then becomes: “How can we teach our students to begin to recognize the inferences they are making, the assumptions they are basing those inferences on, and the point of view, the perspective on the world that they are beginning to form?” That is, “How can we help students to recognize how they are reasoning about the world?”

Our Students Are Not Learning to Reason Well

Though we are “logic-creating” and “logic-using” animals, we typically operate with little awareness of this fact. We create and apply logical systems without knowing that we are doing so. Our intellectual modeling of the world is done sub rosa, without mindfulness. It is small wonder, then, that we often reason poorly.
Imagine a ballet dancer improving her ballet without knowing that she is a dancer or how and when she is dancing. Imagine a chess player who does not know she is playing chess. Or a tennis player who does not know she is playing tennis. We can hardly imagine people developing these physical and intellectual abilities without high consciousness of how and what they are doing in the doing of it. Yet we expect students to develop the ability to reason well without any mindfulness of the nature of reasoning, the elements of reasoning, or the criteria for assessing reasoning. We expect students to become good reasoners, in other words, without any knowledge of the logic of reasoning. Not surprisingly our approach doesn’t work. Most students are very poor reasoners.

WHAT DOES RESEARCH ON LEARNING AND TEACHING TELL US?

By any measure whatsoever, most students are not learning to reason well. A recent summary of research by Mary Kennedy regarding student learning and instruction at the K–12 level documents serious reasoning deficiencies on the part of students. (See figure next page.)

California State-Wide Test Fiasco: Teachers and Testers Who Don’t Understand Reasoning

Before teachers will be able to help students to reason well, it is essential that they learn what reasoning is and how to assess it. A recent statewide test in California demonstrated that many teachers, and even some educational testing experts, have serious misunderstandings about the nature of reasoning and how to assess it.

The student essay below (figure 2) should have been graded at the lower rather than the higher end of the continuum of eight levels: “minimal evidence of achievement” or, at best, “limited evidence of achievement” rather than the highest grade of “exceptional achievement”. For though the essay may have “flair and sparkle” (as one teacher expressed it), it is a poor example of evaluative reasoning, since it systematically confuses the objective goal of reasoned evaluation with the very different goal of explaining subjective preference, an important distinction in critical thinking which the teacher-evaluators apparently missed entirely.
First of all, the instructions themselves are confused. They begin with a clear requirement of “objective” evaluation:

“Students were asked to write an evaluative essay, make judgments about the worth of a book, television program, or type of music and then support their judgments with reasons and evidence. Students must consider possible criteria on which to base an evaluation, analyze their subject in the light of the criteria, and select evidence that clearly supports their judgments.”

Unfortunately, this request for reasoned evaluation is blended in the second half of the instruction with what might possibly be taken, with a little stretching and selective reading, as a request for the expression of a “subjective” preference:
Each student was assigned one of the following evaluative tasks: to write a letter to a favorite author telling why they especially liked one of the author’s books, to explain why they enjoyed one television program more than any others, or to justify their preference for a particular type of music. The tasks made clear that students must argue convincingly for their preferences and not just offer unsupported opinions.
Let’s look closely at this confusion. In the first place, there is still an emphasis on objective evaluation (“The tasks made clear that students must argue convincingly for their preferences and not just offer unsupported opinions”) while the task itself is defined as the justification of a “preference”.

Now most people prefer books, television programs, and types of music for fundamentally subjective, not objective, reasons. They like a particular book, television program, or song for no reason other than that they like it, that is, because they enjoy it or find pleasure in it or are interested or absorbed or excited or amused by it. Their reasons for liking what they like are not the result of an objective evaluation. They have no relation to the objective quality of what is judged. They are about the personal responses of the experiencer, not about the objective qualities of that which is experienced.

Most people, to take the point a step further, do not have “evidence” — other than the stuff of their subjective reactions — to justify their preferences. They prefer because of the way they feel not because of the way they reason. To choose because of these subjective states of feeling is precisely to lack criteria of evaluation or evidence that bears upon objective assessment. When challenged to support subjective preferences, people usually can do little more than repeat their subjective reactions (“I find it boring, amusing, exciting, dull, interesting, etc.”) or rationalize them (“I find it exciting because it has a lot of action in it.”)

A reasoned evaluation of a book, a program, or a type of music requires more than this; it requires some knowledge of the qualities of what we are evaluating and of the criteria appropriate to the evaluation of those qualities. One needs to be well informed about books, about programs, about music if one is to claim to be in a position to objectively evaluate them. If one is not well informed, one is unable to render a justified evaluative judgment, though one can always subjectively react and freely express one’s subjective reactions as (mere) personal preferences. This is what the student (graded as having written an objective evaluation of “exceptional achievement”) actually does. But his evaluators, not having this distinction clear in their own minds, completely miss the difference.

The sample student essay can, for analytic purposes, be divided into three parts. We shall comment briefly on each in turn. The first segment of the essay is an account of a highly emotional exchange between the student and his mother:
“Well, you’re getting to the age when you have to learn to be responsible!” my mother yelled out. “Yes, but I can’t be available all the time to do my appointed chores! I’m only thirteen! I want to be with my friends, to have fun! I don’t think that it is fair for me to baby-sit while you run your little errands!” I snapped back. I sprinted upstairs to my room before my mother could start another sentence.
It is clear that in this segment there is no analysis, no setting out of alternative criteria, no clarification of the question at issue, no hint at reasoning or reasoned evaluation.

In the second part, the student makes a sweeping claim about a purported causal relationship between listening to rock music and his asserted, but unsupported, ability to control his emotions. He does not consider “possible criteria on which to base an evaluation”. He does not present any evidence, though he does cite two examples, one where a song prompts him to punch his pillow and one where another song prompts him to stop. This gives little credence to the notion that rock music leads to his “controlling” his emotions. If anything, his examples seem to imply that, rather than learning control from, he is learning to be controlled by, the music he listens to. His major claim that “Without this music, I might have turned out to be a violent and grumpy person” is without reasoned or evidentiary support. He merely brashly asserts that it is true:

I turned on my radio and “Shout” was playing. I noted how true the song was and I threw some punches at my pillow. The song ended and “Control”, by Janet Jackson came on. I stopped beating my pillow. I suddenly felt at peace with myself. The song had slowed me down. I pondered briefly over all the songs that had helped me to control my feelings. The list was endless. So is my devotion to rock music and pop rock. These songs help me to express my feelings, they make me wind down, and above all they make me feel good. Without this music, I might have turned out to be a violent and grumpy person.

In the third, and final, section of the essay the student closes his remarks with a series of subjective, unsupported, even irrelevant statements:

Some of my favorite songs are by Howard Jones, Pet Shop Boys, and Madonna. I especially like songs that have a message in them, such as “Stand by Me”, by Ben E. King. This song tells me to stand by the people I love and to not question them in time of need. Basically this song is telling me to believe in my friends, because they are my friends.

My favorite type of music is rock and pop rock. Without them, there is no way that I could survive mentally. They are with me in times of trouble, and best of all, they are only a step away.

If this is reasoning, it is very bad reasoning: “Believe in your friends because they are your friends”, “If you feel you cannot survive without rock music, then it follows that you can’t.” Of course, a more appropriate interpretation of what is going on is that the student is not reasoning at all but merely asserting his subjective opinions.

Consider, the student doesn’t examine alternative criteria on which to base an evaluation of music. He doesn’t analyze rock music in the light of evaluative criteria. He doesn’t provide evidence that clearly supports his judgment. His writing is vague where it needs to be precise, logically rambling where it needs to be critically reasoned. We don’t really know what he means by songs “controlling” his feelings.

We are not provided with any evidence on the basis of which we could assess whether there is any truth in his sweeping claims about himself, for example, that he could not survive mentally without rock music. Indeed, common sense experience strongly suggests, we believe, that the student is simply deluding himself on this point, or, alternatively, engaging in unbridled hyperbole.

When a blatantly weak essay such as this is disseminated nationally as an example of “exceptional achievement” in the writing of a reasoned evaluative essay, then it is clear that there are large numbers of educators who are not clear about the assessment of reasoning. Remember, the California Assessment Program of the California State Department of Education is the second largest assessment unit in the country. (I should add that Dale Carlson, the head of CAP, is now putting a major effort into rectifying this problem.)

THE MANY WAYS TEACHERS MIS-ASSESS REASONING

If many teachers take bad reasoning to be good, do they also take good reasoning to be bad? Unfortunately the answer appears to be, “Yes.” This became apparent in a Center for Critical Thinking research project in which teachers were provided with a well-reasoned response to the California prompt, in addition to the poorly reasoned one. The participants were teachers enrolled in critical thinking workshops. They were given the two essays to assess after receiving a morning’s instruction on critical thinking. What is significant is the myriad of confusions and misunderstandings about the assessment of reasoning that emerged and the inconsistencies in both grading and in justifying grades.

Here is the “well-reasoned response” they were asked to assess alongside the poorly reasoned “Rock Around the Clock”.

This second essay (next page) was written by one of the research staff members of the Center who made sure that it was responsive to the directions and displayed all of the critical thinking abilities called for:

1. it distinguished mere subjective preference from well-reasoned assessment,
2. it was responsive to the logic of the question at issue,
3. it formulated and discussed alternative relevant criteria,
4. it distinguished having evidence relevant to a question from lacking such evidence,
5. it displayed intellectual humility,
6. it displayed intellectual integrity,
7. it drew only those conclusions the evidence warranted.

The results highlighted the problem. On one occasion 81 teachers and administrators assessed the two essays. The poorly reasoned essay was given an average score of 5.4 (out of 8) while the well-reasoned essay was given an average score of 3.9. Forty-nine of the teachers gave the poorly-reasoned essay a 6, 7, or 8, while only 18 teachers gave the well-reasoned essay a 6, 7, or 8.

Even more illuminating than the raw scores were the reasons given by the teachers and administrators. Multiple confusions surfaced, as I suggested above, about the nature of reasoning and the appropriate way to assess it. Let’s look at some of the responses. Try to imagine students actually receiving these grades along with the often mistaken, confused, or unintelligible commentary.

I have divided teacher assessments for convenience into two groups. The first consists of those teachers who grade the poorly reasoned essay higher than the well-reasoned essay. The second consists of those teachers who grade the poorly reasoned essay lower than the well-reasoned essay. Reading the teachers’ justifications for their grades reveals a great deal of misunderstanding of the nature of reasoning. [First Essay: “Rock Around the Clock” (the poorly reasoned essay) Second Essay: “Can I Prove Rock Music is Better?” (the well-reasoned essay)]

FIRST GROUP OF TEACHERS

1. The following teachers give a high grade to the poorly reasoned essay and a low grade to the well-reasoned essay. In virtually every case, the teachers reveal no awareness of the importance of intellectual humility, wherein one does not claim to justify a conclusion when one lacks the evidence to do so; instead, one gives good reasons for suspending judgment.
2. 1) A Physical Education Teacher: [#1] “The first essay better fulfills the criteria for the assignment because the writer justifies (his or her) preference for a particular type of music. I think I would give it a 7 though because it was kind of confusing how the writer got on the subject.
   [#2] “The second essay did not justify a preference for any particular type of music. So the writer did not meet the criteria for the assignment. Strangely enough it was easier to read but possibly because the way the writer feels is how I feel about music in general. I think the essay deserves a ‘0’.”
3. An English Teacher: [#1] “I would give this essay a 7 because he/she gave experience from his/her life to support their opinion — gave reasons and evidence by example.
   [#2] “I would give this essay a grade of 2 because he/she did not prove a point — merely rambled from one thing to another searching for a reason.”
4. A Math Teacher: [#1] “I would give the first essay a 5 because it did not support the judgment well but did make many references.
   [#2] “I would give the second essay a 3 because it is not very evaluative! It did analyze the subject but provided no real support of any judgment.”
5. A Math Teacher: [#1] “I would give this paper a grade of 7 because criteria were evident, analysis was good and it had lots of supporting evidence.
   [#2] “I would give this paper a 3 because criteria are given but nothing was analyzed and no supporting evidence.”
6. Freshman Studies Teacher: [#1] “I would give ‘Rock Around the Clock’ a grade of 6 because: a) a more flowing style of writing than a series of loosely related points, b) a personal approach, c) specific information as to records and effects of the songs, d) valid and accurate comparisons, e) personalization, f) availability, g) a well-supported point of view, and h) R&R as an avoidance tool.
   [#2] “I would give ‘Can I Prove Rock Music is Better?’ a 3 because
   a) statement of problem OK, b) no exploration about ‘Why we like it’,
   c) discusses what it is about, not why we listen. Do we listen to the words or music?, d) the idea of ‘better performances’ not followed through on, and e) How do they know they are like ‘most people’?”
7. A Math Teacher: [#1] “The first essay: grade 6. The writer has set up some criteria for his choice, the music gives him a calming influence.... Since the writer is given the opportunity to set his own criteria, this will suffice. He gives examples to justify his conclusions.
   [#2] “The second essay: grade 3. An attempt is made to give reasons for supporting the music but no conclusions are made. The writer cannot make an argument for his case in any area. It is difficult, as the writer has said, to justify choice or preference, but since one can choose one’s own criteria it would seem any position well-argued and justified would fulfill the assignment. The author did not succeed in doing that.”
8. Subject Taught Not Identified: [#1] “‘Rock Around the Clock’ Score: 6. This student does not give any clear criteria to start off as to possible criteria to base their evaluation on. This student based their evaluation on how it made them feel or respond. It was based on reactions — not facts to choose music by, but at least this student used something to justify their preference.
   [#2] “‘Can I Prove Rock Music Is Better?’ Score: 2 Too vague — never really makes a decision about their preference of music. This student talks about possible criteria but never really says anything about it. Shows no support to justify the preference.”
9. Former English Teacher: [#1] “I would give this essay a grade of 8 because: a) essay cites specific examples, b) catchy opening, c) the criteria used was based on student’s personal experience, d) student was asked to justify their preference. I think she did.
   [#2] “I would give this essay a grade of 2 because: a) very generalized,
   b) few, if any, concrete examples, c) essay is not personalized to any extent, d) no specific conclusions drawn.”
10. Special Ed. Teacher: [#1] “Point total: 7. This essay listed three criteria on which to base a judgment. It gave examples of each — maybe better examples could be found. The writer attempted to analyze a basically subjective issue in concrete terms — what the songs do for them: not objective, but a fairly concrete assessment of music’s subjectivity.
    [#2] “Point total: 0. This essay did not seriously attempt to answer the issue at hand. Instead it concluded, quite lamely, that no objective statement of worth could be made. While this may be accurate in the broadest sense, no effort was made to justify that position.”
11. English Teacher: [#1] “I would give this essay a 7 because the author is not afraid to take a stand. Although the ‘proof’ is emotionally based, that was the direction of his/her argument.
    [#2] “I would give this essay a 3 because the writer was not able to take a position. He/she beats around the bush and asks the reader to make the decision when that was the assignment to the writer. The insecurity and negative attitude runs through the entire paper.”

SECOND GROUP OF TEACHERS

The following teachers give a low grade to the poorly reasoned essay and a high or higher grade to the well-reasoned essay. In some cases the teachers revealed some awareness of the importance of intellectual humility. Some are, however, confused or mistaken in part about reasoning and its assessment. For most, thankfully, this confusion is conjoined with some insight into reasoning. For some few others, the fact that they graded the poorly-reasoned essay lower is not based on insight but chance. This is apparent from some of the reasons they give.

1. A Library-Media Teacher: [#1] “Grade: 3 or 4. Reasons: My first thought that it wasn’t a typical essay but rather starts out with a rather clever, attention-getting device. In that sense, the student did catch my attention — and also confused me somewhat. That is, it doesn’t start out as a typical essay. The student is a good writer in that their word choices make sense and there are supporting reasons for why they chose rock music and pop music.... Now that I read this again, I can see that really the writer has only supplied one reason for their selection: the control/expression of feelings. Well, it’s the same old problem in grading a paper, i.e., the student writes well but hasn’t followed the criteria strictly.
   [#2] “Grade: 7. Reasons: Just a first critical response before I re-read it. It strikes me as thoughtful and honest (which always impresses me). Now I’ll see how it fits the criteria. The writer states he needs good reasons for his judgment. I don’t think that ‘good’ is the word he wants.... Why do we like what we like? That’s a provocative question!... A quickie, yes, I think they’ve fulfilled most of the criteria, just not in the usual fashion. Also, it’s an essay (as I define one).”
2. A Special Ed. Teacher: [#1] “The student in this essay never really makes a statement that involves an evaluation of a judgment made concerning a type of music, except to say ‘My favorite type of music is rock and pop rock. Without them there is no way I could survive mentally.’ He does try to show what he means by this statement when he offers examples of music that affect his mood. He lacks a clear evaluation or supportive evidence toward the topic. I think his statement about surviving mentally is a bit much. I give it a 4.
   [#2] “This student doesn’t know what he thinks and he lets you know it continually. His closing paragraph summarizes what he is trying to put down in the essay and it is the most straightforward part of the essay. His title doesn’t quite jibe with the rest of the essay. He was supposed to prove rock music is better, but what he really talked about was whether there was any justification for why people like rock music. I give it a 5.”
3. A Social Studies Teacher: [#1] “I would give essay one a grade of 6. Essay number one lists reasons for liking rock music, but it is very superficial in analyzing them in the light of the criteria. It really does not approach the subject in a way that logically lists possible criteria as a basis for analysis and then applies the criteria to the music. The essay is generally Bull Shit with only a general connection to the instructions.
   [#2] “I would give essay #2 an 8 because the possible criteria for analyzing the issue are covered....”
4. An English Teacher: [#1] “Score: 3. The writer in essay one has discussed how he/she feels about rock and pop music, but generalities are given and his/her statements aren’t supported with evidence. The assignment is to ‘justify’ preference, not discuss that it makes him/her ‘feel good’ period. No criteria have been established, so the essay just rambles on about ‘feelings’ and not much else. Reasons and evidence are lacking.
   [#2] “Score: 5. This essay does a little bit better in attempting an argument. The essay establishes two ‘criteria’ on which to base his/her essay.... Examples of ‘answers’ in paragraph 3 are needed as evidence.... Paragraph 4 isn’t developed. Needs reasons and evidence/ examples. Weak Conclusion.”
5. A Physical Education Teacher: [#1] “I would grade the essay 0. The essay does not show their judgment about worth with reason and evidence as asked in the directions. There are no criteria for evaluation, analysis with criteria or evidence that clearly supports the judgments.
   [#2] “I would grade the essay 5. The essay attempts to set up criteria for evaluation, yet not as completely as it could have been done. There was an attempt to analyze the subject with the criteria, but not complete. There was no evidence to clearly support the judgment.”
6. A Second Grade Teacher: [#1] “The first essay should have a 3 because the stated criterion is subjective. The conclusion comes down to, ‘I like it because I like it.’
   [#2] “The second essay would have a 6 because there was a search for good criteria and no evidence was found to support the good criteria.”
7. A Counselor: [#1] “I would give this essay a 1 because the student did select a topic to evaluate which fit the directions. However, she reported her subjective taste (how some songs have affected her, which songs she likes) rather than evaluating ‘rock music’.
   [#2] “I would give this essay a 7 because: a) she selects an appropriate topic, 2) she considered what criteria would be appropriate to evaluate rock music, c) she made judgments based on the criteria she listed, 4) her conclusion was based on her criteria/judgment. However, she might have considered/used other criteria.”
8. A Sixth Grade Language Arts Teacher: [#1] “A grade of 1. There was no evaluation, went strictly by senses.
   [#2] “A grade of 8. The writer did a good job on a subject that is a matter of preference no matter how you look at it! He tried to objectively judge rock music, but in the end... ‘We like it just because we like it.’”
9. A First Grade Teacher: [#1] “I would give ‘Rock Around the Clock’ a 4 because the writer did give some facts for liking rock music but wrote mostly from emotion without questioning if her facts were sound. For example, ‘believe in my friends because they are my friends’.
   [#2] “I would give ‘Can I Prove Rock Music is Better?’ a 7. The writer stated the purpose, criteria, facts, and gave a conclusion. The writer considered more than just feeling. More facts for liking rock music are needed.”
   

Introduction to the Analysis and Evaluation of Reasoning

There are two obstacles that stand in the way of fostering sound reasoning K–12: 1) teachers must learn how to devise assignments that require reasoning, and 2) teachers must learn how to analyze and evaluate reasoning objectively. This process will not happen overnight, but the sooner it begins, the sooner it can be achieved.
We will shortly take a look at three assignments that call for reasoning as well as at three examples of student work for each of those assignments: student work with no reasoning in it, student work with poor reasoning in it, and student work with good reasoning in it. In each case, we will provide a brief commentary to help make clear what one should look for in the reasoning. But first we will provide a brief overview of what is involved, in general, in the analysis and evaluation of reasoning.

WHAT IS INVOLVED IN ANALYZING AND EVALUATING REASONING?

The fundamental criterion to use in analyzing and evaluating reasoning comes from an analysis of the purpose of the reasoner and the logic of the question or questions raised. For example, if a person raises the question, say, as to whether democracy is failing in the USA (in the light of the dwindling number of people who vote and the growing power of vested interest groups with significant money to expend on campaign contributions), we can establish general criteria for assessing the reasoning by spelling out what in general one would have to do to settle the question. Those criteria would include such matters as the following:

1. An Analysis of the Concept of the Ends of Democracy. What would it be for democracy to succeed? What would it be for it to fail? What do we take the fundamental objective of democracy to be? For democracy to succeed is it enough that it simply ensures the right of the people at large to vote or must it also serve the well being of the people as well?
2. Collection of the Facts About the Numbers of People Not Voting. What is the actual number of people not voting? Is it growing? By what percentage?
3. An Interpretation of the Significance of the Facts Collected in #2. What are the reasons why growing numbers of people are not voting? What are the implications of those facts?
4. Collection of Facts About the Number of Vested Interest Groups Influencing Elections. How many vested interested groups are influencing elections today in comparison to the past? What is the nature and extent of their influence in money spent?
5. An Interpretation of the Significance of the Facts Collected in #4. What is the significance of the growing influence of vested interest groups on election outcomes? What is gained and lost by means of that influence?
6. Synthesis of Numbers 1 through 5. What is the overall significance of what we have found out in 1 through 5? What does it all add up to? What exactly are we gaining and losing as a result of the growing influence of vested interest groups and diminished numbers of voters? In attempting to put everything together we would want to see reflection on this issue from more than one point of view. We would want to assess how the reasoner responds to reasonable objections from other points of view.

These are some of the considerations relevant to reasoning well about the issue. A rational analysis of someone’s response to this issue would involve, then, checking to see if the above considerations were reasonably addressed, to see if the reasoner had done a plausible job in analyzing the functions of democracy, collecting relevant facts and information, interpreting those facts, and putting everything together, with a sensitivity to more than one point of view, into one coherent line of reasoning.

Many of the teachers assessing the reasoning of the essays on rock music above failed to analyze or review the logic of the question at issue. Instead they read the essays impressionistically, allowing the grade they gave to be determined more by whether their impressions were positive or negative than by any close analysis of the degree to which the student responded adequately to the demands inherent in the precise question at issue.

It is the logic of the question at issue that is the “system for thinking” that should guide our reasoning. If we do not develop skill in explicating that logic, our reasoning is apt to become impressionistic, guided by our prejudices and biases, by our egocentrism and ethnocentrism, rather than disciplined by rational considerations.

Three Examples of Student Reasoning

What follows below are three assignments designed to call for reasoning on the part of the students, along with three examples of student “reasoning” in response to those assignments. Two of the assignments are in history and the other in literature. The three issues the students are asked to develop their reasoning on involve: reasoning about the character of the American people, reasoning about the meaning of a poem, and reasoning about the comparative importance of inventions. It would be useful if you thought a little about your own assessment of the students’ reasoning before you looked at ours’. You could then compare the two.

AMERICAN HISTORY: REASONING ABOUT THE AMERICAN CHARACTER

Question at Issue: “Are the Americans you know capable of the kind of mass hysteria which occurred in 1919 and is described in a textbook as the ‘Red Scare’?”

Directions: One of the most important reasons to write our history is to discover who we are and who we are not, how we can develop ourselves, what faults we have to watch out for, and what strengths we can build upon. Read the passage in your textbook on the “Red Scare”. Then write a couple of paragraphs in which you try to figure out whether the Americans you know are “capable” or “not capable” of reacting as many Americans did in 1919. (See textbook, p. 731.) Be sure you show us your reasoning. Support and explain why you think as you do.

Student #1
The people I know are not like the people who lived in 1919. They obey the law and, though they might make some mistakes or do some things they ought not to, they would never hurt someone who was innocent. Most of the people I know go to church and believe in God. They are good Christians. They read the Bible. They try to raise their children to be good and avoid evil. They are kind people. So I don’t believe that what happened in 1919 could ever happen again. It won’t happen in my neighborhood.

Commentary on the Student’s Reasoning
There is very little reasoning in this student’s work and, on the whole, what there is seems uncritical and self-serving: in essence, “My friends are good. Therefore they wouldn’t do anything bad.” There are obvious objections to this reasoning. Presumably, most of the people in 1919 also went to church and believed in God. Presumably, they too would have thought themselves to be good Christians. Presumably, their friends thought of them as kind and as trying to raise their children to be good and to avoid evil. As a result, the student has not really responded to the logic of the question which implicitly requires that we think about mass hysteria, how it occurs, and how it influences otherwise morally sensitive people to behave in a morally insensitive way.

Student #2
Certainly there are always people who go overboard. That is human nature. And it is unreasonable to think that we will ever abandon human nature. The American people rightly recognized the threat that communism posed to our way of life and fought against it. After all, if we had defeated it then we would not have to have fought the Cold War and spent so much money and resources to defeat the communists after WW II. So what is the lesson? Watch out for human nature. Don’t go overboard. But on the other hand, don’t forget who your enemies are and don’t give up the fight against them just because some people punish them too severely or go to an extreme.

Commentary on the Student’s Reasoning
There is more reasoning in this student’s work, but still not very good reasoning: in essence, “It is human nature for some people to lose control. So (by implication) some of us might do so, but whether or not some of us might act as some people in 1919 did, the people in 1919 were right to fight against communists”. This reasoning is weak because it largely ignores the issue raised. The question at issue is not whether it was right for the people in 1919 to oppose communism, such as it was, in the USA at the time. The question is rather how it came to pass that, as we expressed above, otherwise morally sensitive people came to behave in a morally insensitive way. The student didn’t take this question seriously.

Student #3
It is hard to answer the question as to what anyone is capable of. Perhaps what we are capable of is largely a result of the circumstances we are under. If we assume that all humans share human nature and that because of human nature we are capable of acting out of intense fear or insecurity or hate, then a lot depends upon whether something or someone is able to stir those things up in us. Perhaps, of course, there is a way to raise people so that they have so much good character that even when someone tries to stir up the “worst” in them, they do not give in, they resist the temptation to let their worst side take control of them. The question could then be asked whether I and my friends and neighbors are in the first or the second group. Since we have never been “tested” in a crisis situation, since we have never felt deeply threatened, I don’t think I can honesty say we would pass the test.

I don’t know whether we would act like a “Charles Evans Hughes” or a “Billy Sunday”. It’s a scary thought.

Commentary on the Student’s Reasoning
This is better reasoning than in either of the two passages above: in essence, “Everyone has a worse and a better side. Everyone’s worse side can be appealed to. Whether you have the “character” to withstand an appeal to your worse cannot be known until you are “tested”. My friends and I have not been tested. Therefore, we cannot know whether we have the character to withstand such an appeal. Therefore, we don’t know whether we would or would not act as many did in 1919.”

ENGLISH: INTERPRETING POEMS

Question At Issue: What is John Donne saying in his poem “Death Be Not Proud”?
Directions: Carefully read the poem below, trying to figure out what the poet is saying. Be careful to explain what your interpretation is and what exactly it is based on. Show us your reasoning. Make sure your interpretation is consistent with (all of) what the poem says.

Death Be Not Proud
(John Donne 1572–1631)

Death be not proud, though some have called thee
Mighty and dreadful, for, thou art not soe,
For, those, whom thou think’st, thou dost overthrow,
Die not, poore death, nor yet canst thou kill mee.
Much pleasure, then from thee, much more must flow,
And soonest our best men with thee doe goe,
Rest of their bones, and soules deliverie.
Thou art slave to Fate, Chance, kings, and desperate men,
And dost with poyson, warre, and sicknesse dwell,
And poppie, or charmes can make us sleepe as well,
And better then thy stroake; why swell’st thou then?
One short sleepe past, wee wake eternally,
And death shall be no more; death, thou shalt die.

Student #1
I don’t like this poem. It is boring and confusing. The guy does not spell correctly. He talks a lot about death but he does not say anything. I don’t see why he thinks death is mighty or why he thinks it can’t kill him. He says a lot of confusing things. At one time he says it gives pleasure and then talks about bones resting, which makes no sense. Then he talks about flowers and sleeping. Finally he says that death shall be no more and that it shall die. I don’t get it. Why doesn’t he just say what he wants to say? This is a terrible poem. Why do we have to read such stupid stuff?

Commentary on the Student’s Reasoning
This student provides us with virtually no reasoning at all. Rather than attempt to figure out what the poet is saying by closely reading what is said, the student rejects the poem, dismisses it emotionally. The result is that the student flagrantly mis-reads the poem and blames his mis-reading on the poem itself and the poet. The student needs to be introduced to the concept of critical reading in which the reader uses the text as evidence to use in interpreting the meaning.

Student #2
Mr. Donne says that death should not be proud. It is not mighty or dreadful. He says this because death is like sleep and when you go to sleep you rest. Therefore, because it is restful even the best people sleep, even slaves. And sleeping is better than being poisoned or being sick. Finally, he says that we only sleep a while and then we awake. And then death is gone. In fact, it is dead. He thinks this is good.

Commentary on the Student’s Reasoning
There is more reasoning in this student’s work but most of it ignores the evidence of what the poem says. The poem does not say or imply, for example, that “because it [death] is restful even the best people sleep, even slaves”. The poem does not say or imply that “sleeping is better than being poisoned or being sick”. Finally, it is clear that the student is not getting the major point of the poem, namely, that because of the promised resurrection, last judgment, and eternal life in heaven or hell, there is a sense in which “death” is not real and lasting, but only something that will “die”. Like the first student, this student also needs to be introduced to the concept of critical reading in which the reader uses the text as evidence in interpreting meaning.

Student #3
It is clear that Donne believes in God or at least in an afterlife. This is implied in the first four lines which I interpret as saying something like this: “Don’t think you’re so powerful because no one really dies but only appears to die” (People who “die” are really just awaiting their resurrection). This interpretation is supported in the next line which implies that what we call death is really a kind of “sleepe” and is not, therefore, very bad. In fact, as he says sleep often gives us “pleasure”. The next lines make a different kind of point but still are a criticism of the view that death is “mighty” and “dreadful”. Death, he says, is not able to control “Fate, Chance, kings, and desperate men”. Furthermore, not only is it not able to control these other forces, it can’t even get away from such unpleasant associates as “poyson, warre, and sicknesse”. Finally, he reasons, narcotics makes us sleep as well as death does and when everyone is resurrected for final judgment (which I infer is what he means) then death itself will be gone forever, and therefore “shalt die”.

Commentary on the Student’s Reasoning
Finally, we have a student who illustrates the process of critical reading, carefully reasoning her way through the poem, using the words of the poem to carefully back up her interpretation.

HISTORY: REASONING ABOUT THE SIGNIFICANCE OF INVENTIONS

Question at Issue: “Of two inventions discussed in your textbook, which was the most important and why?”

Directions: The textbook for the course describes a number of important inventions, including those of Gutenberg, Edison, and George Washington Carver. Take two inventions, either from those mentioned in the book or some other inventions you know of, and compare their importance. Defend your answer by giving reasons in favor of your judgment.

Student #1
An invention that is very important is the printing press. Johann Gutenberg, who was a man that lived in Germany, invented it. He invented the printing press in the Fifteenth Century. The first book ever printed by Gutenberg was the Bible. But he soon printed many other books as well. The first printing press worked by using movable type.

Another important invention mentioned in the textbook was the dehydration of foods. This was invented by George Washington Carver. When you dehydrate foods you take the water out of them. George Washington Carver wanted many people to use his inventions, so he did not take out any patents on them. He made many other inventions besides dehydration. He even thought of more than 300 uses for the peanut, including facial cream, shoe polish, and ice cream.

Both inventions are very important. Many people read books that are printed on a printing press. Many people eat food that has been dehydrated. But to me the printing press was more important than dehydration.

Commentary on the Student’s Reasoning
The student does not provide any reasoning to support his conclusion. He discusses no criteria for assessing inventions for their importance, nor any evidence to support one or the other with respect to those criteria. Most of the factual detail is irrelevant to the issue.

Student #2
R-r-r-r-ring.

The first sound I hear in the morning is my alarm clock going off. It’s an invention I truly hate.

R-r-r-r-ring.It is not a pretty sound, and as soon as I hear it I feel myself getting angry. If only I didn’t have to get up so early! All my muscles cry out that I want to sleep! Most mornings when I hear that sound, I even cover my ears with my pillow in the hope that I won’t hear it going off.

It is an old-fashioned wind-up alarm clock that loses ten minutes a day. It is not a digital alarm clock because all the digital alarm clocks I’ve ever tried have alarms that are too soft to awaken a really sound sleeper. And believe me I am a very sound sleeper.

R-r-r-r-ring. But no matter what I do, or how I feel, I end up wide-awake and out of bed and getting dressed for school.

Once I am awake I look at my other clock, the one that is hanging on the wall over my dresser. It is a great invention too. It’s a digital clock that keeps perfect time. It has a red LED display and it glows in the dark. It has an emergency battery backup, so that even if the electricity cuts out in the night, my wall clock never loses a second.

Which of the two inventions is more important? That’s the question I ask myself as I head off for school. And then the answer comes to me. No matter how perfectly the digital wall clock keeps time, without the alarm clock I wouldn’t be awake to see it. So without doubt the alarm clock wins the prize as most important.

Commentary on the Student’s Reasoning
The student provides some reasoning but when considered closely it is apparent that the reasoning is absurd. The notion that without the alarm clock people would never wake up is ridiculous. What does this student think happened before the alarm clock was invented? Furthermore, does she really think that loud alarms cannot be built into digital clocks? Once again, the student has not learned to think about the logic of the question at issue. Therefore, the student gives no time to reflecting on the general criteria by means of which we might assess the social worth of inventions by relating that worth to the most basic human values, like the preservation of life, the minimization of pain and suffering, the development of a more just society, and so forth. It is only in terms of the concepts of basic human values that criteria can be generated that give a solid logic to the question and hence a means to assess the reasoning which purports to settle the question.

Student #3
Two inventions mentioned in the book are television and the dehydration of food. Each is important in different ways. The television set, for example, affects many people’s lives. I watch television almost every night and so do all of my friends. But it’s not just me and my friends. The same is true for people all across the country, and in most foreign countries as well. Television allows more people to be entertained than was ever possible before. We witness world news, nature programs, comedies and many other programs. Television lets us see much of what is going on in the world.

Dehydration of foods is important in a very different way. The main effects of dehydration are that it allows food to be kept for a long time without spoiling, and to be shipped for a lower cost. I don’t know how many people in the world today use dehydrated foods, but I’m pretty sure that it’s far smaller than the number of people who enjoy TV. So that seems to show that TV is more important.
And yet I don’t feel right saying that one invention is more important than another simply because it has affected more people. If dehydration is used more than it is now, it could help cut down on the number of people who are starving in the world. Saving just a few people from dying of starvation is more important than taking a lot of people and entertaining them.

Commentary on the Student’s Reasoning

The student provides some reasoning, which might at first appear absurd, but on reflection makes good sense. This student is thinking about the logic of the question at issue and hence is reflecting on the general criteria by means of which we might assess the social worth of inventions by relating that worth to the most basic human values: like the quality or preservation of life, the minimization of pain and suffering, the development of a more just society, and so forth. To say that this student’s reasoning is better than the first two students — because she does respond to the logic of the question at issue — does not mean that her reasoning is perfect, for perhaps there are yet further considerations that might be mentioned about the effects of television which might persuade us that television itself is making so large a contribution to the quality or preservation of human life that it is indeed more important than food dehydration. We may know the basic logic of a question without knowing whether we yet have the best answer to that question, the answer that best fulfills its logic.

Conclusion

The whole of this book is concerned with the process of developing students who reason through what they are learning so as to grasp the logic of it, students who know clearly the difference between coming to terms with the logic of something and merely rotely memorizing it. But reasoning is not a matter to be learned once and for all. It is a matter of life-long learning, a matter of bringing insightful mindfulness into the fabric of our thinking and our action. For the teacher, it is a matter of learning how to design instruction so that students take command of the logic of their own thinking while they are thinking and through that insightful grasp, improve it.

We figure things out better if we can monitor what we are doing, intellectually, in trying to figure them out, so that we go beyond simply using logical structures, so that we go beyond simply making logical moves, so that we start to intentionally, deliberately, and willfully examine and take apart the logical structures we are using, so that we designedly, purposively, and alertly assess our use of the structures in everyday situations, and, of course, so that we do these things well: clearly, accurately, precisely, etc.

To understand logical structures is to integrate them, to establish logical connections between them, to make it possible for the mind to make an extended series of nuanced inferences, deductions, and derivations. “This is so, therefore that also is so, and that, and that.” The logical structures implicit in an educated person’s mind are highly systematized. The well-educated person is able to reason quite directly and deliberately, to begin somewhere, know where one is beginning, and then reason with awareness from that point to other points, all with a given question in mind, with specific evidence in mind, with specific reasons to advance, with specific conclusions to support, with consciousness of one’s point of view and of contrasting points of view. The good reasoner is always reasoning within a system that disciplines and restrains that reasoning.

When the logical structures by which a mind figures out the world are confused, a jumble, a hodgepodge, a mere conglomeration, then that figuring out is radically defective, typically in any of a variety of ways: incomplete, inaccurate, distorted, muddled, inexact, superficial, rigid, inconsistent, and unproductive. Then the mind begins it knows not where, takes things for granted without analysis or questioning, leaps to conclusions without sufficient evidence to back them up, meanders without a consciousness of its point of view or of alternative points of view. Then the mind wanders into its own prejudices and biases, its own egocentricity and socio-centricity. Then the mind is not able to discipline itself by a close analysis of the question at issue and ignores the demands that the logic of that question puts on it and us as rational, logic-creating, logic-using animals.

{Taken from Paul, R. (1993). Critical Thinking: What Every Student Needs to Survive in A Rapidly Changing World, Dillon Beach, CA: Foundation For Critical Thinking).

Go to top