Fooling Students into Not Fooling Themselves

Raymond Hall

Four activities designed to engage students in the methods of science by showing how personal experience is not always to be trusted. 

            Carl Sagan has argued with some success that “pseudoscience is embraced in exact proportion as real science is misunderstood1. The current situation in the US is that a significant percentage of our fellow citizens believe in many topics either unsupported or even refuted by current evidence. A 2001 Gallup poll2 indicated that a third of Americans profess a belief in astrology (up from 27% in 1990), that 28% believe it is possible to converse with the dead (up from 18%), and fully half believe in extra sensory perception.

            As a science educator, I feel that enabling students to understand science involves an emphasis on how to discern scientific claims from that of the many pseudoscientific claims that abound in our media. What's behind the popularity of these claims? Purported reasons for why astrology, siances, and ESP are so widely believed are covered in a number of recent books3, and seem as numerous as the unsupported beliefs themselves. In my research and reading of these books I have come to implicate a common factor among those who hold to such claims; the inability to distinguish reliable evidence. 

I teach a general education course in critical thinking entitled Science & NonsenseThe course involves a study of the nature of the scientific enterprise, and how science, and the knowledge obtained from science, affects our lives and shapes our understanding of the world. I also seek to develop students' critical thinking skills through the study of past and current controversial topics that involve science or claim to be supported by science. The aim of the curriculum then is to enable students to tell the difference between reliable evidence and hearsay, reason and delusion, science and nonsense! 

Science as a Safeguard

 What follows is a set of classroom activities and topics designed to actively demonstrate to students how much of what they commonly take as evidence is unmistakably unreliable. The ability to distinguish reliable evidence first involves a better understand of ourselves, and the many ways we are prone to misinterpret our perceptions. With these activities I try to convince students that, in the words of Richard Feynman4, “A first principle [in science] is that you must not fool yourself—and you are the easiest person to fool!”

            Of course it is not a simple matter getting folks to question what they think they know. I have found that students of critical thinking are initially uncomfortable with the use of reason, since most are defensive of their current beliefs, and often express surprise at the idea of being asked to support their beliefs rationally. One aspect of science that I initially stress is that scientists employ methods developed to mediate what I will call pitfalls of perception, the ways in which our common sense intuition fails us. Many of these pitfalls are documented in the books and articles of Thomas Gilovich5, a professor of psychology at Cornell University. Gilovich suggests that the amazing complexity of our cognition, of how the human mind takes in and processes information, makes it inevitable that there will be ways in which the system can subtly betray us. States Gilovich: “At one level, [that common sense is so wrought with pitfalls] should not come as a surprise: It is precisely because everyday judgment cannot be trusted that the inferential safeguards known as the scientific method were developed;”6 safeguards such as control samples, blind (and double blind) studies, and peer review. 

            I have found that exploring these pitfalls of perception is an effective way of engaging students to critically examine their beliefs. The following four activities describe why safeguards are needed and employed, and underscore how wrong things can go if one does not adequately guard against such pitfalls.

Subjective Validation and Astrology

            Most realize that it is easy to read more into a written passage than was intended by the writer, but feel that this doesn’t present much of a problem. I demonstrate that it can cause alarming misinterpretations with the following activity, most recently popularized by the famous magician and activist against pseudoscientific thinking, James Randi7.

            Checking up on one’s horoscope appears to be a strong American pastime with almost every major newspaper carrying the daily celestial assessments of sun sign astrology. A major claim of sun sign astrology is that one’s personality is largely determined by the position of the sun against the ecliptic constellations at one’s time of birth. Our first activity explores the most prevalent evidence for this claim —that it works!

Activity 1 — Subjective Validation in Reading Horoscopes

           First, share with the students that astrology has it roots with the ancient Babylonians, where a study of the nightly positions of the stars and setting sun may have played an important role in decisions of when to plant and harvest. Such a useful connection between human activities and the heavens were extended and further developed in ancient Greece to the level of divination of the personal lives of individuals.

            Also mention that there are more that 10,000 professional astrologers in North America alone (compared to the roughly 6000 professional astronomers), and that according to a recent Gallup poll belief in astrology among the US population is at 33%, up from 27% a decade ago. Can so many of our fellow citizens be wrong? Let’s see if there is anything to the claim of personality assessment according to sun sign. Below is a list of the signs of the zodiac, select your sign (by clicking on the name) and read the personality description to see just how well it describes you.



21-Mar to 20-Apr



21-Sep to 20-Oct



21-Apr to 20-May



21-Oct to 20-Nov



21-May to 20-Jul



21-Nov to 20-Dec



21-Jun to 20-Jul



21-Dec to 20-Jan



21-Jul to 20-Aug



21-Jan to 20-Feb



21-Aug to 20-Sep



21-Feb to 20-Mar

Now to be a bit quantitative, please assess how well this statement characterized your personality using a scale from 1 to 5, where 1 corresponds to a complete miss and 5 for amazingly dead on.

Finally for sake of comparison please select and read a description for any other sign on the above list.

Surprised? The majority of my class is always astonished. Ask the students to exchange charts. I guarantee many will gasp as they read them over, and a wave of laughter will sweep across the room. In the classroom setting students are asked to pick up a personality assessment labeled according to their astrological sign as they enter the classroom. With ease of identification in mind, the chart for each sign is printed on a different color paper and the sheet is folded to hide the contents.


           The psychologist Bertram Forer was the first to describe this phenomenon, which he labeled subjective validation, and utilize it in his class to demonstrate this inherent bias in assessment of claims about one's self8. We seem to always notice and count the hits and for the most part ignore the misses. Subjective validation, and the wrong impression it creates, has an important role in why people have such firm convictions about astrology, as well as the fantastic claims of palm reading, graphology, self help ideologies, personality inventories, and most paranormal means of personality revelation.  

           The pitfall of subjective validation seems to be inherent in how our mind works, and is just one of many known ways our brains can systematically mislead us. The next pitfall we will look at is how we are fundamentally handicapped when assessing probabilities and degrees of randomness.

Misinterpretation of random events and phone ESP

            You happened to be thinking about your mother; suddenly the phone rings. It’s your mother! Amazing! Come to think of it this has happened to me in the past. There must be some kind of uncanny ESP connection involved.

           Research in psychology has demonstrated that the mind has difficulty in correctly interpreting random patterns in time. We seek the unusual happenstance and mark them. The times when we think of our mom and she doesn’t call, and the reverse, where she calls and she didn’t happen to be on our mind at the moment, are by contrast non-events and don’t have the same impression on our memories. So after a while the false impression of ESP connection is created out of what is an inevitable overlap of common events. Equally troubling is our inability to comprehend short range statistics. 

            A widespread belief among basketball players, professional and blacktop, is that of the hot hand5. Michael Jordan and others have spoken on what they feel is the fact that once they have made a shot, they are more likely to make the next; their hand is hot. Conversely, the belief goes, if they miss a shot their hand has gone cold and they are more likely to miss the next. The more shots made (or missed) in a row the hotter (or colder) the hand. This to many a ballplayer’s mind describes why shots sometimes occur in streaks.

            The question here is understanding what a random distribution looks like. Consider the following flips of a coin (an independent process):


Both series are of course equally probable, but most people would say that the second is too orderly, with too many heads ('X's) in a row, and therefore less likely. This is in contradiction to the math! In 20 tosses one would expect 10% of time to get a six in a row somewhere in the sequence, 25% of the time a five in a row, and a 50-50 chance of having at least a four in a row. The next activity demonstrates this "streaky" nature of random sequences. Note that during a basketball game each player attempts around 20 shots. 

Activity 2: Random Streaks

            With one chance in four of a streak of 5 to occur in 20 tosses, this makes for a surprising demonstration. Obviously, flipping a penny 20 times and writing it down would be a tedious display to say the least. Instead, hand out 20 pennies, one to each student in a couple of rows of seats. Have them flip their coins and call out heads or tails in order of their seat while a recorder scribes them on the blackboard. This will allow a quick and participatory way to collect the data. Of course, it should take only four trials to get at least one 5 in a row. 

            As the data is being recorded, you will note that whenever a series of three appears a tension will overtake the class which will either quickly pass, or mount to gasps of astonishment as seeming streaks of 5 or 6 in a row materialize before them.

            This activity may sound overly simple from this description, however, I have found that it leaves a lasting impression on students. 


          Tom Gilovich and Arnos Tversky demonstrate that people have faulty intuition about what random sequences look like. In fact, they have actually researched the hot hand with data from the 76er's basketball team. Gilovich and Tversky found that there is no meaningful statistical correlation between shots, and that a particular player's shooting average is the same for shots made after a basket to those after a missed shot 9. This analysis has fairly straight forward statistical arguments and is ideal for class presentation in conjunction with the above activity.


Figure 1. Optical illusion as an analogy to the cognitive clustering illusion. Even once one verifies that the brim of the hat is equal to the height, the eye continues to perceive the hat as taller than wide.

In general most conceive a random distribution to alternate back and forth much more than the math demonstrates; sequences of six in a row seem to most beyond chance. The propensity to assign a causal connection to such random sequences is called the clustering illusion, another pitfall of perception related to misinterpretation of random events. Gilovich argues that this bogus intuition is a cognitive illusion much like the optical illusion of the hat in Figure 1 in that even once one verifies that the brim of the hat is equal to the height, the eye continues to perceive the hat as taller than wide.

            So here is yet further instances of the mind's tendency to misinterpret. But wait, it gets worse!

Expectation Bias and Seeing Things

            Here is a short account of a famous instance of expectation bias, or just plain jumping to conclusions. In 1903, during a time of major discoveries of many new forms of radiation, Professor Rene Blondlot of the University of Nancy reported the discovery of a remarkable new radiation he labeled N-rays10. He claimed these rays were emitted by all things except green wood and some treated metals, and had similar penetrating properties akin to X-rays. A number of other French scientists had corroborated his findings by duplicating his experiments. In one experimental arrangement, the N-rays were said to refract through a metal prism, and that a spectrum of dark and light N-ray bands could be cast. Instead of an eyepiece the spectrometer had a vertical thread treated with luminous paint. N-ray bands were detected by Blondlot, determining by eye the faint glow of the string as an assistant called out angles and rotated the prism through a set of intervals.

            The journal Nature sent American physicist James Wood to investigate the amazing claims of the N-ray experiments. Wood was invited into Blondlot’s lab for a demonstration, and while waiting in the dark for Blondlot’s eyes to adjust, Wood quietly removed the metal prism from the apparatus. Although the prism was in Wood’s pocket, thus completely disabling the apparatus, Professor Blondlot nevertheless called out the presence and absence of N-rays exactly where he had reported and expected them to be11.

            The detection mechanism of Blondlot’s experiment had an unfortunately large subjective aspect, that of visually distinguishing a very feeble illumination, literally on the threshold of detection. Could Blondlot’s strong expectation to see the string glow really manifest in his perception, so that he really saw a glow when none were present? Many have come to this conclusion.

Activity 3: Observation of Bogus Inference 12

            Students are grouped into teams of four and given the following items; a plain white candle, a ruler, and a magnifying glass. They are instructed to brainstorm in their groups and write down as many observations about the candle as possible. After some allotted time (say 10 minutes), engage the groups to share what they observed and catalog the list on the board. Many aspects of the candle will be listed, including measurements of dimensions, color and texture. Invariably a student will remark that “it’s been burned”. This comment is used as a lead in to a discussion of the difference between an observation and an inference from many observations. At this point the students find that things are not as they initially appear.

            This activity does take some initial effort to set up. The candles that are distributed to the students are prepared in the following way. Starting with a brand new candle use a small sharp tool to scrape some wax from around the base of the wick to form a small crater. Next, take a permanent black marker and color the wick black. Finally, select one candle, actually light it, and from it drip wax on to the doctored candle, being careful to make the drips fall at odd angles.

            Returning to the activity: After a brief discussion on inferences, have the students concentrate on the wick of the candle, again compiling a list of observations. Here they will note that the black color does not come off upon handling (unlike a burnt candle) and that the wick is flexible (again quite unlike a burnt candle). Finally ask what alternative hypotheses could explain the appearance of the candle. Here they will be astonished that someone would actually have colored the wicks black, and indeed it is a bit of a dirty trick. At the same time, however, there were numerous clues that the candle could not have been previously burned: the scratch marks around the base of the wick, the flexible nature of the wick, the fact that the drip marks could not have possibly come from the top of the candle. All these striking inconsistencies were overlooked due to the expectation bias set by the assumptions formed upon first glance.


            In the case of Blondlot, perhaps the expectation came from his considerable investment in his own hypothesis, or was reinforced by his lab assistants not wanting to contradict their esteemed professor. Whatever the case, the lesson for the students is that his experimental procedure screamed out for the application of a blind test. If Blondlot had asked his assistant to do in a controlled fashion what Wood had imposed on him, N-rays may never have seen the printed page.

            There are other instructive and entertaining incidents in the annals of physics; one of which I highly recommend is the story of Martin Fleischmann and Stanley Pons' announcement of cold fusion in 1989. The account as told in Robert Park's book Voodoo Science13 gets to the very heart of the problem: signal on the threshold of detection above noise, subversion of peer review, lack of use of control samples (what is the result if you do not use heavy water in your vessel?), and of course a wide berth for expectation bias. 

The human pitfall of expectation bias is sometimes referred to as wishful thinking, and plays a role in the acceptance of many questionable beliefs including N-rays, cold fusion, ancient astronauts, claims of perpetual motion ("over unity") devices, and many alternative healing claims, to name a few.

Confirmation Bias and Overlooking Opportunities

            Related to expectation bias is another human pitfall, that of confirmation bias. Inductive reasoning has preeminence in science, and early on I explain how disconfirming evidence is more powerful that confirming evidence in deciding among competing hypotheses. Yet it seems almost unnatural for us to seek to disconfirm, as illustrated in the next activity.

Activity 4: Pick a Card

            Ask the class to consider the four cards in Figure 2. and to determine which card is the best choice to turn over to test the following statement: All cards with a vowel on one side have an even number on their opposite side. After giving the students adequate time for consideration, ask for a show of hands and write down the number of votes for each. Now repeat the process but this time ask them which one would be second best and record the votes accordingly.

Figure 2. Cards used in evaluation of the validity of the statement: All cards with a vowel on one side have an even number on their opposite side.


Typically for the selection of best card to turn over you will see a majority of votes for the A card, a significant number will choose the 2, a few will choose the 7, and hardly ever a vote for the J. For second best the statistics are similar except the 2 gets the majority and the A gets the second most, with the 7 obtaining a couple votes.

            What is the best choice?

Consider the A: If there is an even number then we have some confirming evidence, well and good. If there is an odd number then the statement is refuted, which is actually more decisive evidence— the statement is invalidated.

Consider the 2: If the opposite side is a vowel, again we have some positive evidence. However, if the opposite side of the card is a consonant, we have no new useful data.

Consider the 7: Here again, if the opposite side is a consonant we have no useful data. If,  however, the opposite side is a vowel we have again refuted the statement— this evidence eliminates the statement as a possible truth. 

Consider the J: Of course neither a vowel nor a consonant gives us any useful data.

            So we see that to invalidate the statement is to throw it out in one quick step, and such disconfirming evidence is much more powerful than confirmation. The 7 card should be given the second place spot. Yet, test after test shows that human judgment has a bias for confirmation; we automatically seek the confirming instance, that which agrees. 


           The most telling aspect of this activity is that it shows that even when we don’t have a vested interest in the validity of the statement (no wish or need for the statement to be true), we still seek the confirmatory evidence, however more powerful disconfirmation might be. This pitfall of perception is well documented in the psychology literature14.

          Once students have been made aware of these pitfalls, I have them research a number of popular topics with extraordinary claims: ancient astronauts, the Loch Ness monster, abductions by extraterrestrial visitors, palmistry, psychic detectives, bigfoot, spontaneous human combustion, out of body experiences, etc., the list is long! My students learn to recognize the potential role of these pitfalls in the evidence presented by the proponents for these claims, and I have found that these activities leave a strong and lasting impression.

           You, yourself, are the easiest person to fool... Once my students are convinced of this, I feel there is hope that many will graduate with the ability to discern the difference between astrology and astronomy.


Notes and References:

1. Carl Sagan, The Demon-Haunted World: Science as a Candle in the Dark. New York, Ballantine Books, 1997.

2. Frank Newport and Maura Strausberg, "Americans' Belief in Psychic and Paranormal Phenomena is up Over Last Decade", Gallup News Service, June 8, 2001,

3. Theodore Schick and Lewis Vaughn, How to Think About Weird Things, Critical Thinking for a New Age, Mayfield Publishing Co., California 1995


Michael Shermer, Why People Believe Weird Things : Pseudoscience, Superstition, and Other Confusions of Our Time, W H Freeman & Co., 1998


Nicholas Humphrey, Leaps of Faith: Science, Miracles, and the Search for Supernatural Consolation, Copernicus Books, 1999

4. Richard Feynman, Edward Hutchings (Ed), " Cargo Cult Science" in Surely You're Joking, Mr. Feynman!: Adventures of a Curious Character, W.W. Norton & Company 1997

5.  Thomas Gilovich, "Some Systematic Biases of Everyday Judgment", Skeptical Inquirer, pg. 231 March/April 1997

6. Thomas Gilovich, How We Know What Isn't So; The Fallibility of Human Reason in Everyday Life, The Free Press, New York 1991

7. James Randi, Secrets of the Physics, NOVA, PBS video, 1993

8. Forer, B.R., "The Fallacy of Personal Validation: A classroom Demonstration of Gullibility," Journal of Abnormal Psychology, 44, (1949) 118-121.

9. Arnos Tversky and Tomas Gilovich, "The Cold Facts About the Hot Hand", Chance, Vol 22, No. 2 (1980) 18-21

10. Mary Jo Nye, "N-rays: An episode in the history and psychology of science", Historical Studies in the Physical Sciences 11 (1), (1980) 127-156.


Irving Langmuir and Robert N. Hall, "Pathological science" Physics Today 42 (10), (1989) 36-48.

11. Robert W. Wood, "The N-rays", Nature 70, (1904) 530-531.

12. Terrance Shaw, "Spider Sniffing, Oobleck, Candles and Other Skeptic Creating Activities", Proceedings of the 1998 California Science Teachers Association Annual Meeting, (unpublished)

13. Robert L. Park, Voodoo Science : The Road from Foolishness to Fraud, Oxford Univ Press, 1999

14. P.C. Watson, "Reasoning", in B.M.Foss(Ed.), New Horizons in Psychology. Harmondsworth: Penguin.