Statistical literacy for classification under risk: an educational perspective

Our quality of life as individuals and the functioning of our society depend on our ability to understand risks and act appropriately in response. Recognizing this need, the education community has placed increasing emphasis on education for risk literacy. Effective education for risk literacy draws on the theory of rational choice under uncertainty, behavioral research on how people perceive and respond to risks, and educational research on how youngsters learn about risk. The aim is to develop educational approaches to guide the development of risk literacy and inculcate effective strategies for coping with risk.

Probability and decision theory emerged during the Enlightenment as a model of rational belief and decision-making in the presence of risk (Daston 1995). According to Laplace, “the theory of probabilities is nothing but common sense reduced to calculus; it enables us to appreciate with exactness that which accurate minds feel with a sort of instinct for which often they are unable to account,” (Laplace 1812). That is, probability formalizes the intuitions of the enlightened man. In the nineteenth century, with the rise of positivism and the quest for objectivity in science, probability fell out of favor as a model of rational thought. The mid-twentieth century brought a resurgence, with the introduction of personal probability (Savage 1954; de Finetti 1934). The personalist theory explicitly allows rational individuals to disagree on the probability of an event. Its inherent subjectivity brought skepticism and even hostility from adherents of an objective view of science. Another line of attack against probability came from the cognitive and behavioral sciences, with a flurry of research demonstrating systematic ways in which human reasoning fails to conform to the probability calculus (Kahneman et al. 1982).

Some researchers (Gigerenzer et al. 1999) argued that systematic deviations of human reasoning from probability are rational in an ecological sense. That is, humans have evolved a toolbox of “fast and frugal” strategies to draw inferences and make decisions in an environment of bounded cognitive resources and limited time. These ecologically rational strategies give results that are nearly as good as optimal but infeasible probabilistic methods. Arguments for ecological rationality are supported by comparing the output of computer models inspired by human reasoning with that of explicitly probabilistic computer models. For many problems we face in everyday life, cognitively inspired “fast and frugal” strategies perform nearly as well and, often better than, much more computationally intensive probabilistic approaches.

Taking a somewhat different approach, the field of decision analysis has focused on developing “cognitive tools” (von Winterfeldt and Edwards 1986) to help people come closer to the decision theoretic norm. Decision support tools informed by cognitive research help people to construct and reason with models that explore the logical consequences of their intuitive judgments, using the computer to perform probability calculations that exceed their intuitive capacity.

In our view, these streams of research are complementary, and together suggest promising directions for education in risk literacy. Acknowledging the systematic deviations of human thinking from the probability calculus, we seek to exploit naturalistic human reasoning and cognitive development to develop pedagogical strategies that capitalize on human strengths while overcoming the weaknesses of unaided reasoning. In this paper, we focus on a simple but commonly occurring class of problems—using evidential cues to classify a situation into one of two categories. We examine “fast and frugal” heuristics proposed in the literature, and show that their performance compares favorably to more computationally intensive methods from statistics and machine learning. We discuss the role of normative theory in justifying the performance of the simpler models, and the educational value of instilling a deep understanding of both the normative approaches and the simpler heuristics. We conclude with remarks on implications for mathematics curriculum.

Risk Literacy consists of four basic components (see Fig. 1, below):

The first component refers to the ability to identify risks and uncertainties properly. It is well known that humans tend to overestimate certain dangers and underestimate others, due to evolutionary biases and the effect of vivid news reports. Developing sound methods for estimating the impact of phenomena on our life requires plain statistical literacy (Martignon and Hoffrage 2019). The second component is one that requires elementary probabilistic training in classifying situations as dangerous or innocuous. Some tools for acquiring this component will be discussed in the next session. The third component also relies on mathematical skills for dealing with trade-off situations. Most human decisions are between alternatives, each one with certain advantages and disadvantages. Risk literacy requires an ability to evaluate the relative strength of advantages versus disadvantages. The fourth component has as a precondition being acquainted with the irreversibility of decisions: one has to cross the Rubicon and act. Here psychological conditions play important roles. Some people tend to make sudden, intuitive decisions, without even pondering about consequences, i.e., without passing through the third component. In this paper, we emphasize the second component and present mathematical tools, which may be acquired even in school, to support the ability to analyze and model risks.

Although the time allotted to probabilistic training in school varies across countries, there is consensus that the basic elements of probabilities, including Bayesian reasoning and expected values should be taught in secondary school. In some countries pre-service teachers for elementary school are now being trained in rudimentary probability based on icon arrays and simple proportions.

Classification problems pervade everyday life. As an exemplar, we consider a doctor examining a patient. The doctor typically is presented with a set of cues: symptoms reported by the patient; background information such as age, weight and gender; test results; physician observations. The task faced by the physician is to arrive at a diagnosis.

For illustrative purposes, we simplify the problem as follows. First, we suppose the doctor is answering a simple yes/no question: does the patient have a specific condition under consideration? Second, we assume that the input is a set of binary cues, e.g., a symptom is or is not present; a test reading is high or low. Finally, we assume that the physician reports a single answer: yes (condition present) or no (condition absent), with no opportunity to hedge the result. We are interested in whether the answer given by the physician is correct or incorrect. For this simplified set of problems, we examine the performance of two different “fast and frugal” classification tree methods and compare them with several methods drawn from the statistics and machine learning literature.

We compared the performance of five classification methods on eleven data sets taken from the medical and veterinary domains (Laskey and Martignon 2014). Most of the data sets were taken from the UC Irvine Machine Learning Repository (Bache and Lichman 2013). Each data set consisted of a class variable and between five and 22 evidence variables. The number of observations ranged from a minimum of 62 to a maximum of 768. Many of the evidence variables were two-valued; others were numerical. The numerical variables were dichotomized by assigning values larger than the median to the “high” category and values less than or equal to the median to the “low” category. A classifier was estimated for each method on each data set by selecting a random subset of the data as a training sample, dichotomizing continuous variables (if any), applying the classifier fitting method to the training sample, and then classifying each element of the remaining test sample. This process was repeated 1000 times for each classifier. This whole process was carried out for training samples of 15%, 50% and 90% of the data set. Thus, training samples ranged from a minimum of 9 observations to a maximum of 691 observations. Classifying a different set of observations from the ones used to train the classifiers is standard practice in classification research to test the ability of the method to generalize to new data. The five classification methods are:

Our results for logistic regression and the three classification tree methods are taken from (Martignon et al. 2012); the naïve Bayes results were first presented by the authors at ICOTs 9. Unsurprisingly, accuracy of all methods increases as the training sample becomes larger. Performance of the five methods is remarkably similar. Naïve Bayes has the best performance across the board, living up to its reputation as a simple and robust benchmark. In fact, we tried other more complex Bayesian methods, but none performed better than naïve Bayes, most likely due to the relatively small (by machine learning standards) data sets. However, naïve Bayes performed only slightly better than the other methods. The fast and frugal trees are only slightly less accurate than the computationally more expensive naïve Bayes (Fig. 4).

Our results have important implications for risk education, especially for the debate over how best to respond to the well-established literature demonstrating that people’s intuitive judgments do not live up to the Bayesian ideal. Fast and frugal trees are extremely simple computationally. Cue validities can be estimated using only counting and ratios. Once cue validities have been estimated, tree construction involves only a few simple rules. After the tree has been constructed, its use for classification involves only traversing the tree and answering one simple question at each node. This simple process yields a classifier almost as accurate as the Bayesian benchmark.

The most advanced operation required for constructing fast and frugal trees is estimating cue validities. It has been demonstrated that children as young as fourth grade can understand the concept of cue validity through enactive education approaches, manipulating towers of colored tinker cubes to represent the relationship between cues and outcomes (Martignon and Monti 2010). Children can apply their understanding to can answer questions on the validity of cues. Thus, even at a young age, children can acquire basic reasoning strategies for coping with risks, strategies that will serve them well as they reach adulthood.

Building on this foundation, students in secondary school can comprehend the concept of conditional independence, and are prepared to understand the naïve Bayes model as well as more complex Bayesian network models (Krauss et al. 2010). At this stage, students can evaluate trade-offs between computational cost and accuracy, and to choose an approach that balances these objectives appropriately for the situation. Students at the secondary level could perform studies of the kind described in this paper, comparing computation and accuracy of naïve Bayes with fast and frugal strategies. Thus, they would be able to conclude for themselves that fast and frugal approaches yield nearly the same accuracy as the Bayesian benchmark, while requiring far less computation.