A new use case for argumentation support tools: supporting discussions of Bayesian analyses of complex criminal cases

Probability theory (Hacking 2001) defines how probabilities between 0 and 1 (or equivalently between 0% and 100%) can be assigned to the truth of statements. As for notation, Pr(A) stands for the unconditional probability of A while \(Pr(A \mid B)\) stands for the conditional probability of A given B. In criminal cases the court is (on a Bayesian account) interested in the conditional probability \(Pr(H \mid E)\) of a hypothesis of interest (for instance, that the suspect is guilty of the charge) given evidence E (where E may be a conjunction of individual pieces of evidence). For any statement A, the probabilities of A and \(\lnot A\) add up to 1. The same holds for \(Pr(A \mid C)\) and \(Pr(\lnot A \mid C)\) for any C. Two pieces of evidence \(E_1\) and \(E_2\) are said to be statistically independent given a hypothesis H if learning that \(E_2\) is true does not change \(Pr(E_1 \mid H)\), i.e., if \(Pr(E_1 \mid H \wedge E_2) = Pr(E_1 \mid H)\). The axioms of probability imply that such independence is symmetric. The axioms also imply the following theorems (here given in odds form). Let \(E_1, \ldots , E_n\) be pieces of evidence and H a hypothesis. Then:This formula is often called the chain rule (in odds form). The fractions on the extreme right and left are, respectively, the prior and posterior odds of H and \(\lnot H\). Given that probabilities of H and \(\lnot H\) add up to 1, the prior, respectively, posterior probability of H can be easily computed from them.

If all of \(E_1, \ldots , E_n\) are statistically independent from each other given H, then the chain rule reduces toThis is the formula used by A in his reports. Its attractiveness is that to determine the posterior odds of a hypothesis, it suffices to, respectively, multiply its prior odds with the so-called likelihood ratio, or evidential force, of each piece of evidence. For each piece of evidence \(E_i\) all that needs to be specified is how much more or less likely \(E_i\) is given H than given \(\lnot H\). If this value exceeds (is less than) 1, then \(E_i\) makes H more (less) probable compared to before \(E_i\) was known, while if the value equals 1, the probability of H remains the same, so \(E_i\) is irrelevant for H.

Elegant as this way of thinking is, it is usually not applicable since often the global independence assumption concerning the evidence is not justified. Hence the name naive Bayes. Resorting to the general version of the theorem, the chain rule, is often also cumbersome, because of the many combinations of pieces of evidence that have to be taken into account. As a solution, Bayesian networks (Fenton and Neil 2013) have been proposed, which graphically display possible independencies with directed links between nodes representing probabilistic variables (e.g. statements that can be true or false). For each value of each node, all that needs to be specified is its conditional probability given all combinations of all values of all its parents. Evidence can be entered in the network by setting the probability of the value of the corresponding node to 1, after which the probabilities of the values of the remaining nodes can be updated. For present purposes the most relevant observation is that to specify a Bayesian network, not only probabilities but also specific (in)dependencies have to be asserted.

Argumentation is the process of evaluating claims by providing and critically examining grounds for or against the claim. An important notion here is that of argument schemes (Walton et al. 2008), which capture typical forms of arguments as a scheme with a set of premises and a conclusion, plus a set of critical questions that have to be answered before the scheme can be used to derive conclusions. If a scheme is deductively valid, that is, if its premises guarantee the conclusion, then all critical questions of a scheme ask whether a premise is true. If a scheme is defeasibly valid, that is, if its premises create a presumption in favour of its conclusion, then the scheme also has critical questions pointing at exceptional circumstances under which this presumption is not warranted. One formal approach to argumentation, ASPIC⁺ (Modgil and Prakken 2014), formalises argument schemes as (deductive or defeasible) inference rules and critical questions as pointers to counterarguments: underminers attack an argument’s premise, undercutters state that there is an exception to a defeasible rule, and rebuttals have a conclusion that contradicts the conclusion of a defeasible inference. Arguments can be formed by chaining inference rules into directed graphs (which are trees if no premise is used more than once). Conflicts between arguments can be resolved with a given relative notion of argument strength, to see which arguments defeat each other. Then Dung’s (1995) abstract theory of argument evaluation can be used to determine which arguments are acceptable.

In the present paper argument schemes and their critical questions will be semiformally displayed. Critical questions asking whether the premises of the scheme are true will be left implicit. As the formal background ASPIC⁺ is assumed but the analysis will be such that it can also be formalised in similar argumentation formalisms or in related formalisms such as Defeasible Logic (Governatori et al. 2004). A formal background can provide a semantics for the notations and it can support automatic evaluation of reconstructed discussions. For example, as described by Bex et al. (2013), the reconstruction could be stored in the Argument Interchange Format format and then be exported to implementations of an argumentation logic, such as the online TOAST implementation of ASPIC⁺ (http://​toast.​arg-tech.​org).

When modelling expert testimony, an important scheme is, of course, the scheme of arguments from expert opinion. This also holds for Bayesian analyses, since expert judgement is a recognised source of subjective probabilities. So there is every reason to discuss the expertise issue in detail. The scheme below is modelled after Walton et al. (2008).

Argument scheme from expert opinion

The double horizontal line indicates that the scheme is presumptive. Therefore, the scheme has critical questions:Question (1) is about the level of expertise while question (2) is about personal bias. With respect to question (3) an implicit condition of use becomes relevant, namely, that the expert opinion scheme can only be used by those who are not themselves experts in domain D, such as the judges in the cases. I could, of course, not defeat A’s arguments by saying that I am also an expert and I say \(\lnot P\).

In probability theory sometimes a sharp distinction is made between frequentist (objective) and epistemic (subjective) Bayesian probability theory. Probabilities based on frequencies as reported by statistics would be objectively justified, while probabilities reflecting a person’s degrees of belief would be just subjective. However, selecting, interpreting and applying statistics involves judgement, which could be subjective. Moreover, a person’s degrees of belief could be more than just subjective if they are about a subject matter in which s/he is an expert. The same actually holds for the judgements involved in applying frequency information and statistics: if these judgements are made by someone who is an expert in the problem at hand, these judgements may again be more than purely subjective. So the issue of expertise is crucial in both ‘objective’ (frequentist) and ‘subjective’ (epistemic) Bayes (likewise Biedermann et al. (2017)).

It should be noted that in many cases, an expert asserts not a proposition but an argument. In this section I confine myself to assertions of statements; in Sect. 4.2 I will discuss how expert assertions of arguments can be modelled as sequences of assertions of statements.

In Sect. 4.1 I assumed that an expert asserts propositions but often an expert will assert an argument. Asserting an argument includes but goes beyond asserting its premises and conclusion: the expert also claims that the conclusion has to be accepted because of the premises. In many cases such an argument can be attacked by rebutting, undercutting or undermining it. However, sometimes a critic might want to say that the argument is inherently fallacious. This is not the same as stating an undercutting argument, since an undercutter merely claims that there is an exception to an otherwise acceptable inference rule. Although any expert may make reasoning errors, such errors will especially in the present domain, with many complex probabilistic arguments and sometimes complex statistical arguments, be frequent. So arguments from reasoning errors deserve to be studied in this paper.

In the two cases of the present case study, several arguments about argument validity were exchanged. An example from the Breda Six case is when I claimed that contrary to what A claimed, a probability concluded by A does not follow in probability theory from other probabilities assumed by A. After some discussion I had to admit that I was wrong and that A’s argument was deductively valid.

In his report in the Oosterland case, A first estimated the probability of fifteen arson cases in a town like Oosterland in a six-months period given the hypothesis that they were not related as at most one in a million. He then concluded from this that the fifteen arson cases that he considered in his report cannot have been coincidence and that they must have been related. More formally, this argument can be rendered as ‘The probability that the fifteen incidents happen given that they are unrelated is at most 1 in a million, therefore the probability that the fifteen incidents are unrelated given that they happen is very low’. From this he in turn concluded that serial arsonists must have been active, since no other relation would be realistically possible. In my report I claimed that this argument is an instance of the prosecutor fallacy, since it confuses the probability that the fifteen incidents happen given that they are not related with the probability that the fifteen incidents are not related given that they happen (this is sometimes also called the fallacy of the transposed conditional). A full argument would not just state but show that given the axioms of probability theory the second probability is not implied by the first. However, I left this part of my argument implicit since I assumed it to be generally known.

One way to show that A’s argument is fallacious is by giving a simple formal counterexample, for example, to specify for some E and H that \(Pr(E \mid H) = Pr(E \mid \lnot H) = 1/1.000.000\) so that the likelihood ratio of E with respect to H equals 1, so that the posterior probability \(Pr(H \mid E)\) equals the prior probability Pr(H), which can be any value. As a reply, it might be argued that in the present case \(Pr(E \mid \lnot H)\) (i.e. the probability of the 15 incidents given that they are related) is much higher than 1 / 1.000.000 and that, moreover, the prior is such that an application of Bayes’ rule would yield a very low posterior \(Pr(H \mid E)\). However, this would not reinstate the fallacious argument but instead replace it with a valid argument.

From the point of view of argument visualisation one would like to have the following. For a given probabilistic statement \(\varphi \), such as a link or probability in a Bayesian network, or a probability that is part of a likelihood ratio specified by an expert, the user could click on the statement and be able to inspect the following argument:

Expert E asserts that \(\psi _1,\ldots ,\psi _n\)

Expert E asserts that \(\psi _1,\ldots ,\psi _n\) imply \(\varphi \)

Therefore, \(\varphi \) because of \(\psi _1,\ldots ,\psi _n\).

Discussions about reasoning errors by experts can then schematically be represented as follows with several applications of the witness testimony scheme combined with an inference from their conclusions:

Then if the fact finder both accepts \(\psi _1,\ldots ,\psi _n\) and that \(\psi _1,\ldots ,\psi _n\) imply \(\varphi \) on the basis of the expert testimony, the fact finder should also accept \(\varphi \).

This approach can be used to model any debate about reasoning errors by experts. I now illustrate it with a probabilistic example from the present case studies, namely, the dispute in the Oosterland case about my claim that A had committed the prosecutor fallacy. In terms of the just-sketched approach, this dispute was about the final application of this sequence of expert testimony schemes and can be modelled as follows.

Here P is the conclusion of the first argument and \(>>\) means ‘much greater than’. The conclusions of these two arguments deductively imply \(Pr( related \mid incidents )>> 0.5\).

My counterargument can be modelled as follows, where C stands for a description of the above-given counterexample:

In ASPIC⁺ and similar argumentation systems this argument defeats the preceding one, since it is a deductive argument with universally true premises while its target is defeasible.

The reliability of the method used by A was an issue in both cases. One might expect that this issue will frequently arise in debates between experts, so a discussion is in order, even though the arguments in the two cases did not clearly instantiate recognised argument schemes. Neither courts defined what they meant with reliability. I defined it as the question whether different analysts applying the same method to the same problem will arrive at the same or at least similar result. In both my reports I argued that Bayesian analysis is not a reliable method in this sense, since in the academic literature there is no consensus on the right way of using Bayes for analysing complex criminal cases. In fact, there seems to be consensus on two things: that naive Bayes is too simplistic for complex criminal cases and that further research is needed before reliable methods can be offered to courts. Much current research concentrates on applying Bayesian networks (Fenton and Neil 2011; Lagnado et al. 2013; Vlek et al. 2016; de Zoete et al. 2015) but the results are still preliminary while, moreover, it is almost exclusively concerned with the structure of Bayesian networks and leaves aside the question how reliable probabilities can be established. For these reasons, it seems quite likely that different analysts will produce quite different Bayesian analyses of the same case. This warrants the conclusion that A’s method is therefore not reliable in the sense defined above. This conclusion generalises to any use of Bayesian probability theory to analyse complex criminal cases.

In neither of the two cases did A respond to this criticism. The court in the Six of Breda case agreed with my analysis and therefore disregarded the conclusions of A’s report. The court in the Oosterland case did not comment on this issue except for summarising my argument. Since the court chose to disregard A’s report “considering” my criticisms, this indicates that they may have agreed with my argument.

Like with the scheme from expert testimony, methodological issues can concern either the entire report or specific issues. An example of the latter was the debates in both cases between A and me about the appropriateness of the global independence assumption implied by A’s adoption of naive Bayes. For example, in both cases A used quite specific pieces of evidence to assess the prior probabilities of his hypotheses. The axioms of probability theory then imply that in every likelihood ratio the evidence has to be conditioned not just on the hypotheses but also on the evidence used for assess their priors, unless the pieces of evidence can be regarded as statistically independent given these hypotheses. A did not condition the likelihood ratios in this way but neither provided arguments for the required independence assumptions. I criticised him on the grounds that according to the axioms of probability theory he should have done either one or the other of these things. This is general methodological criticism that translates into specific criticism of A’s final argument that the posterior odds can be calculated by multiplying his prior odds and likelihood ratios. There are two ways to formally model this specific criticism. One is to interpret A’s final argument as having as additional premises the required independence assumptions and then to observe that these additional assumptions are not backed by evidence or argument. The other way is to leave these assumptions out of the interpreted argument and then to build an argument in the style of Sect. 4.2 that A’s final argument is deductively invalid.

In the two case studies, several analogical arguments were used. The following version of the scheme for such arguments is fairly standard; cf. (Walton et al. 2008, pp. 58,315).

Argument scheme from analogy

Critical questions:As is well-known, for special domains the scheme can be concretised, such as for case-based reasoning in the law (Bench-Capon 2017). However, for present purposes the above version will do.

One use of analogy was in the Breda Six case, concerning the evidence that two of the three accused women worked in a snack-bar next door to the crime scene. In his report, A specified a likelihood ratio of the “coincidence” between this piece of evidence and other evidence that these two suspects knew three other suspects who by an anonymous informant of the criminal intelligence unit of the police (CID) had been mentioned as being involved in the crime. A first specified the denominator of this likelihood ratio (the probability of the coincidence given innocence of all six accused) as 1 in 500 to 1 in 1000 (on grounds that are irrelevant here). He then said “this agrees with a likelihood ratio of the coincidence of 500 to 1000”. Taking this literally, this is a simple error, since before this conclusion can be drawn, first the probability of the coincidence given A’s offender hypothesis has to be determined. However, other parts of A’s report reveal that he set this probability to 1. Here he used an analogy with a hypothetical case in which a burglar breaks into a house by using a key of the house. Suppose a suspect is caught in possession of the key. According to A, possession of the key is a necessary element of the crime, so given that of the suspect committed the burglary, the probability that he possesses the key is 1. In the same way, A argued, the coincidence in the Breda Six case is a necessary element in the crime, since A’s offender hypothesis was that at least some of the six accused were involved in the crime, where one or more female accused lured the victim to the restaurant where the crime took place. I criticised this on the grounds that, firstly, such luring can also be done by someone who does not work next door to the restaurant, such as the third female suspect; and, second, that the joint innocence of the two female suspects working next door to the restaurant is consistent with A’s offender hypothesis. So the coincidence cannot be regarded as a necessary element of the crime. In pointing this out, I observed that this is a relevant difference with A’s hypothetical burglary case, in which possession of the key is a necessary element of the crime. Thus I criticised A’s analogy by using the first critical question of the analogy scheme.

Arguably another use of analogy in the Breda Six case is in the reasoning why A can be regarded as an expert in Bayesian analysis of complex criminal cases (Sect. 4.1 above). A reasoning step that is arguably implicit in the court’s ruling on this issue is that experts in Bayesian analysis in physics are also experts in Bayesian analysis of complex criminal cases. This argument can be seen as an argument from analogy, referring to the supposed similarity between Bayesian analysis of problems in physics and of complex criminal cases. My argument that the one does not imply the other can be regarded as another application of the first critical question of the analogy scheme. One might expect that in debates about the use of Bayes by experts in court such analogical arguments referring to expertise in supposedly similar fields will arise more often.

In fact, the same holds for arguments used in combination with arguments from statistics, such as the arguments discussed above that statistics on arson in Japan and the UK also apply to the Netherlands. These arguments, too, can be seen as analogical arguments so they, too, can be criticised with the critical questions of the analogy scheme.

Finally, in my reply to A’s reply in the Oosterland case I made use of a medical hypothetical to criticise A’s claim that he is able to show the court which questions it has to consider. My hypothetical was part of a rhetorical question whether a medical specialist investigating a seriously ill patient would ask a climate physicist to tell him which medical investigations he had to perform.

One might expect that in a probabilistic analysis of a complex criminal case, arguments from statistics to individual probability statements are frequent. Yet in my two cases most probability judgements were not based on statistics; in just a few cases A used them to support his judgements. In some other cases A used a quasi-frequentist approach. For example, in the Oosterland case he assessed the probability that the suspect and someone else (a suspect in a related case) were best friends given the innocence hypothesis by first observing that Oosterland has 2400 inhabitants and then supposing that for men like the suspect there were 200 candidates in Oosterland for being his best friend, thus arriving at a probability of 1 in 200 given innocence of both. This illustrates that even if probability judgements are based on data, the step from data to probabilities can involve subjective assumptions (in this case that there were 200 candidates for being the suspect’s best friend).

In its most basic form, an argument from a statistical frequency to an individual probability takes the following form.

Argument scheme from statistical frequencies

It is important to note that this scheme is presumptive: there is no necessary relation between a frequency statement about a class and a conditional probability statement about a member of that class. So the scheme has critical questions other than whether the premises are true. Hacking (2001) calls the scheme the ‘frequency principle’. He notes that it is justified on the assumption that nothing else relevant is known besides the frequency and that a is an F; he also notes that a lot of judgement can go into the question which other information can be relevant.

Before considering the scheme’s critical questions, let us look at how the first premise can be established. One way is by statistical induction:

This scheme is not treated in the usual accounts of argument schemes, such as Walton et al. (2008). A full investigation of ways to criticise uses of the scheme would lead us to the field of statistics, which is beyond the scope of this paper. For now it suffices to list two obvious critical questions: whether the sample of investigated F’s is biased and whether it is large enough.

In my cases, A derived some statistical information from sources. For example, in the Breda Six case he used statistics reported in a criminological publication on the frequencies of confessions of denials among various ethnic groups in the Netherlands. The reasoning then becomes:

E says that S is a relevant statistic, E is expert on this, therefore (presumably), S is a relevant statistic. Furthermore, S says that the proportion of investigated F’s that were G’s is n / m, therefore (presumably) the proportion of investigated F’s that were G’s is n / m.

The final conclusion then feeds into the scheme from statistical frequencies. In my report on the Breda Six case, I did not criticise A’s specific selection of statistics on confessions and denials but I did note in general that selection of relevant and reliable statistics from the research literature requires expertise in the subject matter of that literature. I then observed that there was no evidence that A possessed criminological expertise of the relevant kind, thus in fact attacking the second premise of this line of reasoning. All this illustrates that even in reasoning from statistics the argument scheme from expert opinion is relevant.

I now turn to three possible critical questions of the scheme from statistical frequencies (there may be more).Another scheme that was used by A in deriving probability judgements from statistics was the scheme from analogy (cf. Sect. 4.4). For example, in his report in the Oosterland case, he based his assessment of the probability of fifteen arson cases in a town like Oosterland in a half-year period given that no serial arsonist was active in Oosterland in that period among other things on statistics on arson in Japan and the United Kingdom. Applying this statistic to The Netherlands assumes that Japan and the United Kingdom are relevantly similar to the Netherlands as regards (serial) arson. This seems a quite common way of using statistics for deriving probability judgements. Here again the expertise issue comes up, since judging whether two countries are relevantly similar as regards (serial) arson requires domain expertise relevant to that question. Here too my general criticism was that there was no evidence that A, being a climate physicist, possesses such relevant expertise.

Summarising, reasoning from statistics can be a combination of at least the following presumptive argument schemes: arguments from statistical frequencies, arguments from statistical induction, arguments from expert opinion and arguments from analogy. In addition, specific methodological concerns from the field of statistics can arise. Therefore, a full model of evaluating arguments from statistics cannot be developed without involving statisticians.

Finally, the analysis in this section is also relevant for argumentative and story-based reasoning approaches. In both these approaches qualitative defeasible generalisations are very important. For example, in argumentative approaches evidential generalisations connect evidence to conclusions, such as witnesses usually speak the truth. And in story-based approaches causal generalisations connect hypotheses to evidence, such as extreme jealousy can cause feelings of murderous revenge. A qualitative version of the statistical induction scheme could be used to argue for such generalisations. For example:

Moreover, the critical questions of the scheme from statistical frequencies have their counterparts in critical examinations of arguments applying defeasible generalisations. The first two questions indicate the possibility of rebutting arguments (arguments for contradictary conclusions) on the basis of conflicting generalisations while the third question points at undercutting on uses of the qualitative variant of the statistical induction scheme.