Brain fingerprinting field studies comparing P300-MERMER and P300 brainwave responses in the detection of concealed information

The error rates for all four of the current studies were less than 1 %. The median statistical confidence for individual determinations was 99.9 %. Countermeasures had no effect. Previous studies that used the same or very similar methods have achieved very similar results: the same error rate, and nearly as high statistical confidences. These include the original Farwell and Donchin (1991) brain fingerprinting study, Farwell (1992), Farwell and Smith (2001), and the replication of the brain fingerprinting methods by Allen and Iacono (1997). (For a review, see Farwell 2012).

The results achieved when the standards have been followed are dramatically different, however, from the results achieved in studies that have applied different methods in an attempt to obtain the same results. Our data suggest that meeting all of the 20 brain fingerprinting scientific standards specified above is sufficient to produce less than 1 % error rate, extremely high statistical confidence for each determination, a low (or 0) rate of indeterminates, and extremely high resistance to countermeasures. Additional research is required to determine which of these standards are also necessary conditions for such performance.

Farwell and Donchin (1991) met 17 of the 20 standards, all but standards 4, 8, and 10. This demonstrates that these three standards, although part of the set of sufficient standards to produce high accuracy and statistical confidence, are not necessary. These three standards had not been formulated at the time of Farwell and Donchin. Standard 4 was developed in the FBI Agent Study reported herein, as described above. Standards 8 and 10 were developed in response to the challenges of field applications in situations with major consequences of the outcome.

Research in other laboratories suggests that at least the most essential of these standards are necessary for valid, accurate, and reliable results. Several studies that used different methods reported error rates an order of magnitude higher than those of the present study and previous similar ones, as well as much lower statistical confidences (no better than chance for information-absent subjects). (For a review see Farwell 2012). For example, Rosenfeld et al. (2004) reported overall 35 % error rate without countermeasures and 67 % error rate with countermeasures. Rosenfeld et al. (2008) and subsequent studies on the “complex trial protocol” reported an overall error rate of 15 % without countermeasures and 29 % with countermeasures. In all of these studies, statistical confidences (when reported) for information-present subjects were over 90 %, and statistical confidences for information-absent subjects averaged approximately 50 % (chance). Approximately half of the statistical confidences for information-absent subjects were less than chance (50 %). Miyake et al. (1993) reported an error rate of 35 %, with 17 % indeterminate. Mertens and Allen (2008) reported varying error rates for different methods, ranging from 18 % error rate with 58 % indeterminate to 42 % error rate with 0 % indeterminate.

Different methods produce different results. The standard brain fingerprinting methods applied in the current studies have produced the kind of results reported herein, and similar results in independent replications elsewhere (e.g. Allen and Iacono 1997). Different methods have produced dramatically different results—more than an order of magnitude higher error rates, dramatically lower statistical confidences (some methods averaging no better than chance for information-absent subjects), and susceptibility to countermeasures. We suggest that the differences in methods responsible for these differences in results have included the features discussed below. These are discussed in more detail, with specific reference to all related publications to date in English, in Farwell (2012). The following methods are highly valuable in our view for achieving results comparable to those of the present studies. Which standards are entirely necessary is a matter to be clarified by future research.

This is addressed by brain fingerprinting scientific standards 4–6.

We use targets that are relevant details of the investigated situation that have been revealed to the subject, and irrelevants that are plausible but incorrect situation-relevant details. For an information-present subject, the probes are virtually identical to the targets. The only difference is that a different button is pressed. For an information-absent subject, the probes are indistinguishable from the irrelevants. This allows us to directly address the critical question: For this subject, are the probes, like the targets, known details of the investigated situation, or are the probes, like the irrelevants, unknown or irrelevant but plausible details of the investigated situation. This allows us to use a powerful classification algorithm to answer the corresponding statistical question.⁵

Methods that use fundamentally different types of targets, such as meaningless number strings (Rosenfeld et al. 2008 and subsequent “complex trial protocol” studies), do not allow for a meaningful and accurate comparison of probe and target responses, thus disallowing the use of a classification algorithm and eliminating the possibility of obtaining high or meaningful statistical confidences for information-absent subjects.

This is addressed by brain fingerprinting scientific standards 14–16.

We apply a classification algorithm to classify the probe responses as being more similar to the target responses, which contain a P300-MERMER, or to the irrelevant responses, which do not. We use bootstrapping on correlations to determine whether the probe responses are more similar to the target responses or to the irrelevant responses. Bootstrapping computes the probability that the probe responses are more similar to the target responses, or to the irrelevant responses. This is the statistical confidence for each determination. If the probability is not high enough in either direction, then no determination is made; the outcome is indeterminate (although there were no indeterminates in the present studies.)

An alternative method is to ignore the target responses and use a comparison algorithm to determine if the probe responses are “larger” (variously defined) than the irrelevant responses (Rosenfeld et al. 2004, 2008; and all other studies from that laboratory). This results in two difficulties. First, according to both the predictions of the statistical model and the actual results reported in these studies, either the statistical confidence will be at or below chance, or the accuracy will be at or below chance. Second, if a reasonable, meaningful criterion (such as 90 %) is established for information-present determinations, then some subjects will be determined as information absent when there is up to an 89 % computed statistical probability that the opposite determination would be correct, and the selected determination is incorrect. The mathematical and statistical reasons for this are as follows.

Bootstrapping on P300 amplitude (however amplitude is defined) computes the probability that the probe P300s are “larger” than the irrelevant P300s. This constitutes the probability that information present is the correct determination, and thus provides a meaningful statistical confidence for an information-present determination (sometimes called “guilty”). All subjects whose data fail to meet the criterion (usually 90 %) for an information-present determination are classified as information absent (or “innocent”). This means that if the probability is as high as 89 % that information-present is the correct classification, the subjects is classified as information absent, with a statistical confidence of 11 %. This results in a determination that places a subject in a category where the statistics computed have determined a probability of up to 89 % that the opposite determination is in fact correct, and the selected determination is incorrect.⁶

This situation cannot be corrected by simply changing the criterion. This is because if the true state of the subject is information absent, then the probe and irrelevant waveforms are expected to be identical, and the expected value of any statistic to determine whether the probe responses are “larger” than the irrelevant responses is 50 %. That is, if the statistics work as designed, the average statistical confidence for an information-absent determination will be 50 %. The actual statistical confidences reported in, for example, Meixner et al. (2009) do in fact average approximately 50 % (chance) for information-absent subjects.⁷ If the criterion is lowered so that an information absent classification requires at least a better than chance statistical confidence (>50 %), then the accuracy rate falls to chance (50 %) or less, since on average half of the information-absent subjects will have less than 50 % statistical confidence. This is borne out by both the predictions of the statistical model and the actual reported data for the studies cited above that use bootstrapping to implement the comparison method rather than the classification method.

Rosenfeld et al. (2004, 2008) and their similar studies also demonstrated definitively that their comparison algorithm is much more susceptible to countermeasures than the classification algorithm applied in Farwell and Donchin (1991), Farwell and Smith (2001), and the present studies. To date, no one has ever beaten a brain fingerprinting test with countermeasures, despite a $100,000 reward for doing so. One reason for this difference is that with the comparison algorithm of Rosenfeld et al., all a subject has to accomplish with countermeasures is to reduce the statistical confidence for an information-present determination to less than 90 % (equivalent to greater than 10 % statistical confidence that information-absent is the correct determination), and he will be determined to be information absent.

With a classification algorithm, by contrast, an information absent-determination requires a meaningfully high statistical confidence that the subject is in fact information absent. In the studies reported herein, for example, all statistical confidences were over 95 % using the P300-MERMER and over 90 % using the P300.

For these reasons, the comparison method is inadequate for real-world forensic use. Obviously, there is no practical use in the real world for a result where there is as low as 11 % statistical confidence that the result is correct, as computed by the actual statistics applied to reach that result. It is simply not viable for an expert to testify in court, “We determined the subject to be information absent, and we have an 11 % confidence that this is the correct determination.” Even the average statistical confidence for information-absent determinations with the comparison method—50 %, no better than chance—is obviously insufficient for application in any case with non-trivial consequences.

Various classification methods using bootstrapping, and other classification methods using other statistical techniques, may be adequate for classifying the brain responses. The method we apply in the current research and the other replications cited above, using bootstrapping on double-centered correlations, has several advantages that we are convinced constitute one of the factors leading to the low error rate and high statistical confidence produced by this method. Bootstrapping on correlations provides a precise metric of the similarity between the pairs of response types, probe-target and probe-irrelevant. This metric takes into account not only the amplitude but also the latency, time course, and morphology of the response. It extracts information from every point in the relevant time epoch of the waveform (Farwell 2012).

This is addressed by brain fingerprinting scientific standard 17.

We compute the probability that the probe response is more similar to the target response, or, alternatively, to the irrelevant response. (The latter probability is 100 % minus the former.) The former is the statistical confidence for an information-present determination. The latter is the statistical confidence for an information-absent determination. We set a criterion for an information-present determination that requires a reasonable level of probability that this determination is correct. In the present study the criterion was 90 %. All information-present determinations would still have been correct, however, if we had used a 95 % criterion.

We set a separate criterion for an information-absent determination that also requires a reasonable level of probability that the determination is correct. In the present studies we used a 70 % criterion for information-absent determinations. All determinations would still have been correct, however, if we had used a 95 % criterion.

If a subject’s data do not meet either criterion, then the subject is not classified as either information present or information absent. The outcome is indeterminate.

When only one criterion is set, then, if the criterion is high enough to be meaningful for information-present determinations, it is not viable for information-absent determinations. As described immediately above, use of only a single criterion inevitably results in substantial decrements in statistical confidence, accuracy, or both.

This is addressed by brain fingerprinting scientific standards 7–10.

We describe the significance of each probe stimulus in the context of the investigated situation, without of course revealing to the subject which stimulus is the probe and which are similar irrelevants. For example, “Mr. Subject, you will see a knife, a pistol, a rifle, and a baseball bat. One of these is the murder weapon.”

The P300-MERMER manifests the process of context updating. That is, the subject takes note of relevant, meaningful information delivered by the stimulus. Things are significant to a person in context. Establishing the context of the investigated situation, and the significance of each probe stimulus in that context, ensures that if the subject knows the relevant information he will recognize the probes.

For example, if the victim wore a yellow blouse, simply presenting the words “yellow,” “blue,” “red,” and “green” as stimuli, without any explanation of the context in which one of these is meaningful, would likely be ineffective. Telling the subject that one of the stimuli would be an item of clothing worn by the victim, and then presenting “yellow blouse,” “blue skirt,” etc., would be much more likely to elicit a response revealing the concealed knowledge for a subject who possessed it.

It is not surprising that studies that did not describe the significance of the probes have reported higher error rates. This is discussed in detail in Farwell (2012).

This is addressed by brain fingerprinting scientific standard 18.

Brain fingerprinting detects information stored in the brain. It does not detect how it got there. It does not detect what a subject should know, would know, or could know and under what circumstances (e.g., if he committed a crime). It only detects what a subject actually does know. It detects information. It does not detect guilt or innocence. It does not detect past actions.

Science, and the scientific testimony of expert witnesses in cases where brain fingerprinting is applied, must confine conclusions to what the science actually shows. A brain fingerprinting test shows that the subject does or does not know the specific details about the investigated situation embodied in the probe stimuli. It is up to the attorneys to argue, and the judge and jury to decide, based on brain fingerprinting and all other available evidence, whether a crime took place, what the crime was, and who is guilty or not. When brain fingerprinting experts have testified in court (e.g., Harrington v. State 2001), they have correctly confined their testimony to explaining the science and presenting the result in terms of what the subject knows or does not know.

Prior to a brain fingerprinting test, a criminal investigator investigates the crime, develops his theory of the crime, and develops a set of relevant information embodied in the probe stimuli that he believes constitute the salient features of the crime. This is the non-scientific opinion of the criminal investigator, based on his professional expertise and judgment. The scientist who testifies regarding the brain fingerprinting test does not opine on whether the suspect took part in a crime. He does not even opine on whether the information contained in the probe stimuli which was provided by the criminal investigator accurately represents the crime, or whether a crime took place at all. His testimony about the case is confined to the scientific conclusion that is warranted by the data: at the time of the test, this subject does or does not know these specific details about the crime. These results must be interpreted in light of all of the other available evidence.

Brain fingerprinting provides objective evidence of the contents of human memory. Witness testimony provides a subjective (and not always truthful) account of the contents of human memory. Like eyewitness testimony, brain fingerprinting evidence must be evaluated in light of the well-known limitations of human memory. Judges and juries must apply the same considerations in evaluating brain fingerprinting evidence that they do in evaluating witness testimony. All of this is in the domain of the judge and jury. The role of the brain fingerprinting scientist is simply to present the scientifically established facts regarding what the subject does or does not know, along with an explanation of the science that establishes these fact.

In any case, any legitimate scientific report must be confined to reporting what brain fingerprinting actually detects: the subject’s knowledge or lack of knowledge of the specific information contained in the probe stimuli as relevant in the context of the crime.

Brain fingerprinting is like all other forensic sciences in this regard. A DNA scientist reports that Sample A (ostensibly from the crime scene) matches Sample B (ostensibly from the suspect). He does not conclude that therefore the subject committed a murder. Similarly, brain fingerprinting experts whose testimony as expert witnesses has been ruled admissible in court have confined their testimony to the science, specifically to the finding of what information about the crime the subject possessed, along with an explanation of the scientific principles and methods involved in testing for the presence of that information (Harrington v. State 2001; Farwell 2012).

Some studies that have failed to confine conclusions to what the science measures have consequently reported high error rates. Some examples are described immediately below.

This is addressed by brain fingerprinting scientific standards 3, 4, and 18–20.

Experimental control is vital to the validity and success of every experiment. Understanding what the science measures and does not measure is critical for establishing and maintaining proper experimental control.

In the field, the relevant knowledge embodied in the probes is provided to the scientist by the criminal investigator. In the laboratory, the relevant knowledge is fabricated by the experimenter and imparted to the subject through a knowledge-imparting procedure such as a mock crime or training program.

The brainwave test does not test whether or not the subject participated in the crime (in a field study) or the knowledge-imparting procedure (in a laboratory study). Drawing conclusions beyond what the science addresses, and mistakenly assuming that the testing procedure detects (or somehow should detect) past actions rather than knowledge, has led to errors in research that have produced marked decrements in accuracy as well as erroneous conclusions and misinterpretation of results. The mistaken assumption that brainwave testing detects (or should detect) actions rather than knowledge has led some researchers to confound the knowledge-imparting procedure with the knowledge-detection procedure. This may result in apparent low accuracy that is actually caused by the failure of the knowledge-imparting procedure rather than the failure of the knowledge-detection procedure. Some researchers (e.g., Mertens and Allen 2008) implement a knowledge-imparting procedure such as a mock crime, and then test for knowledge that is intended to be imparted by the knowledge-imparting procedure, without any independent verification that the knowledge-imparting procedure actually succeeded in imparting the relevant knowledge. When the knowledge-detection procedure does not detect the knowledge, they conclude that the knowledge-detection procedure failed. In fact, the knowledge-imparting procedure may have failed to impart the knowledge, and the knowledge detection procedure then correctly determined that the knowledge was not there. Failure of the knowledge-imparting procedure to impart the knowledge must not be confounded with failure of the knowledge-detection procedure to detect it. No technique, no matter how perfect, can detect knowledge that is not there to be detected. (For a review and discussion, see Farwell 2012).

One of the reasons that the present studies were able to achieve, correctly evaluate and quantify, and report a high level of accuracy is that the experimental design independently verified the presence or absence of the knowledge that the test sought to detect. In other words, ground truth was properly defined and clearly determined. This was accomplished through post-test interviews and other means, as described above.

This is addressed by brain fingerprinting scientific standard 11.

In order to emit a P300-MERMER, the subject must read and process the eliciting probe stimulus. Accommodating subjects such as undergraduate students who are facing no life-changing consequences of the outcome of the test can generally be trusted to read and process the stimuli simply because they are instructed to do so. Subjects in field tests whose life or freedom may depend on the outcome of the test are less accommodating. In our field experience, such subjects will do everything they can to avoid cooperating with the test, to the extent that they can do so without any overt indication.

In some alternative techniques such as the “complex trial protocol” (Rosenfeld et al. 2008 and subsequent studies), there is no such requirement. Stimuli alternate one-to-one, totally predictably, between a target or nontarget (meaningless number strings) followed by a meaningful probe or irrelevant stimulus. Differential button presses are required on every other stimulus presentation, only to the target/nontarget stimuli. In response to the probe and irrelevant stimuli, the subjects is only required to press an “I saw it” button indicating that something appeared in that general area of the screen. In the predictably alternating sequence, the subject can read and process the target/nontarget stimulus and make the appropriate button press, then ignore or not even look at the next probe/irrelevant stimulus but simply press a button when there is a brightening in that area of the visual field, and then read and respond to the next target/nontarget, and so on.

By not reading and processing the probe and irrelevant stimuli, the subject can perform the necessary task on every trial and avoid emitting a differential brain response indicating that he has recognized the situation-relevant probes. The analysis compares the probe and irrelevant responses as described above in the discussion of comparison versus classification algorithms. (Targets and nontargets are ignored in the analysis.) The subject is instructed to read and process the probe and irrelevant stimuli, but the task demands do not require him to do so and to prove that he has done so on every trial.

Motivated subjects with something real to hide and non-trivial consequences of being detected do not read and process the probe and irrelevant stimuli (Farwell 2012). They can and do perform the overt task as instructed without exhibiting any brain responses revealing their concealed knowledge. We verified this in the course of the present research.

In the present research program, we began a fourth study on the “complex trial protocol” that we discontinued for scientific and ethical reasons. We applied this method to test information-present subjects who had already been correctly detected with over 99 % statistical confidence by brain fingerprinting. The “complex trial protocol” produced a 100 % error rate, 0 % accuracy, for our first three motivated, real-world subjects. This is because all such subjects whom we tested figured out on their own how to beat the test. The simple method they used is not a countermeasure that must be taught, but simply the way that motivated subjects with something real to hide respond to the task requirements, as described immediately above. (In the absence of non-trivial consequences, accommodating laboratory subjects do not respond in this way, and are detected with greater than chance accuracy.) We discontinued the study when we concluded that it would be unethical to administer the “complex trial protocol” to any more subjects in a field setting with real-world consequences, in light of the 0 % accuracy rate and other shortcomings described above.

It has been suggested (Rosenfeld et al. 2008) that subjects can be coerced or tricked into reading and processing the probe and irrelevant stimuli by periodically asking the subject to remember what stimuli have been presented, and threatening to file a false report that the subject has “failed” the brainwave test if he does not correctly remember the stimuli. Such a report would be patently false, and in the real world the threat to file such a false report would be empty, because anyone who knows how the test works will realize that such a subject has in fact passed the brainwave test: his brain responses will show no indication of his possessing the concealed knowledge. In our view, such false threats are unethical in real-world cases.⁸ In any case, such threats are ineffective with a knowledgeable subject who has something real to hide and is facing non-trivial consequences. (For a detailed discussion, see Farwell 2012.)

This is addressed by brain fingerprinting scientific standard 13. All or virtually all studies that have reported low accuracy rates have failed to meet this standard (see Farwell 2012 for a review).

While the brain is engaging in the information-processing tasks of interest in our experiments, it is also engaging in a host of other activities. Consequently the signals that we seek to detect are mixed with myriad other patterns in the EEG, which for our purposes are noise. Fortunately, however, the signals of interest are time locked to the stimulus, since they manifest information-processing brain activity elicited by the stimulus. The other brain activity is not time locked to the stimulus. When the stimulus is presented many times and the responses time locked to the stimulus are averaged, the noise, which is not time-locked to the stimulus, averages to zero. Thus, noise is eliminated or minimized, leaving the signal that is of interest for our purposes. The more trials we include in the average, the greater the reduction of noise, the greater the signal-to-noise enhancement, and the greater the accuracy and statistical confidence of the results.

Bootstrapping, through repeated iterative sampling and averaging, provides a means of creating smooth, relatively noise-free averages while reintroducing the variability across the single trials that contribute to the averages. Accounting for this variability is essential for establishing the statistical confidence of the determination, and creating a smooth average is essential for applying the classification algorithm. We have found that using a minimum of 100 probe trials with a ratio of 1/6 probes, 1/6 targets and 2/3 irrelevants produces excellent results. Presenting fewer than 100 probe trials may produce adequate results, but in some studies has produced markedly lower accuracy (see Farwell 2012).

We also applied other signal-processing refinements that may have increased the accuracy and statistical confidence of the results obtained. For example, we used equal-ripple, optimal, finite impulse response, digital filters that are optimal in the strict mathematical meaning of the term. These have been shown to be highly effective in reducing noise while maintaining the amplitude, latency, and morphology of the response (Farwell et al. 1993). Such signal-to-noise enhancement techniques may have contributed to the accuracy we obtained, particularly since our classification algorithm takes into account all points in the relevant time epoch of the waveform.