Experts and evidence in deliberation: scrutinising the role of witnesses and evidence in mini-publics, a case study

It was intended that the same witnesses would be in all three of the wind farm juries. This made witness recruitment all the more challenging because witnesses were being asked to commit to three separate dates in different parts of the country. In practice, few witnesses could attend all three juries, and the anticipated time commitment to attend three juries on weekend days clearly made recruitment challenging. In total, the organisers approached 25 people to secure seven witnesses, one of which was a member of the Stewarding Board because simply no one else could be found. Two-thirds of the candidate witnesses declined on the grounds of existing time commitments, of which half explicitly stated family or caring responsibilities. Securing participation might have been particularly difficult in this case because it was a research project, with jury outcomes not feeding into decision-making. It is unclear whether a higher stakes outcome from the deliberative process constitutes a better incentive for prospective witnesses. Several of the organisers for projects in the meta-analysis reported that the identification, selection and recruitment of witnesses were time consuming and challenging, particularly, where the topic was sensitive or controversial. Organisers of the Electrical Futures project approached 34 candidates in order to recruit seven witnesses (Beckley 2016). The Oregon Citizens Initiative Review (CIR) projects found identifying non-advocate (‘neutral’) witnesses particularly problematic (Gastil et al. 2015).

What was also notable was that no female witnesses were involved in the wind farm juries. Although the organisers invited several women to participate as a witness, in practice none were able to attend. In this case, the lack of gender diversity in the witnesses could reflect poor representation of women in the energy and environment sectors more generally (IUCN-GGO 2018; UKCES 2015). Alternatively, it is possible that the process of witness recruitment does not facilitate diversity. Since female witnesses are underrepresented in nearly all of the meta-analysis projects, with three having no female witnesses at all, we suggest this is an area deserving of further research.

In the wind farms project, the witnesses’ expenses were covered, but they were otherwise not remunerated for their time. This is the case for all meta-analysis projects, too. The witnesses must therefore have had a strong motivation to participate, stemming from; for example, their work or community role, their personal interest in the topic, or their support for deliberative processes. Motivation is more likely to be stronger for individuals with strong views or interest in the matter, which might be in part why it can be particularly difficult to recruit ‘neutral’ witnesses, especially when ‘the issue involves extensive political controversy and scientific uncertainty’ as expert neutrality is compromised in these situations anyway (Elstub 2014). In these circumstances, the imperative to have witnesses with a diversity of political views is enhanced. The deliberative ideal is then for all views to be included on an issue and mini-publics attempt to at least achieve the values of diversity and balance. Our findings show, however, the ‘real world’ challenges in achieving this, which could then compromise the deliberative credentials of mini-publics.

In citizens’ juries, witnesses are included to ensure the participants have access to a range of relevant opinions and evidence, which they can then scrutinise and synthesise with their own views to form collective recommendations. For the wind farm project, the witnesses were drawn on to provide background information, impart knowledge from their experience in a specific area or field of work, to represent a certain stakeholder or standpoint on the issue.

Prior to the juries, in some cases as part of the recruitment process, the organisers briefed the witnesses on the purpose of the project, their role as witnesses, the rationale for including witnesses and guidelines on what to expect and level to pitch information at. The recruited witnesses were collectively advised by the organisers that their role was to “assist jurors in getting to grips with key aspects of the topic, and equip and empower them to deliberate and respond to the [jurors] task” (Roberts and Escobar 2015).

The ‘neutral’ witness was invited to “explain the wider context and cover the range of issues that are relevant to the topic, rather as a teacher might”, whereas the other witnesses were asked to “present evidence, stories and arguments that dig deeper into the topic and raise more specific issues and perspectives” relevant to their own stance on wind energy and onshore wind farm development (quotes from the Witnesses Brief). However, the witnesses’ interviews reveal that at least three of the witnesses felt their role was to ‘educate’ the jurors. As one expert commented, ‘We are custodians of knowledge, have we not got a right and a responsibility to actually pass that knowledge on to others, in a way that they will understand, which will help them to enhance their own lives, to inspire people to do something more’. Four of the witnesses doubted how much the jurors were actually able to learn during the process: ‘We are taking ordinary people, literally off the street…and expecting them, in a day, to take [in] pretty advanced material…I suspect that is unlikely’ and ‘People are not that interested in being engaged in these sorts of things and I doubt that you would get a hall full of people to take part or show interest in such a thing’. Two of the witnesses felt the format ought to have allowed them to engage in more teaching-style activities; to ‘try to explain issues to them that they may not understand or have difficulty understanding’ and ‘create an environment more like a classroom, the way you would teach at a university’. One of the witnesses ‘didn’t think the citizens were able to take anything in at all’ owing to the format (it is worth noting that evidence to the contrary is presented in Roberts and Escobar 2015).

Such remarks reiterate that these witnesses clearly felt their role was to stimulate learning rather than to “raise…issues and perspectives” relevant to the topic and the jurors’ task. Moreover, it is clear that the witnesses tend to see mini-public participants as having knowledge deficits that need to be corrected rather than being active scrutinisers of their own knowledge and evidence: ‘We can’t leave people to just look at newspapers…’,‘I think that [the process] needs rethinking through as to what you can expect you can get over to these people’ and ‘I don’t think people exercise a huge amount of thought’.

It may also be worth noting that in their brief the witnesses were asked to keep the information that they planned to present to a manageable level for the jurors. One could argue that it was the witnesses’ responsibility to ensure that their content was accessible and appropriate to their audience. Yet, one witness expressed that he ‘felt like I had to simplify, or even dumb down, the information for this process rather than really get into the subject’.

The witnesses, therefore, did not see the jurors as equal participants that they could learn from, nor appreciate the jurors experiential and contextual knowledge. Again, we see how the involvement of witnesses might approximate important deliberative norms, thereby compromising the mini-public.

Mini-publics routinely include independent epistemic experts. However, on contested issues advocates are also thought to be important to ensure participants are exposed to all salient views and information. The witnesses which advocated for a certain position for or against wind farms had—or could be perceived to have had—a vested interest in the matter at hand. The ‘pro-’ witnesses (of which there were three in total involved) all worked for a renewable energy company or trade body, both of which could be perceived to profit from wind energy development. In our interviews, the witnesses had conflicting ideas about whether having an allegiance ensured valuable expertise or biased perspectives. Interestingly, two of the witnesses felt that ‘an obvious vested interest… cheapens the value of the evidence’.

Perhaps for similar reasons of integrity, five of the seven witnesses we interviewed felt that remuneration was not appropriate or necessary, with one witness stating ‘if you’re paid by someone, it compromises your position’.

Several witnesses reflected in interview that it is difficult for any knowledgeable expert to be truly disinterested or impartial on their topic of interest. Most interviewees felt that the affiliations of other witnesses (industry or otherwise) affected the credibility of the evidence that those witnesses presented and the fairness of the overall evidence-giving process. They suggested that the ‘trustworthiness’ of the witness is more important than the evidence she/he provides.

While all witnesses expressed concern about bias in their fellow witnesses’ perspectives, several did not indicate that they recognise the potential for their own bias. The renewable energy lobbyists were accused of being the ‘industry mouthpiece’ who ‘would not deviate from the [industry] line’; whilst those witnesses in turn saw themselves as accountable to stakeholders/investors. Public trust in academics over industry representatives tends to be higher (Aitken et al. 2016), and yet academics were perceived by some other (non-academic) witnesses to be ‘loose cannons’. It was felt that academics were not accountable for the information that they presented in the way that industry representatives felt they were, and so the academics were freer to deviate from credible information sources. It is perhaps also worth noting that in their brief from the organisers, the witnesses were asked to provide evidence, stories and insights: they were not explicitly asked to provide references to their information, though they were encouraged to consider providing jurors with links to other materials such as briefs, blogs, websites, or videos which could act as such. Few of the witnesses opted to provide links to these materials.

While the jurors reported that the information provided by the ‘neutral’ introductory witness to be helpful, the jurors—and other witnesses—questioned his neutrality (Roberts and Escobar 2015). They perceived that the witness presented ‘quite a slanted bias’, and commented ‘there’s nobody to cross-examine him’; again highlighting the expert’s propensity to assume that the jurors were incapable of effectively scrutinising evidence. This was common for other projects; the neutrality of ‘neutral’ witnesses in the CIR process was queried by advocacy group representatives (Gastil et al. 2015).

The witnesses identified by the wind farm project Stewarding Board were from industry, academia and advocacy communities; they chose not to include experiential witnesses. Indeed, our meta-analysis shows that including experiential public witnesses is rare. This could be a weakness to the process; in our interviews, at least half of the witnesses in the wind farm juries felt that the information-giving sessions would have been strengthened had the jurors also heard testimonies from public ‘eyewitnesses’ or ‘everyday citizens’ about their personal experiences, in this case, around local wind farm development. It was felt that experiential witnesses may be perceived to be unaffiliated, and thus having a vested interest or bias. As one witness said: involving a member of the public would ‘greatly increase audience participation and outtake, … they tend to speak similar languages, you know, they recognise where he or she [the speaker] is coming from and are likely to pay much more attention. So, if I want to convince someone that community renewables is a good idea, I don’t send someone in a suit that knows everything about wind turbines. I send in someone that up until recently didn’t know anything about it and then was able to develop a successful project. That’s the kind of person who carries the needed credibility’. However, other witnesses claimed that experiential witnesses, pressure group speakers and community representatives were unqualified to be witnesses, since they would merely regurgitate the academic literature without the scientific judgement that ‘experts’ offer. The role of experiential witnesses as providers of evidence in citizens’ juries is a topic worthy of further investigation. Again, our evidence indicates that witnesses question the ability of mini-public participants to effectively evaluate and scrutinise evidence. This inhibits the ability of mini-publics to meet the deliberative ideal of joint policy enquiry.

As outlined in “Research approach: our case study” section, the wind farm juries involved three witness sessions, two of which were structured as a pro- and con-debate between two witnesses with ‘opposing’ perspectives. Each jury heard from five witnesses who attended in-person and gave oral presentations followed by a facilitated Q&A, all of which was in plenary.

Our meta-analysis finds five witnesses as the minimum and most common number. Further, similar to the wind farm case study, the most common format for evidence provision is 2–3 themed witness sessions, where each witness has 15–20 min to verbally present their perspective, perhaps with the assistance of slides, followed by longer Q&A with the jurors, sometimes as a panel of experts. Our meta-analysis indicates that the number of witnesses is to some extent controlled by the length of the process (Fig. 2a), and also the number of participants involved in the juries (Fig. 2b), which may both reflect the time available and the size of the project budget and scope. Regardless, we can infer that fewer than two sessions and five perspectives is generally thought to be insufficient for participants to gain a full understanding of the issue at hand.

All of the wind farm witnesses we interviewed felt the ‘pro and con’ debate format was problematic. Most felt it was too simplistic because there are more than two sides to complex issues, and three witnesses specifically expressed that they felt the debate style format was not conducive to learning. They would have preferred a more conversational session with the jurors (rather than one-way information provision) or learning activities over a longer time: ‘it has to be a conversation, people shouldn’t just listen’. There is some virtue of this approach, although according to research on deliberative polls ‘once participants learn that the experts disagree they feel freer to examine the issue for themselves’ (Fishkin 2009: 120), and thus the desire for continued expert interaction diminishes.

The wind farm project organisers felt that the third evidence-giving session was perhaps unnecessary (Roberts and Escobar 2015). Similarly, the CIR process evaluation concluded that having three evidence-giving sessions was superfluous, and that time would be better spent hearing from more ‘background’ or ‘neutral’ witnesses (Gastil et al. 2015).

Information-giving in the wind farm jury was all in plenary, and witnesses had to be physically present. This model of giving evidence is based upon ‘traditional—largely outdated—conceptions of knowledge and expertise’ (Fischer 2009) and knowledge transfer and there is scope to enhance and broaden this. Other projects have allowed for alternative modes of participation. For example, in the New South Wales Energy Enquiry witnesses could join via teleconference facilities, and witnesses who were unavailable for the Get to Know Nuclear juries could provide a pre-recorded statement or a written testimony. Further, in these juries, sometimes the interaction with witnesses took place as carousel style discussions in groups with the expert, rather than plenary talks. There could be merit in adopting non-traditional information-giving formats, particularly those which enhance how witness perspectives are communicated to jurors and receive information in return from the participants, to facilitate joint policy enquiry. For the first time, we see evidence that there was a desire from the witnesses to engage in interactive dialogue with the participants, but the evidence-giving format hindered this. Other evidence-giving formats show more promise, indicating that mini-publics can promote deliberative ideals on evidence scrutiny if appropriately organised.

Involving a range of witnesses in a citizens’ jury brings the added benefit—or confounding factor—of exposing the jurors to a variety of communication styles. It is common that the expert is more trusted than the evidence they provide (Aitken et al. 2016; Howell et al. 2014). The qualities of the witnesses (including their affiliation, discussed previously) and their communication style influence how witnesses will be received by jurors (Lansdell 2011).

The seven witnesses involved in the wind farm juries varied in terms of their presence, presentations, storytelling, delivery, interaction with the jurors, and interaction with each other (Roberts and Escobar 2015). These differences affected how the information was received by the jurors and—importantly—their perceptions of its veracity. In their interviews, the witnesses expressed concern that charisma would influence how the jurors take the information on board. It is perhaps to be expected that some witnesses might use emotion and persuasion to present their argument. These techniques are a key part of communication processes, and it is common for jurors to be exposed to them (Johnson et al. 2016). However, some approaches are not constructive for dialogue—phrases such as ‘you can’t argue with the facts’, ‘the science says’, or ‘the evidence is incontrovertible’ discourage reflection and dialogue, but are commonly used by witnesses, particularly when presenting technical evidence (Lansdell 2011).

For effective deliberation, it is therefore crucial that the jurors look beyond the performance and communication style to the argument beneath. There is good evidence that deliberations facilitated this in the wind farms project. The project researchers observe that, in their questions and discussions (during reflection on the first day and during deliberations on the second day), the jurors tended to focus on expanding their factual knowledge and clarifying understanding rather than on emotional reactions (Roberts and Escobar 2015).

The ability of a witness to answer questions appropriately also affected participants’ perceptions of witness veracity, especially when conflicting answers are given and there is a lack of certainty. Reflecting on this evidence, we suggest that active mediation by the facilitators can help ensure that Q&A exchanges are of maximum value for the participants’ deliberations. Further, the ability to answer questions is partly a product of how well prepared the witnesses are. In interview, several witnesses observed that, despite their briefing, they were not expecting such a demanding and adversarial experience. Therefore, the organisers may need to consider better ways to support witnesses to prepare, for example, via pre-event communication with someone with experience of giving testimonies to a mini-public audience.

Our interview witnesses raised concerns about differences in time and resources available to witnesses for preparation. One witness felt that it was unfair to put industry representatives alongside representatives from voluntary or interest groups, with industry groups able to create ‘slick presentations’ that interest groups cannot parallel. While all witnesses were invited to provide resources to supplement their presentation, few opted to do so, indicating differences in time resource or commitment. This is perhaps inevitable where any number of witnesses are involved in any event, and particularly so for contested topics. However, these differences can affect the perceived fairness of the overall evidence-giving process and can bring out other feelings, too. For example, in interview some wind farm witnesses called the skills of others into question, for example claiming ‘there aren’t really incentives for most academics to be good at communicating to non-academics’, adding, ‘it is not necessarily the case that academics know about issues that are critical in the ‘real world’’. This links to the witness’s critiques of each other’s’ biases and each other’s’ evidence presented in “The affiliations of the witnesses” and “Evidence quality” sections, emphasising that they doubted the jurors’ ability to scrutinise evidence. It is worth noting that these feelings were not necessarily observable during the juries themselves. The level of questioning the witnesses was subject to does though indicate that the jurors felt empowered and were actively scrutinising the evidence and consequently that mini-publics help promote this important deliberative norm.

As previously noted, during witness recruitment the organisers gave the witnesses a briefing note about their role, topics that the witness may choose to touch on, and some communication guidelines. Ultimately, however, the witnesses had freedom to present whatever information they felt was appropriate to inform the participants’ deliberations. The time restrictions of the evidence-giving format make this a particular challenge: to make effective use of the allotted time, witnesses must think quickly, articulate their arguments concisely, and avoid superfluous or distracting information.

The witnesses also had to consider the pitch of their information. In their brief, the witnesses were encouraged to prepare well, speak clearly and relatively slowly and should they opt to use slides, to pare back the text and use a large font. They were also advised that any graphs and diagrams that they opt to use should be very simple and that they should not assume any prior knowledge or familiarity with concepts or terms that are specific to the topic. Despite this, the project’s evaluators noted that some witness presentations nevertheless contained overly complex information, and their arguments were at times inaccessible, particularly in the Q&A. Further, most of the witnesses did not revise their presentation slides between juries (Roberts and Escobar 2015).

The majority of the witnesses had a great deal of experience addressing an expert audience, or a relatively uninformed audience of the general public. These skills and qualities were evidenced by all jurors, and yet at times the witnesses found it difficult to speak to the jurors’ level of expertise. This could be because the jurors are neither experts nor novices; some of the participants started the process with limited appreciation of the arguments around a topic, however, they were supported to rapidly elevate their technical and normative understanding (Bohman 1996). As such, the jurors became much more informed than the average citizen, but still lacked the issue-specific information held by experts. Citizens’ juries are therefore an unusual audience, and it seemed that the witnesses found it difficult to navigate the middle ground of the quasi-expert. Once again, this compromises the capacity of mini-publics to facilitate a conducive interactive dialogue between citizens and epistemic communities.

In interviews, the witnesses involved in the wind farm juries expressed concern about the quality of the evidence and the arguments put forward by their fellow witnesses, and for those advocating a position, their opponents in particular. The witnesses all felt similarly: that their own evidence was reputable and robust, and that they felt the other witness’s evidence was not: ‘Facts that were inaccurate presented on the other side and also a general dismissiveness about what I was presenting’, other witnesses ‘used assertions or misrepresented facts’ and presented findings that were ‘misleading [and] unsubstantiated’. There is a clear link here between these issues of reputability and those around affiliation presented earlier in the paper (“The affiliations of the witnesses” section).

Although interviewees expressed different perspectives about the reputability of various information sources, none mentioned which information sources and bodies citizens might trust the most. The witnesses were concerned that there were no repercussions for providing ‘bad evidence’ and felt that this was a limitation of the process. When asked what characteristics they felt constituted ‘good’ evidence, the witnesses we interviewed referred mostly to good communication rather than content, they also felt that sources should be ‘neutral’, ‘non-biased’ and ‘independently peer reviewed’. They felt that arguments should not be highly politicised, should highlight complexities and nuances of a given topic and should stand up to interrogation, and that the information presented should be clear and simple, or ‘boiled down’ to the necessary. These are all characteristics that are in line with the qualities that the jurors also desire in the evidence that they receive (Roberts and Escobar 2015), but ultimately these qualities are almost unattainable on a complex and nuanced issue such as onshore wind farm development (and topics suited to mini-publics more generally).

There were occasions in where poor-quality evidence was presented or unsupported claims were made during the wind farm jury witness testimonies. Three suggested in interview that the organisers should have screened the evidence prior to the juries. Our evidence does again indicate that the witnesses doubted the jurors’ ability to critically engage with evidence, particularly with contentious topics. This perceived gap is important because they also doubt the independence and quality of other witnesses.

Lansdell (2011) notes that slide-vetting could be a mechanism for increasing fairness for the witnesses, particularly for those with little experience of communicating in these sorts of forums, as well as for ensuring the witnesses fulfil their brief. Thus, not vetting the evidence beforehand might have advantaged some of the witnesses. That said, attempted controls on the witnesses’ evidence were not always well received, nor heeded (Roberts and Escobar 2015); the witnesses were requested to provide their slides in advance of the event, and none of them opted to do so. Notwithstanding this, organisers should be clear that part of the witnesses’ task is to present evidence that stands up to scrutiny. This is also important to support the validity of the citizen jury process itself, and its outcomes. The selection of quality evidence might be incentivised should the process be open for public scrutiny. For example, two projects in the meta-analysis were live-streamed, and the audio or video recordings of the witnesses’ testimonies were made publicly available after the events (see Table 2).

In the wind farm project, the jurors learnt a great deal from the witnesses. During the evidence-giving process, witnesses could engage with the jurors several times, and through different means (including by email, managed by the organisers), so the jurors could clarify points or pick-up on previously unexamined issues. The jurors unpacked the witnesses claims in the Q&A and referred to the witnesses evidence in their deliberations, where they also grappled to understand the underlying reasons for (differences in) witness arguments (Roberts and Escobar 2015). This conclusion aligns with deliberative theory—not only do lay participants learn from the experts, they are able to challenge them too (Brown 2014).

Deliberation in citizens’ juries provides the opportunity for both real and perceived bias to be considered and balanced by the diversity of the jury (NewDemocracy 2016b). However, jurors frequently suggest they would like more time to weigh up considerations and alternatives, and allowing sufficient time for discussion and debate is deemed to be important for quality deliberation (Flynn 2009). Indeed, though the jurors involved in the wind farm project were largely satisfied by the process, several felt that more time would have been useful.

The jury process that the wind farm project designed aimed to create conditions where emotional expressions could be combined with more reasoned deliberation and space for critical thinking. However, prior to hearing the witnesses’ testimonies, the participants did not have the opportunity to think together about how to critically unpack and interpret evidence. The research report recognised this as a limitation and recommended that an introductory session to support participants to critically interpret evidence (Fischer 2009; Roberts and Escobar 2015) in line with the format in (Fischer 2009). Further, (Fischer 2009; Roberts and Escobar 2015), also recommend that, at the end of each information session, jurors should be encouraged to share reflections on the evidence heard, acknowledge their reactions and further digest and rationalise the information.

A number of the deliberative processes reviewed included processes to support jurors to critically interpret evidence. For example, the New Democracy foundation (who organised the New South Wales Energy Enquiry, Infrastructure Victoria, and Get to Know Nuclear projects, Table 2) includes activities to improve the critical thinking skills of their participants (Carson 2017). Similarly, the first day of the CIR process included an exercise to help participants weigh up competing arguments and consider factual and values-based disagreements. Lansdell (2011) recommends the involvement of a ‘technical friend’, to enhance communication between the experts and participants, where the facilitator might lack the expertise to identify where language needs translating, or where technical issues are unclear. In these juries, during the deliberations, two ‘information officers’ (two of the organisers) answered questions on-demand (referring only to information in the Handbook or presented by the witnesses), and occasionally challenged ‘uncritical consensus’ to help the jurors articulate and justify arguments. It was felt that the information officers played a useful role in helping to deepen the jurors’ understanding and ability to tackle complexity (Roberts and Escobar 2015); however, this role carried risk, considering the potential power and influence they could have on the deliberations. These techniques combined can give additional resources to mini-public participants to help them scrutinise the evidence and opinions provided by the witnesses and each other.