Visually Modelling Collaborative Research into Innovative Community Disaster Resilience Practice, Strategy, and Governance

Owen et al. (2013) outlined a solution for managing this kind of emergency management dilemma, where stakeholders with very different priorities would need to collaborate despite clear differences in viewpoint. It seemed vital to document a range of understandings, without insisting that every single aspect was shared between every single stakeholder. For Owen et al. (2013), documentation that served this purpose would become “boundary objects” (p. 1) for communicating diverse understandings in a way that spanned multiple divides between diverse collaborators.

Case studies by Huggins and Jones (2012) and Huggins and Peace (2014) looked at similarly complicated planning, monitoring, and evaluation contexts where similar communication issues were addressed through the use of visual logic models. Both of these cases involved planning, monitoring, and evaluating complex projects at a national scale. The latter case study focused on integrated project-management—to support implementation that was both well-informed and timely. Both studies highlighted the potential of a specialist software tool, DoView™, which is used for Visual Monitoring and Evaluation Planning (VMEP) developed by Duignan (2013). The kind of visual logic models produced in Huggins and Jones (2012) and Huggins and Peace (2014) represented a boundary object document in the context of the ICoE:CR. A simplified example of these visual logic models, created for the purposes of VMEP, is provided in Fig. 1. This document format was used in the current research to mobilize the third, more pragmatic, pattern of opinions identified in antecedent Q-method research as the need to evaluate opportunities to improve complex post-disaster outcomes at a range of societal levels. Using an action research methodology gave the ICoE:CR researchers and practitioners an opportunity to deepen their collective understanding of the planning, evaluation, and monitoring context for the resilience strategy as they participated in the research.

Observed strengths of the visual logic models used in VMEP were documented by Huggins and Jones (2012) and Huggins and Peace (2014). These strengths include the ability to visually communicate multiple levels of shared activities and objectives across diverse stakeholder groups, and to generate a dialogic approach to the production of indicators and evaluation questions. The outcomes theory background to VMEP outlines the concept of outcome hierarchies: “a cascading set of causes in the real world” (Duignan 2012a, p. 1). This is how VMEP suited the pragmatism shared amongst ICoE:CR researchers and practitioners, by visualizing multiple levels of shared actions and outcomes before connecting them to relevant indicators and evaluation questions.

The deployment of VMEP within the ICoE:CR also came informed by a growing body of cognitive research into how visual imagery complements other forms of communication. For example, a critical review of associated scientific literature by Tversky (2011) concluded that rich visual representations can improve understanding of action in space. Klingner et al. (2010) found that visual representations required fewer internal cognitive resources than a verbal equivalent when measured by pupil dilation during cognitive tasks. Visual representations also appear to help people engage with nonlinear concepts. For example, one experiment by Kessell and Tversky (2009) found that 60 % of participants habitually drew cycle dynamics, from part A to part B to part C to part A etcetera, along a single line. However 80 % of participants in the same experiment preferred to see these dynamics on a circular diagram.

The main objective of the current research was to detail how and why the ICoE:CR visual logic model transformed over time. Visual logic models had played particularly pragmatic roles in prior case studies by Huggins and Jones (2012) and Huggins and Peace (2014). We therefore hypothesized that the current visual model would serve a pragmatically useful purpose for workshop participants, and that this would be evidenced by their accounts of the VMEP process.

We also wanted to show how opinion factors identified through Q-method could endure over time. It is often assumed that social science analyses, such as Q-method factor analyses, are unreliable and represent little more than a moment in time, under very particular conditions. For emphatic examples of this viewpoint, see Faigman (1989) and Kampen and Tamás (2014). The current research assumed that statistically robust factors identified through Q-method research were relatively reliable and of enduring relevance to ICoE:CR development. Our hypothesis was that the distribution of interview content related to these opinion factors would closely resemble the distribution of original Q-method factors amongst ICoE:CR groups.

The data collected, developed, and analyzed in the current research were compiled in two phases. The first phase involved the development of the VMEP model, and the second phase involved in-depth interviews with participants about their engagement in the VMEP process. The development of the model is described first, followed by a brief description of data collected from interviews with VMEP participants.

VMEP uses a visual logic model to plan strategically relevant monitoring and evaluation activities. According to Duignan (2013), this is achieved by: (1) drawing an initial diagram of intended outcomes and the steps required to achieve those outcomes; (2) marking the relative priority of outcomes and steps and drawing causal linkages between them; (3) identifying key performance indicators that can help gauge performance towards intended outcomes; (4) developing evaluation projects; and (5) reporting evaluation results back onto the overall diagram and revising that diagram in response to those results.

Participants in the current action research were generally unfamiliar with evaluation frameworks, so the standard VMEP process outlined above was modified in four minor ways. First, although Duignan (2012b) had advised against defining distinctions between objectives and outcomes, we were unlikely to engage ICoE:CR coordinators without a clear overarching structure. This meant that all strategy components were neatly divided into activities, objectives, and outcomes. Second, although they are not usually highlighted so explicitly during a VMEP process, overarching ethical principles were written in text above the main visual logic model. This reflected a recommendation from Huggins et al. (2015), that a strong pragmatic impetus within the ICoE:CR needed to be complemented by explicit ethical principles. According to Huggins et al. (2015), a prominent set of ethical principles would help avoid using a range of disaster-related ends to justify any given means. Third, we chose to use the term “research question” rather than “evaluation question” due to many participants’ lack of familiarity with the latter concept. The former term was more intuitive for non-practitioner representatives in particular, who were all representing research institutions. Finally, VMEP step five was not included in the scope and timeframe of the current research because indicator data would take many more months to collect.

The first three VMEP steps were completed prior to the workshop, with the Manager, WREMO Community Resilience. This involved a series of meetings where, over a period of six months, an evolving visual logic model was used to represent what the WREMO Community Resilience team was doing, show why they were doing it, and demonstrate how they were tracking progress against internally established targets. It is important to note that framework components representing higher level impacts did not survive discussions to establish this initial series of VMEP steps. Engaging a range of stakeholders during these preliminary steps could have led to a different outcome. However the main focus of pre-workshop activities was to produce a representation of the WREMO community resilience program, that WREMO management would then release for wider discussions. The workshop proper was then attended by a combination of seven representatives of research institutions and seven WREMO practitioners, all of whom were selected and invited by ICoE:CR coordinators. The workshop began with a very brief introduction to the WREMO Community Resilience Strategy, delivered by the Manager, WREMO Community Resilience. To prepare workshop attendees for active VMEP engagement, they were introduced to three preliminary pieces of information: recently developed standards from the Information and Knowledge Management for Disaster Risk Reduction group (UNISDR 2013); a summary of results from the antecedent Q-method research; and an animated Prezi™ introduction to Visual Monitoring and Evaluation Planning.

Results from VMEP steps one to three were introduced to workshop participants, who then broke out into subgroups that combined research and practitioner stakeholders. Each subgroup was invited to complete VMEP step four by writing research questions onto relevant sections of their own visual logic model printout. All subgroups then reported back to the main group to outline their research questions. These questions were recorded in real time, and typed in red alongside relevant sections of the VMEP diagram. Each question was finalized after being discussed, and agreed upon by the group as a whole. The resulting visual logic model is shown in Fig. 2. This is a verbatim copy of the workshop outcome that is usually presented as an A3 minimum printout or on a projector screen.

Although Fig. 2 is difficult to read in the current format, readers may note that several research questions have been added in red, to a more elaborate version of the example shown in Fig. 1. Activities are listed in boxes to the left of Fig. 2. Priority activities are indicated by a traffic light at their top right corner and these boxes are linked to other boxes showing relevant objectives towards the right of the logic model. Existing performance indicators are shown by a small yellow ruler beneath the activity or objective components that they are meant to gauge. Potential research questions are shown in italic red type and questions that were already being addressed by existing research are shown in bold red type. Each of these research questions is displayed next to the section of the logic model to which it refers. A key and a directional arrow were added to the bottom of the original version, to help provide these explanations. The original logic model also included hyperlinks that led to further details for boxes and text with small triangles in their bottom right corner.

Following the production of this model in the workshop, ten out of 14 workshop participants agreed to give semistructured, 90 min interviews about what had occurred during the VMEP process. All workshop participants’ names were replaced with pseudonyms on interview transcripts, and all minor utterances were deleted unless they were essential for interpretation. Interviewees were offered the opportunity to delete any additional text and clarify any part of their interview before the following analysis.

Interview transcripts were subjected to systematic thematic analysis prior to a more statistically oriented content analysis. Relying on content analysis alone was likely to neglect richer details about “the natural everyday world of human group life” (Wilson 1985, p. 398). The thematic analysis employed was what Braun and Clarke (2006) called “theory-driven,” because our entire analysis was based upon preestablished themes derived from the antecedent Q-method research, in Huggins et al. (2015), preexisting case studies of using visual logic models by Huggins and Jones (2012) and Huggins and Peace (2014), and other literature outlined below. Besides being transparent about preconceived ideas, Marks and Yardley (2004) stated that this approach to thematic analysis allows researchers to seek contradictions to many preconceptions. A theory-driven analysis therefore made a good match with the process of reformulating critical realist hypotheses, as outlined by Porter (2000).

Systematic thematic analysis relies on the development of a coding frame in which key themes can be noted and annotated. Our initial coding frame reworked the three opinion factors from antecedent Q-method research. According to Wolf (2014), in-depth interviews extend insights that can be constrained by rushing into full, and relatively arbitrary, factor descriptions. With reference to original elements outlined in Huggins et al. (2015), the code of a Constructive Focus on WREMO Activities was derived from the opinion factor, of being “against insular, top-down decision making.” A second code, Scientist Leadership, reflected the Q factor of a “need for complicated analysis to inform strategic decisions.” Finally, the opinion factor “need to evaluate opportunities to improve complex post-disaster outcomes at a range of societal levels” gave rise to a concise code of Positive Resilience Outcomes.

Further initial codes were based on other research literature that has been largely outlined in Sect. 1 of the current article. These codes included: Professional Collaborations with reference to visual modelling case studies by Huggins and Jones (2012) and Huggins and Peace (2014); Business Marketing with reference to the role of financial rationale in the case study by Huggins and Jones (2012); Diversity with reference to diverse engagement in complex domains outlined by Rogers (2008); and Documentation to represent the boundary objects outlined by Owen et al. (2013). Relevant interview excerpts that did not fit the seven initial codes were sorted into an Other category. Like subcodes sitting beneath the initial theory-based code, codes making up the Other category were linked and spliced as thematic analysis progressed.

For the purposes of content analysis, the action research was divided into three phases: (1) before the WREMO Collaborative Research Design Workshop; (2) during the workshop; and (3) after the workshop. Determining the most salient codes for each particular phase of the action research helped keep the current analysis manageable and coherent. Analysis included as much of the original interview text as possible in order to avoid the dilemma outlined by Dey (1993), of arbitrarily excluding material that could disconfirm researchers’ preconceived assumptions. Excerpts used in the current analysis therefore incorporated over 95,000 of a total 130,514 words appearing on interview transcripts. Most of the remaining words made up the interview preamble and other interviewer utterances.

As outlined in Sect. 3 results, a Constructive Focus on WREMO Activities was distinctly related to accounts of what happened before the VMEP workshop (rho = 0.13, p < 0.01). As described above, this code related to an opinion factor from antecedent Q-method research: “against insular, top-down decision making.” The relevance of this main code to what happened before the workshop suggests that participants had been motivated to constructively engage with operational decisions being made at WREMO before the workshop began. Restricting the VMEP model towards the level of activities and objectives may have helped meet this interest once the workshop began. Content concerning Positive Resilience Outcomes (rho = 0.10, p < 0.01) was also used to account for what occurred prior to the workshop. The most prevalent subcodes for Positive Resilience Outcomes were “improved coping and response” (6 % of all phase excerpts); and “knowledge, skills, and assistance via network” (3 % of phase excerpts). This content was related to the prevalent opinion factor identified by antecedent opinion Q-method research: “need to evaluate opportunities to improve complex post-disaster outcomes at a range of societal levels.” It was not surprising to observe the ongoing relevance of this opinion factor. A strong drive within the ICoE:CR, to analyze and improve the WREMO Community Resilience Strategy was well documented and announced to many interested parties (JCDR 2014). Nonetheless, the ongoing relevance of this particularly pragmatic opinion suggests that the VMEP process was well matched with enduring motivations at the core of the ICoE:CR. VMEP was adopted to leverage strong pragmatic motivations, which were shared between two distinct groups of ICoE:CR stakeholders.

Diverse Groups and Individuals (rho = 0.07, p < 0.05) was the only main code that exclusively related to content about the lead up to the workshop. The other two main codes for this phase were also highly relevant to during and after the workshop, respectively. Participants suggested that there was a need for a more diverse array of participants at the core of ICOE:CR activities. This included suggestions for representatives from a greater number of research institutions, other agencies working with community resilience, and community members from a diverse range of cultural backgrounds. Highlighting the importance of cultural considerations in the lead up to the workshop, one researcher's interview stated: “…the idea of having to look out for people after a disaster, I don’t understand all those details but that cultural perspective…” (Alana, lines 165–167). Several participants also detailed a need to adapt the WREMO strategy for diverse groups and individuals. We assumed that these participants’ accounts align with Parkinson (2009), who has described how socially orientated programs need to systematically consider diverse stakeholder needs and viewpoints.

Although diversity was a popular topic overall, researchers and practitioners did not appear to share the same understanding. Practitioners seemed reluctant to discuss culture as such, for example one practitioner’s interview stated:

We’re prioritizing partnerships because partnerships happen between everybody…and we’re prioritizing participation because everyone needs to participate…and protection because everyone needs to be looked after so it’s not, it’s not because of race or previous history, it’s because people are people and so all of our principles effectively cover all people. (Bridget, lines 1126–1130)

Other interview content regarding diversity appeared to discount the value of research. An apparent focus on variability and exceptions to theoretical rules could have been constraining researchers and participants and any generalized theoretical models of resilience that they had to offer. In this way, highly prevalent subcodes, such as “considering diverse individuals” (20 % of phase excerpts) and “considering diverse places” (11 % of phase excerpts), may have represented an obstacle to generalizations from a large body of preexisting community resilience research (see for example, Birkmann et al. 2012). The subcode, “considering diverse individuals” (20 % of phase excerpts) can also be considered alongside the subcode “egalitarianism” (3 % of phase excerpts). Workshop participants may have focused on reducing the status that researcher views often receive. This egalitarian approach to diminishing researcher views may have also meant down-grading, and eventually eliminating, researchers’ focus on predictive models and downstream outcomes.

There was subsequently very little discussion about downstream resilience impacts identified in surrounding academic research, such as Burton (2012). No workshop participant asked to detail downstream impacts during the workshop, even though they went on to receive an introduction to VMEP that outlined the way these components are usually represented. Likewise, no participant voiced concerns about the way that program benefits appeared unrelated to surrounding research, until this issue was specifically probed during interviews.

Participants’ comments about the workshop itself were centered around efforts to work with a visual logic model without any research questions and convert it into the version shown in Fig. 2. Positive Resilience Outcomes (rho = 0.13, p < 0.01) was the only main code that was significantly relevant to what happened during the VMEP workshop. The way that participants used content about Positive Resilience Outcomes to explain what happened before and during the workshop suggests that they found the VMEP workshop worthwhile. The VMEP workshop appears to have helped maintain the pragmatic impetus identified in antecedent research, concerning a “need to evaluate opportunities to improve complex post-disaster outcomes at a range of societal levels.”

Prevalent subcodes for Positive Resilience Outcomes concerning the actual workshop included “improved coping and response” (4 % of phase excerpts) and “knowledge, skills and assistance via a network” (4 % of phase excerpts). Content related to these subcodes reflected a very optimistic perspective of community disaster resilience in Wellington, which is exemplified by the following quote from a participant interview:

Yeah so if there’s lots of people out there with our It’s Easy books and stuff and talking to their neighbors …. It’s like I can talk to 10 people and say, talk to…the importance of talking to networks and things, and they go to their networks and that’s great. (Kirsty, lines 457–460)

This is also how the relevance of Positive Resilience Outcomes marks a pitfall in using VMEP to support emergency management. The focus on positive, and fairly tautological, outcomes serves as a reminder of how the VMEP process does not explicitly address unintended consequences. The VMEP process itself could be seen as optimistic, because it does not involve searching for negative implications. Many of these consequences fall outside of an initial strategic planning schema, that is, the outcomes specified on a visual logic model could nonetheless be very negative. For example, the popular uptake of preparedness kits may lead to overconfidence, resulting in a lack of interest in other aspects of preparedness and mitigation activities. VMEP for emergency management may therefore need to include a more deliberate and wide-ranging search for changes in a hazard affected environment.

Workshop participants were also asked about what was going to happen following the VMEP workshop. The analysis of participant accounts highlighted two main codes. The first of these main codes was a focus on Documents (rho = 0.12, p < 0.01), including the prevalent subcodes “different formats for different purposes” (12 % of phase excerpts) and “VMEP framework as a catalyst” (11 % of phase excerpts). The prevalence of “VMEP framework as a catalyst” shows how the VMEP diagram had represented a catalyst for dialogue, but was not necessarily a document for wider circulation.

Further subcodes for Documents included a “need for written text” and “different formats for different purposes.” These subcodes highlight a stated need to document workshop outcomes in a more traditional format. One participant’s interview made the relevance of both these subcodes particularly clear: “But I imagine that these will be turned into a document. And then they are more publicly, widely available…” (Alanna, lines 973–974). Here the word “document” refers to a traditional text-based document, like the original WREMO Community Resilience Strategy, which was released as a mainly text-based narrative document. This focus on an established text-based format therefore provides a context for the second most relevant theme: ‘Constructive Focus on WREMO Strategy’ (rho = 0.08, p < 0.01). Subcodes detailed in Table 1 show how many participants thought that the strategy included several assumptions, which remained to be tested and improved. This did not mean that participants supported a shift to working from another sort of document all together—hence the need to edit and redistribute the existing, text-based, version of the strategy.

As discussed in Sect. 1.1, Owen et al. (2013) outlined the need for boundary objects that link multiple organizational systems, beyond members of a single emergency management team. With this in mind, the integration of several preexisting organizations with the ICoE:CR may depend on common and largely text-based narrative documents. Text-based documents represent a strong status quo for formalizing agreements between organizations and other collaborators. Organizational change often fails when trying to eradicate all established norms and impose completely new practices (Clegg and Walsh 2004). This rationale for organizational development comes supported by blatant requests for a text-based outcome, from workshop participants and ICoE:CR coordinators alike. The visual logic model was therefore only likely to be formalized when it had been converted to a more traditional document format, such as Microsoft Office Word or Microsoft Excel formats.

This does not mean that narrative documents function well as a catalyst for creative and responsive thinking about complex dynamics. Diagram-based approaches can provide an important avenue to help us think about, and plan for, complex systems (Tversky 2011; Kessell and Tversky 2009; Huggins and Jones 2012). However it is equally important to note how text-based narratives can provide particularly in-depth descriptions. These descriptions clarify details that are relatively opaque in other forms of communication (Tversky 2011). If we were, for example, to compare the 24 pages of the WREMO (2012) Community Resilience Strategy with the Fig. 2 diagram, we would note that the diagram is brief and can be read in much less time. It also lacks a great deal of the clarifying detail available in the text document. While it has been important to develop the VMEP visual logic model (Fig. 2) as a critical engagement and process tool, this approach continues to require a supplementary, narrative explanation—especially for new collaborating parties.

The current research also aimed to test whether the distribution of Q-method opinion factors shifted during the current action research, where “distribution” refers to distribution between researcher and practitioner groups, and across interview content as a whole. As outlined in Sect. 3, the distribution of relevant interview content closely resembled the original distribution of two of the three Q-method opinion factors. In contrast, a Constructive Focus on WREMO Activities was now observed across both researcher and practitioner groups. This code was equivalent to Q-method opinion factor 1, which had not previously been consistently observed amongst ICoE:CR researchers as a group. Implications of this changing pattern of opinions are outlined in Sect. 5.

VMEP outputs were eventually converted to a large 10 page spreadsheet to suit requests from participants and ICoE:CR coordinators. This spreadsheet detailed draft research questions from the visual logic model by different categories: theme; methods; other details; relevant sections of WREMO strategy; existing researchers; and a column for WREMO to show the likelihood of uptake. At the time of writing, this spreadsheet had already been requested and used by researchers wanting to work with WREMO as part of the ICoE:CR. As stated by workshop participants, in terms of “VMEP as a catalyst,” this spreadsheet would not exist if it were not for the VMEP diagram underpinning it. Likewise, the spreadsheet is still being displayed to potential ICoE:CR researchers alongside the original VMEP diagram. This approach to using multiple formats has been directly informed by the current analysis, and by follow up discussions with ICoE:CR coordinators.

A clear limitation of using visual logic models for emergency management has appeared throughout the current VMEP process. As detailed in Sect. 2.1, early stages of the VMEP process at the ICoE:CR dictated that the visual logic model layout would be simplified for sign off by busy WREMO management. Although there had been some convincing reasons to avoid a more richly layered model, the lack of higher level outcomes has now reinforced by participants’ overall focus on the operational level of WREMO activities. The resulting visual logic model and surrounding VMEP process has neglected a range of surrounding research into resilience predictions, and requires expansion as part of longer term considerations for the WREMO community resilience program.

Roorda and Nunns (2009) suggested that logic model diagrams such as visual logic models can struggle to integrate surrounding research in the same way as systems diagrams. It seems reasonable to suggest that other processes, such as researcher-led systems modelling, may help further develop collaborative initiatives such as the Wellington ICoE:CR. For the ICoE:CR, this may substantially improve on the rapid systems model developed by WREMO practitioners and outlined in WREMO (2012).

These issues reflect neglect of research-based considerations and a failure to observe unintended consequences, outlined in Sect. 4.2, which are naturally due to more than document format alone. Process itself has a very important role to play. VMEP for emergency management needs to incorporate information from much further afield than internally established key performance indicators. Relevant changes to the surrounding environment, including unintended consequences, could be just as important to respond to as the success of particular interventions. An appropriately wider search could even reflect a strategic intelligence cycle that incorporates planning and tasking, data collection, processing/exploitation, analysis, production, dissemination, and user requirements and feedback alongside internal evaluation (Krizan 1999).

Attempts to integrate planning, monitoring, evaluation, and research may benefit from some form of visual logic model at many points of development. According to Huggins and Jones (2012), systems models and logic models represent a heuristic, abbreviated understanding of the world around us and are best assessed in pragmatic terms. We must ask ourselves whether the boundary object in front of us will really help us achieve what we need it to, while asking what other tools we need to deploy. To do otherwise would distance organizational development from what Huggins and Jones (2012) and Todd and Gigerenzer (2003) have referred to as “ecological rationality,” where selective and efficient understandings help make genuinely useful decisions, in particular situations. Further research into planning, monitoring, evaluating, and/or researching community disaster resilience programs will benefit from considering this established criterion.

At the time of writing concluding the current research, the new Sendai Framework for Disaster Risk Reduction 2015–2030 (UNISDR 2015) has been adopted and published as a largely linear, narrative document. Formats such as visual logic models have the potential to complement such an extended narrative document as a catalyst for relevant monitoring, evaluation, and research strategies. For example, visual logic models can use a “drill down” function to break an international scale down into regional, national, and subnational levels. This kind of rich visual and layered approach could help emergency managers meet contemporary community resilience criteria such as “collaboration and integration” and “issue and place-specific responses” (Local Government New Zealand, 2014, p. 1).

In light of the current research, processes such as VMEP and documents such as visual logic models are not simply a replacement for traditional narrative documents. The complementary role of VMEP for complex policy processes is a constructive finding, not a criticism. Experts in constructing extended linear narratives have no need to feel threatened by some new resistance to the clarity of official text-based agreements. It has become difficult to deny that VMEP represents a very different way of facilitating and documenting complicated dialogue between diverse disaster risk reduction stakeholders. But the current research also highlights the relative ease of moving into this mode of richly visual, boundary objects in dialogic, workshop contexts—before moving back to more traditional, text-based formats.

This movement between formats should certainly be considered for documenting many aspects of large-scale policies and strategies such as the Sendai Framework for Disaster Risk Reduction. Given the operational focus observed during the current research, VMEP may be particularly valuable in the implementation of such large-scale policies and strategies. Failure to consider implementation-focused innovations, such as VMEP integrated project-management proposed by Huggins and Peace (2014), could mean doing what we have always done to get what we have always got. As paraphrased from a United Nations Office for Disaster Risk Reduction review of the prior UNISDR disaster risk reduction framework (UNISDR 2014) the Hyogo Framework for Action 2005–2015, doing what we have always done to monitor and utilize core indicators is simply not good enough. As illustrated by the current research, VMEP represents a constructive challenge to this status quo. This process combines the expertise of practitioners and researchers towards improving the research-informed implementation of contemporary disaster risk reduction initiatives.