On the Meaning of Impact in Disaster Risk Reduction

For obvious reasons, disaster risk reduction is a field of imperatives. Therefore, theory has a special role, in that it needs to be immediately applicable and capable of generating positive results in the short term. Lives are at risk from disasters and damage needs to be avoided. Theory is the road map by which we make sense of complex situations such as those that arise in disasters so that we can reduce their human and environmental consequences. This is a powerful argument in favor of trying to ensure that research on disasters has recognizable positive impact. However, public debate does not always set priorities according to the most logically and morally justifiable set of criteria (Quintavalla and Heine 2019). This may mean that there is sometimes an element of illusion in what appears to be high impact.

We begin with some observations on the impact of major discoveries in history and the conditions of acceptance when they were made.

History is replete with luminaries whose work had little or no impact on society or the advancement of humanity during their own lifetimes. In Europe through much of the recorded past, enforcement of the status quo posed strict limitations on free thought and its expression. Very often, heretics were at risk of their lives because unorthodox thinking was deemed to threaten the social order, or at least the established power structures. Perhaps it still does. The response was usually swift and devastating (Shrady 2009). Perhaps it still is in many parts of the world. In history, there was little tolerance of “disruptive thinking” (Vollmer 2013), and where it did catch on, it led to prolonged and bitter armed conflict (Wilson 2011).

One example stands out. In the Renaissance, paper was a scarce and expensive commodity. Leonardo da Vinci was known to have covered at least 5000 sheets with sketches, drawings, plans, and notes (Pedretti 2006). Half of these documents survive after being kept for centuries in cupboards and on shelves, preserved lackadaisically for their curiosity value and the quality of his sketches, but shorn of impact. It was not until 380 years after his death that Leonardo began to be recognized widely as much more than a painter. As the child of a middle-class accountant, moreover born out of wedlock, he had no access to a Classical education, although in adulthood he taught himself Latin and Greek. With his mind untrammelled by the prevailing dogma, he was able to apply his particular brand of superbly rational, creative thinking to problems of nature, mechanics, and anatomy. He never finished the books he intended to write (on the behavior of water, on the human body, and on the scientific basis of painting), but since his work has been rediscovered he has remained a constant source of study, inspiration, and citation (Kemp 2007).

It is often stated that Leonardo, who was left-handed but largely ambidextrous, wrote in mirror image to disguise his thoughts from zealots who would have taken them as evidence of punishable heresy. It is now considered more likely that he did so as a form of relaxation of his writing hand (Affatigato 2019), but there is no doubt that ill-considered originality could lead an author to imprisonment, trial by torture, or execution (Nigg 1990), and in his early manhood Leonardo narrowly escaped such a fate.

In the present age of science, we deem ourselves to live in more enlightened circumstances. This is perhaps true, although not in all human societies in the twenty-first century. It brings with it a need to avoid what the historian Gordon Herries Davies and others termed “historical whiggishness” (Herries Davies 1989; Mayr 1990). This is the tendency to assume that all developments in history lead up to the current triumph of enlightenment over ignorance and obscurity. In reality, human development is part of a progression in which the achievements of the current age will sooner or later appear to be as paltry and inadequate as their predecessors. Moreover, such is the variety of people’s perceptions and opinions, that progress is marred by constant setbacks. The struggle to induce people to accept the concept of human-induced climate change should remind us of that (Whitmarsh 2011). In disaster risk reduction (a field that has borne that name for hardly more than two decades), there is still an on-going struggle to induce people to see disasters as generated by human actions and decisions, despite emphatic statements decades ago about the role of vulnerability in causing disaster (Hewitt 1983).

We are like dwarfs on the shoulders of giants, so that we can see more than they, and things at a greater distance, not by virtue of any sharpness of sight on our part, or any physical distinction, but because we are carried high and raised up by their giant size. (Bernard of Chartres 1159—see Stock 1979)

Nowadays, we live in the age of impact. Moreover, it usually has to be instant impact. The reasons for this are three. First, those who fund scientific endeavor want tangible evidence that they are receiving value for money. Secondly, there is an unwritten compact between scholars and the general public that their work should be beneficial to society. Moreover, it has to be seen to be beneficial. Thirdly, in a competitive age there is a desire to measure performance. Gauging impact is regarded as one of the ways in which this can be done for researchers and scholars.

Although the English word “impact” stems from the Latin impactus, a variant of impingere (to impinge), its use in a figurative sense apparently began only in the nineteenth century (OED). Thus it is a rather young term in the Anglophone lexicon. In the context of this discussion, “impact” can be given the following (rather arbitrary) working definition. It is “the presence of improvements that can be measured or otherwise assessed as a direct result of the publication or sharing of one or more pieces of work.” This begs several questions. What defines improvement? Can impact be measured or even simply gauged? When in the cycle of production and consumption of knowledge does it have to occur? How can we prove that there is a causal relationship between the work and its apparent impact? Finally, does impact have to be positive (that is, a net gain in a beneficial quality, such as peace, prosperity, safety, or harmony) or can it be appreciated if instead it is negative? In many cases, the answers to these questions are difficult to establish.

In disaster risk reduction the impact of academic work is often particularly difficult to gauge because of the complexity of relations between the many components of the systems involved. Not only are cascading, concurrent, and compound risks and impacts involved, but unintended consequences abound. For example, there has been a major global effort (albeit a rather heterogeneous one) to apply science and technology to disaster risk reduction (Aitsi-Selmi et al. 2016). However, as Quarantelli (1997) noted, technology leads a double life: that which its makers intended, and that which they did not. In particular, the fashion for optimization algorithms (Boonmee et al. 2017) is particularly difficult to assess in terms of impact. In most disaster situations it is doubtful whether anything can be optimized.

Remaining in the context of the present discussion, impact can take several forms. Alternatively, it can respond to several means of classification:

This further complicates assessment and measurement.

If a significant change is observed in disaster mitigation, preparedness, response, or recovery, and if this can be directly and unequivocally attributed to a particular study, then the work doubtless has had impact. However, there are distinct problems with assuming such a relationship. Causality may be very difficult to establish. Many other factors may create change such that isolating one cause is counterproductive or even illusory. Moreover, we may be quite unable to measure delayed impact.

Such is the weight given to “impact” in the modern world of science and academia that it is often automatically gauged by, for example, bibliometric measures. This offers us the paradox of seeing the value of science measured by essentially unscientific means (Casadevall and Fang 2014).

Popularity is not impact, although it is often mistaken for such. There is no robust reason for saying that, in scholarly circles, that which is most popular is that which is most important. In science, there are fashions and fads, as there are in any other walk of life (Abrahamson 2009). Particular forms of language or jargon, particular methodologies and styles of analysis, and particular ways of looking at a problem, all contribute to legitimizing studies by making them appear up-to-date and in line with present tendencies (Woodcock 2014). Rather than enhancing their impact, this often has the opposite effect. For example, the popularity of statistical methods such as the analytical hierarchy process (AHP) may be misleading as a measure of its true utility. No doubt when properly used, this technique is a powerful tool for resolving complexity. However, it suffers from the weaknesses of all inductive methods: first it is dependent on the assumptions that underpin it, secondly it can be used uncritically without profound thought about its implications, and thirdly it may simply be based on the wrong criteria (Schenkerman 2007). Despite this, it remains extremely popular in disaster risk reduction as a problem-solving tool. One might therefore argue that AHP studies have impact simply by means of their proliferation and popularity, and only sometimes because of their superior ability to solve problems.

As noted above, the history of science is rich in discoveries that in their time were unpopular, unrecognized, or treated with great scepticism but that turned out to be fundamental once they had found their proper milieu. This transition is sometimes termed “mainstreaming,” and it is not quite clear whether it is a process that can be engineered or whether it must necessarily be left to occur naturally. It is a function of context. Thus, disaster risk reduction can only be “mainstreamed” (that is, propelled into the mainstream of discourse, dialogue, and decision making) when the context allows, particularly the political context. Nonetheless, the procedures and tools exist for the change to be made (Benson et al. 2007; ADPC/USAID 2010).

As Bernard of Chartres observed in AD1159 (Stock 1979), all learned discovery is derivative, or better still cumulative. This either modulates or mitigates the impact of most discoveries, but it also creates the conditions for impact to occur, as a discovery requires a certain set of conditions for it to be accepted by potential users or beneficiaries. The question hinges on how we define the utility of scientific work (Sikorav 1991). Once a concept has been discovered and presented to potential users, it is mainstreamed when it has been put to use widely enough for it to be recognized as a valuable or essential tool.

Mainstreaming occurs both outside and within science. In the former case this usually involves impact upon public policy; in the latter, it can create a paradigm, as more and more scientists and scholars follow the lead. Nevertheless, decades of debate on Thomas Kuhn’s hypothesis of scientific revolutions indicate that the concept of “paradigm,” though convincing, is by no means as universal as one might think, or in other words that sciences tend to be more pluralistic than Kuhnian disciples might believe (Bird 2012). For example, if resilience can be regarded as a paradigm in disaster risk reduction (Sudmeier-Rieux 2014), there are plenty of researchers who do not think it deserves that status (for example, Keating and Hanger-Kopp 2020). It is therefore likely to be very rare that the impact of a piece of work drives all before it.

In 1996 the American political scientist Samuel P. Huntington published his book The Clash of Civilizations and Remaking of the World Order, which was based on ideas he first put forward in a lecture four years earlier (Huntington 1996). He argued that ideology would no longer be the principal source of conflict, but instead the world would retreat into rival cultural groupings. His assessment of what these are was very broad and involved nine major domains at the world scale. Huntington’s theory has been very popular in right-wing political circles where there is a growing sense of “culture warfare.” However, the book has been intensively criticized by scholars and commentators from many different backgrounds. In brief, it is regarded as a misleading oversimplification of the trajectory of world affairs. Critiques and alternative models have proliferated, including a particularly dismissive one by the Nobel Prize winning economist Amartya Sen (Sen 1999).

Similar criticisms could be levelled against Francis Fukuyama’s 1992 book The End of History and the Last Man (Fukuyama 1992), which argued that the end of the Cold War led to the termination of humanity’s ideological evolution. Once again there was a large groundswell of criticism. The French Philosopher Jacques Derrida went so far as to label Fukuyama as devious (Derrida 1994).

As it has been widely discussed all over the world, Huntington’s work has undoubtedly had enormous impact. On balance, much of that appears to be negative in the sense that influential thinkers have mobilized in large numbers to contest his thesis. Should we therefore regard his work as of high or low value? One might argue that he has muddied the waters of international political science by advancing an untenable thesis. On the other hand he has stimulated debate and that has led to alternative theories of human social evolution.

In disaster risk reduction, the concept of social capital has been widely embraced (Aldrich 2012) despite considerable reservations about its robustness and utility (Haynes 2009). Again, this begs the question of whether value must lie entirely in positive developments or to what extent honest appraisal of negative elements can also be treated as a positive result.

There is no clear answer to the question of whether negative impact should be respected, but as the novelist du Maurier (1981) wrote, “Living as we do in an age of noise and bluster, success is now measured accordingly. We must all be seen, and heard, and on the air.” Or as Tacitus (2009) remarked in his The Histories, “Etiam sapientibus cupido gloriæ novissima exuitur” (“Even for the learned, love of fame is the last thing to be given up”). On that resignedly philosophical note I shall leave the question hanging.

One further question remains to be given a brief critical examination. If the impact of a study can be demonstrated, who does it benefit? In environmental matters there is a “cure-to-damage ratio” (Burton 2018). In the simplest terms, this means that the amount of money or size of resources invested in solving a problem can be measured against the amount spent on making it worse. Hence, although the world may be concentrating increasingly on “decarbonization” in the hope of containing anthropogenic climate change, it is still true that orders of magnitude more funds are invested in fossil fuel exploration, extraction, and refinement than on promoting alternative, “greener” sources of energy (Laville 2021). Indeed, the cure to damage ratio is reckoned to be about 1:1000. Likewise, in disaster risk reduction, more is spent on creating disaster risk than reducing it, as can be seen, for instance, in unwise construction and land use (Lewis and Kelman 2012). Marie Aronsson-Storrier (2020) argued that, while international agreements such as the Sendai Framework for Disaster Risk Reduction 2015−2030 (UNISDR 2015) may increase the incentive to reduce disaster risk, international investment law acts to undermine such efforts.

As a general observation, research must first benefit the organization that funded it. This has, of course, led to numerous conflicts of interest in the medical and pharmaceutical fields when profit-making companies have funded research on their products (Lo and Field 2009). The same is true for the fossil fuels industries. Nevertheless, it would be unfair to condemn research simply because it fulfils the needs of its sponsor, as this is not inherently or necessarily a violation of academic integrity or scientific objectivity.

The need to present, assess, and measure impact is a function of the desire to make scientific and scholarly endeavor efficient, as well as directly responsive to people’s needs. Many works now need to be preceded by an “impact statement,” which is ironic as the impact, if any, will come after publication and any attempt to prejudge it may be highly presumptuous. At the very least, much of it is likely to be wishful thinking. No doubt this desire is laudable, but it has been overemphasized. As noted above, impact occurs along a spectrum, or rather a series of intersecting spectra, for it is a multi-dimensional phenomenon.

Is a contribution to knowledge a form of impact in its own right? The promoters of impact statements would argue that it is not unless the contribution has been recognized and in some positive way acted upon. Others might be more cautious, given the propensity for discoveries to lie dormant until conditions are right to activate them.

To have written a critique almost automatically puts one in the position of being expected to provide an alternative to what has been criticized. Here, the simplest approach would be to stop trying to measure and judge the impact of research studies. It is highly debatable as to whether this would lead authors and scientists astray, as the desire to “do something useful” is pretty much universal. Perhaps there would be a slight lessening in the desire “to be seen to be doing something useful.” This would be a positive development if it gave researchers more freedom to concentrate on what they personally regard as important and what they think they are best at doing. It might lessen the influence of fads and fashions, which would probably be a positive development, but there is no guarantee that this would happen. However, it would not help abate the clamor for value for money in research.

If we must continue trying to measure impact, I recommend that the following points be taken into account.

Regarding point (3), no increase in impact has been greater or more dramatic than that of years of patient work on viral disease pandemics. The year 2020 marked a turning point in attention paid to this problem, funding streams devoted to it, public debate on pandemics and viruses, and much more (Horton 2021). There are still neglected areas where impact should be greater. One of these is emergency planning (Alexander 2020). Regarding the last point, the study of existential risk (Currie and Ó hÉigeartaigh 2018) is an example of work whose impact is difficult to assess. Nonetheless, there are circumstances in which it could suddenly become central to public discourse and official decision making.

In conclusion, on a personal level, I look back over a career spanning the better part of half a century and I find it extremely difficult to assess whether my efforts in the field of disaster risk reduction have “had impact.” I know from correspondence that a book I wrote on emergency planning was much appreciated by a few people in Latvia and the Shetland Islands, but I have no idea of its influence more generally. It is, of course, even harder to assess whether one’s efforts will have any impact in the future. At the least we should continue to play devil’s advocate and ask whether the emphasis on the impact of scholarly work is a mere distraction or whether it does indeed play a useful role.