A framework for estimating information security risk assessment method completeness

Information security (InfoSec) risk comes from applying technology to information [1], where the risks revolve around securing the confidentiality, integrity, and availability of information. InfoSec risk management (ISRM) is the process of managing these risks, to be more specific; the practice of continuously identifying, reviewing, treating, and monitoring risks to achieve risk acceptance, illustrated in Fig. 1. A baseline level of security can be achieved through compliance with current law and legislation, but best practice InfoSec is highly dependent on well-functioning ISRM processes [1], which requires a tailored program to suit the risk taking of the organization. Typically, risks for information systems are analyzed using a probabilistic risk analysis, where risk is a measure of the probability of occurrence of an event and the associated consequences for the organization (e.g., financial loss if a risk occurred).

InfoSec risk assessment (ISRA) practices vary between industries, disciplines, and even within the same organization, which has brought a variety of ISRA methods [2] and risk definitions [3]. This article covers the risk assessment process, including risk identification, estimation, and evaluation, and compares the completeness of eleven surveyed ISRA methods. For clarification, the main difference between risk assessment and analysis is, according to ISO/IEC 27000:2016 [4], that the latter does not include the risk evaluation.Further, we develop a framework for comparing ISRA methods on their completeness. We demonstrate the utility of the framework by applying it to a collection of risk assessment methods, identifying several limitations and weaknesses of existing risk assessment approaches, of which several were previously not well known. For example, besides the FAIR approach [5] there are few detailed approaches to obtaining quantitative estimates regarding the probability of occurrence. All of the surveyed methods include an approach for qualitatively describing risk impact, while only three of the eleven methods provide guidance on how to quantify loss estimates. According to our results, asset identification and evaluation are two of the most common risk identification activities. Although business processes are defined as one of two primary assets in ISO/IEC 27005:2011 (ISO27005) [6], very few methods include the business process in the asset identification. Further, our results show that risk concepts, such as opportunity cost, cloud risk, incentive calculations, and privacy risk estimations, are only present in topic-specific methods and have a low adaptation rate in the surveyed methods. Also, none of the studied methods discuss the Black Swan concept proposed by Taleb [7], or wholly adopted the qualitative knowledge metric of qualitative risk assessments as suggested by Aven and Renn [8].

Note that our comparison framework is restricted to ISRA and that we apply the framework to the risk analysis and evaluation part of the surveyed methods. Thus, a comparison of non-risk assessment elements from full risk management methods is outside of scope.

Using the terminology established by Campbell and Stamp [9], the extent of this work is primarily functional approaches [9], which are the formal ISRA methods that focus on assessments of threats and protections, often with measures of probability and consequence. As opposed to temporal approaches that tests components of actual attacks, such as penetration tests and red teams, while comparative methods compare systems to best practices and establish security baselines. We have not evaluated accompanying software tools for each method in the Core Unified Risk Framework (CURF). Some methods, such as FAIR and CRAMM [10], come with software that expands aspects of the approach, but these are outside of scope.

The remainder of the paper is organized as follows, and Sect. 2 provides general background information on the eleven surveyed ISRA methods. In Sect. 3, we present the design science research approach applied to develop CURF. Further, in Sect. 4, we implement the framework on popular ISRA methods and show the results. Further, we discuss the completeness of each surveyed method and limitations of current approaches in Sects. 5 and 6. Lastly, we establish the relationship to other literature in Sect. 7, discuss limitations and propose future work in Sect. 8, and conclude in Sect. 9.

We have reviewed nine well-documented ISRA methods which all have in common that they have been specifically developed to address InfoSec risk and are well-documented. Besides, CURF contains one review of both a privacy and a cloud risk assessment method. The following is a summary of the eleven methods:

CIRA is a risk assessment method developed primarily by Rajbhandari [11] and Snekkenes [12]. CIRA frames risk as conflicting incentives between stakeholders, such as information asymmetry situations and moral hazard situations. It focuses on the stakeholders, their actions, and perceived outcomes of these actions.

CORAS is a UML (Unified Modeling Language) model-based security risk analysis method developed for InfoSec [13, 14]. CORAS defines a UML language for security concepts such as threat, asset, vulnerability, and scenario, which is applied to model unwanted incidents and risks.

The CCTA Risk Analysis and Management Method (CRAMM v.5) is a qualitative ISRA method [10]. CRAMM centers on the establishment of objectives, assessment of risk, and identification and selection of countermeasures. The method is specifically built around the supporting tool with the same name and refers to descriptions provided in the repositories and databases present in the tool.

FAIR (Factor Analysis of Information Risks) is one of the few primarily quantitative ISRA approaches [5, 15]. FAIR provides a risk taxonomy that breaks risks down into twelve specific factors, where each factor contains four well-defined factors for the loss and probability calculations. FAIR includes ways to measure the different factors and to derive quantitative analysis results.

The Norwegian National Security Authority Risk and Vulnerability Assessment (NSMROS) [16] approach was designed for aiding organizations in their effort to become compliant with the Norwegian Security Act. NSMROS is written in Norwegian and provides a basic description of the risk management process and associated activities.

OCTAVE (Operationally Critical Threat, Asset, and Vulnerability Evaluation) Allegro methodology is the most recent method of the OCTAVE-family [17], aimed at being less extensive than the previous installments of OCTAVE. It is a lightweight version of the original OCTAVE and was designed as a streamlined process to facilitate risk assessments without the need for InfoSec experts and still produce robust results [17] (p. 4).

The ISO/IEC 27005:2011—Information technology, Security techniques, Information security risk management [6] (ISO27005) details the complete process of ISRM/RA, with activities, inputs, and outputs of each task. It centers on assets, threats, controls, vulnerabilities, consequences, and likelihood. Since we regard ISO27005 as the industry best practice, it also provides the frame for CURF.

The current installment of the NIST SP 800-30 - Guide for Conducting Risk Assessments is at revision one [18] (NIST80030) and was developed to further statutory responsibilities under the Federal Information Security Management Act. NIST800-30 was designed to aid larger and complex organizations in information risk management. The purpose of the publication was to produce a unified information security framework for the US federal government.

The ISACA (Information Systems Audit and Control Association) Risk IT Framework and Practitioner Guide [19, 20] is an ISRM/RA approach where the Practitioner Guide complements the Risk IT Framework. The former provides examples of how the concepts from the framework can be realized. It is an established approach developed by ISACA, based on ValIT and CobIT, and, therefore, has a business view on InfoSec risks, defining several risk areas and factors.

Privacy impact assessments are methods that identify and analyze specific risks to privacy in a system or a project. The Norwegian Data Protection Authority’s (Datatilsynet) risk assessment of information systems (RAIS) [21] are ISRA guidelines that primarily are designed to aid data handlers in their effort to become compliant with the Norwegian Data Protection and Privacy Act with corresponding regulations.

Outsourcing services to the cloud bring new third-party risks to the organization. Microsoft’s Cloud Risk Decision Framework [22] is a method designed for addressing this problem by risk assessing cloud environments. The method is derived from the ISO 31000 standard for Risk Management and provides a framework for working with cloud-associated risk.

The necessity of a bottom-up approach for comparing ISRA methods became apparent when we were studying cause and effect relationships between applying an ISRA method, the work process, and the resulting output. ISRA methods are often comprehensive and comparing tasks at a sufficient level of detail is challenging. There exist multiple frameworks for comparing ISRM/RA methods [2, 9, 23‐27]; however, these are primarily scoped to compare method content to a predetermined set of criteria. In these frameworks, evaluation proceeds from the predetermined criteria at the top to methods at the bottom. The existing approaches yield differences within the criteria and are equivalent of top-down static comparison. This approach is restrictive because the framework will overlook any tasks or parameters that the criteria do not cover. CURF’s bottom-up approach solves this problem by providing a way to review each ISRA method, structure its tasks within ISO27005’s risk management process, and use the complete task set as comparison criteria.

The framework idea is as follows: For each method, CURF users identify which tasks the approach covers and then combine all the tasks covered by all the surveyed methods into a combined set. For example, one method might propose to identify threats as a task, but another does not, so “threat identification” becomes a task under the risk identification group. Further, we unify all issues covered by each of the methods into a superset. An application of the framework to a risk assessment method amounts first to identify the issues covered by the method and then merging this set with the larger set of issues constructed previously. The completeness evaluation of a risk analysis method amounts to investigating the extent the said method covers all issues present in the superset constructed previously. The superset should provide the practitioner with insight into which aspects each method cover, together with an overview of where to seek knowledge in the literature to solve other specific issues or for comparison purposes.

Further, in the following sections, we describe the choice of method for framework development, specific CURF development issues, and inclusion and exclusion criteria for the ISRA methods.

In this section, we propose the Core Unified Risk Framework (CURF) for comparing issues in ISRA methods. Following the method outlined in Sect. 3, we surveyed each of the eleven methods described in Sect. 2 and created the CURF model. Figure 3 is a high-level representation of the results where the colored tasks indicate sub-activities which we describe in more detail in the subsequent section. Following, we outline each of CURF’s descriptive categories, identified process tasks, and sub-activities. The tasks are grouped and presented in three primary categories: (i) risk identification, (ii) estimation, and (iii) evaluation.

In this section, we evaluate the completeness of each surveyed method according to the identified activities based in the CURF, Fig. 3. The results in Tables 1, 2, and 3 form the basis for the discussion. Table 4 displays the total measure of ISRA method completeness together with the mean value. The overall most complete method is ISO27005 with FAIR scoring second highest in the risk estimation process. Following is a summary of differences between the surveyed methods and their completeness.

One of our most significant findings is that no method is complete in CURF, while the most complete is ISO27005 which addresses several issues in some way. However, falls short when compared to FAIR’s detailed risk assessment approach. The following discussion analyzes the scope of ISRA methods, then row scores, and, lastly, the presence of modern risk concepts in the surveyed ISRA methods, starting with the risk identification process, followed by an analysis of the risk estimation and evaluation processes. Lastly, we discuss modern risk concepts in ISRA. In Tables 1, 2, and 3, we also summed each row to show where the area of focus for ISRA developers lie. We apply the ranges 0–7 \(=\) low, 8–15 \(=\) medium, and 16–22 \(=\) high, to simplify discussion.

In this section, we discuss the previous work in the research field and how our work differs and extends previously published work. There are several comparison studies of ISRM/RA methods in the related work. We have previously referenced the Sandia Report [9] which presents a classification scheme where ISRM methods are sorted in a 3-by-3 matrix by the level of expertise required and type of approach. The Sandia classification complements our results by stipulating the level of skill needed to apply an ISRA method. The historical and recent development trends of the risk concept proposed by Aven [3] also complements this framework by providing the background and foundation of each risk approach.

There exist multiple comparative studies outlining ISRA approach content to aid organizations in choosing a method, for example ENISA’s high-level summary of existing methods [23] and methodology for evaluating usage and comparison of risk assessment and risk management items [24]. The latter is a well-developed approach for comparing and benchmarking possible ISRM processes, together with expected inputs and outputs. The benchmark follows the classic ISRM process (Fig. 1), including the six main ISRM stages and fifteen defined sub-process. There are several resemblances to CURF in the comparison method; for example, both have the ISO27005 as a starting point and apply a similar scoring system. However, they are also different as the ENISA method compares to a set of items that we interpret as best practices, while CURF compares with items present in methods and grows if the new item is added. The former ENISA comparison [23] is a high-level comparison of methods, based on four predefined categories for ISRM and ISRA, eight in total. While similarly, Syalim et al [27] have published a comparative analysis that applies four predefined generic steps of the ISRA process for comparison. Both these studies compare a set of ISRA methods within a predefined set of criteria. An approach that risks leaving important aspects out of the comparison. For example, both comparisons downplay the role of the asset identification and evaluation process, which often is the foundation of the risk assessment. The results in our paper differ from these in that ours are versatile and adaptable, allowing for other tasks and activities beyond predefined categories to be added and analyzed. In this context, Bornman and Labuschagne [25] present a very detailed framework for comparing the complete ISRM process, divided into five categories, where the processes category is interesting for our work. The authors built their comparison criteria on CobiT (Control Objectives for Information and Related Technology (COBIT) by ISACA). This framework focuses on what the compared methods address and contains about COBIT, but not differences in how they recommend solving the task, or the distinct differences between the approaches.

Another similar study was conducted by Shamala et al. [26] which defines a detailed information structure for ISRA methodology contents. This comparative framework was developed to evaluate ISRA methods primarily on the information structure regarding what is needed at a particular step in the assessment. The contents of the framework are derived from a detailed comparison of popular ISRM/RA methods and, therefore, have a similar approach to our work, but with a different purpose and scope, and, therefore, different results regarding criteria. Whereas Shamala et al. focus on how and what information to collect, our results look at how ISRA methods address particular tasks and issues.

Agrawal [40] has published a comparative study of ISRA methods, in which the author summarizes four methods using an ontology. The paper compares the four ISRA methods to eight predefined criteria, whereas it considers if a method is primarily qualitative or quantitative, purpose, and if it is scalable. Agrawal also describes the expected input, effort, and outcome of each process step and then discusses the pros and cons of each reviewed method. This study overlaps with CURF in some of the criteria, such as methodology, outcome, and the use of Sandia classification [9]. However, the main methodology and approach to the problem are different, as Agrawal also considers a set of predefined criteria for each method.

One of the most comprehensive taxonomies of ISRA regarding reviewed methods is the Shameli-Sendi et al. [2] study, in which the authors have reviewed 125 papers. The study provides a modern taxonomy of ISRA methods based on a set of four categories identified by the authors. The first category, appraisement, is defined as the type of input and output of the risk calculation, such as if it is qualitative, quantitative, or a combination of both (hybrid). The second category addresses the ISRA method’s perspective, which is either business, asset, or service driven. An additional category, threat driven [18], could also have been considered for the perspective category. The third category, resource valuation, primarily considers how the ISRA method suggests evaluating valuables: either asset, service, or business process, and if it considers functional dependencies between them. Whereas compromising one asset may inflict consequences on another, and such on. The fourth category is risk measurement in which the taxonomy classifies ISRA methods regarding how they consider impact propagation, meaning if the method advises considering an impact only to the asset itself (non-propagated) or if it considers dependency between assets and other resources. The Shameli-Sendi et al. taxonomy classifies an ISRA method within the predefined criteria identified by the authors, while CURF compares on criteria and tasks present in the methods. Both approaches aim to assist practitioners and organizations in the choice of ISRA approach, while Shameli-Sendi et al. are at a higher level of abstraction addressing four core issues in ISRA, and CURF provides in-depth analysis of how well each method addresses each task. Thus, these two approaches have complementary features.

On the topic of research problems, both Wangen and Snekkenes [36] and Fenz et al. [35] have published articles on current challenges in ISRM; the former is a literature review that categorizes research problems into a taxonomy. The latter discusses current challenges in ISRM, predefines a set of research challenges, and compares how the existing ISRM methods support them.

The related work contains several approaches to comparing method content. However, these are primarily studies of properties and content based on a predefined set of criteria. None of which address how to compare full ISRA processes and content beyond these criteria. Thus, the gap in the research literature lies in the lack of a bottom-up approach to compare ISRM/RA methods.

CURF has limitations in the abstraction layer as we chose to keep the comparison at a high level, the model does not display deeper differences between methods such as specific approaches to asset identification, vulnerability assessments, and risk estimation. For example, the new version of FAIR [5] comes with a detailed approach to risk estimation, while other methods that appear somewhat equal in the comparison, such as NSMROS [16], only describes the activity at a high abstraction which means that a closer study of the methods and a possible expansion of the tables will reveal deeper differences in scope and methodology not present in our work. This includes the time-parameter for PxI calculations suggested by one of the reviewers. A possible addition to the framework is to expand it with experience-based knowledge and to grow it by making it available to other scholars and practitioners. Comprehensiveness of activities and accessibility of the ISRA methods is not considered in this comparison, which are issues we uncovered for some of the frameworks. Another limitation is that, for example, ISO 27005:2011 scored low on the cloud-specific criteria, third-party management is covered in the supporting material (ISO/IEC 27001 and 27002 standards). This may also be true for other reviewed methods.

Although the included set adhered to our inclusion and exclusion criteria, there are several methods that we could have included in this study. We have a limitation regarding the amount of methods included in the study; however, after adding the highest scoring methods the framework started to saturate and mature. Wherein each additionally reviewed method added fewer tasks than the previous, for example, both RAIS and MCRDF were added last and each added two risk-specific tasks to CURF. A path for future work is to add new methods within specific tasks or philosophies to expand the framework and make it more representative for all ISRA methods. Among possible future additions to CURF are attack tree methods [41], the Information Security Risk Analysis Method (ISRAM) [42], and the IS risk analysis based on a business model [43].

However, growing CURF by including more methods will increase the complexity and make it harder for the reader to obtain an understanding of the whole picture. It can be a challenge for someone who is not security literate to understand all the CURF tasks and make informed choices on methods. Although we provide a brief description of each task and sub-task, it is a limitation that the CURF user must possess a certain amount ISRA knowledge to fully utilize CURF. A path for future work to address this issue is to develop software support for CURF for collaboration and online use, in which we provide more detailed method and task descriptions. This path would make both the method and results more accessible to the ISRA community, together with an option for users to add and qualitatively score methods. There are multiple possibilities with such an approach; for example, a comprehensive overview and description of existing tasks would allow the risk practitioner to choose the parts of the framework he needs to construct a risk assessment model tailored to his requirements. Another path for future work is to apply CURF to map differences between ISRA methods, apply the methods on case studies, and then to analyze the risk assessment outcomes. This approach will allow for a study of cause (task) and effect (outcome) and allow researchers to determine how specific tasks influence the outcomes. This approach has already been partly developed [38], but additional research is required to increase the knowledge base.

Another limitation of the CURF method is that the completeness score will be dependent on the methods chosen to populate CURF. The completeness score will reflect whether it supports similar functionality to the methods already reviewed. In cases where the user wishes to assess a new method with several innovative tasks, the approach will add many several tasks to CURF, but the method is likely to obtain a low completeness score. In such cases, static evaluation criteria might be preferable as a decision basis.

Further, this work highlights the need for a more research into what the lower levels of an ISRA should consist of beyond asset, threat, vulnerability, and control assessments. For example, what are the key estimators of an asset value or a threat assessment?

To conclude this paper, we have presented CURF, which was developed inductively through reviewing eleven well-documented ISRA methods: CIRA, CORAS, CRAMM, FAIR, NSM ROS, OCTAVE, ISO27005, NIST SP 800-30, and Risk IT, in addition to two domain-specific methods, one for cloud (CRDF) and one for privacy (RAIS). Literature studies show that there exist multiple comparative assessments of ISRM/RA methods, but these are all scoped to compare method contents to a predefined set of criteria, equivalent to a top-down approach. For most cases, this is less flexible concerning tasks missing in the predefined criteria. With CURF, we have shown the utility of comparing methods and building the framework from a bottom-up point of view. Our results, therefore, consists of a larger superset of issues and tasks from all reviewed ISRA methods using ISO/IEC 27005:2011 as a reference point and then comparing the ISRA methods as a measure of completeness covering all the issues and activities added to the superset. The possibility to add new problems makes our proposed framework highly flexible to changes in future methods and for comparing methods that are very different.

No evaluated method is complete in CURF, but from all of the methods reviewed, ISO/IEC 27005:2011 is the most complete and covers most issues in one way or another. However, FAIR had the most complete risk estimation process. Another finding is that besides FAIR, there is little information on how to quantify probabilities in reviewed methods. There are several ISRA frameworks and practices; however, we find variations of asset evaluation, threat, vulnerability, and control assessments at the core of the most reviewed frameworks, while the more specific issues, such as cloud risk assessment, are primarily addressed by methods developed for that purpose. It was also interesting to find that none of the ISRA methods discuss the presence of unknown unknowns (Black Swans), which is highly relevant due to the dynamic and rapid changes in ICT systems, which continue to grow and increase complexity. Besides CIRA, the human motivational element of InfoSec and ICT systems seems mostly neglected.