Comparison of the FMEA and STPA safety analysis methods–a case study

The increasing dependence of our society on IT systems brings not only new development opportunities but also new severe risks and threats. As our daily life is almost completely dependent on IT systems, both for individuals and for organizations (private and public), failures of these IT systems can have serious negative consequences and effects on the society. In-depth and a completely performed risk and hazard analyses help to minimize the potential harm of IT system failures on the society (Leveson 2012; Sulaman et al. 2013). However, risk/hazard analyses of modern socio-technical systems are far from trivial, mainly due to the dynamic behavior that pervades almost every modern software-intensive system and a high number of interacting components. As a result, many traditional low-level risk or hazard analysis methods fail to encompass the dynamic behavior of the systems, as they focus solely on the system component failures (Leveson 2012). These traditional methods mainly focus on identification of critical components of a system and then either try to prevent the failures of these components or add redundant components. In case of dynamically changing systems, a new risk can emerge from wrong or non-synchronized commands that may lead to severe accidents. Therefore, new methods for performing risk and hazard analysis, optimized for dynamic systems, are required.

There are still a number of uncertainties when it comes to what risk and hazard analysis method to apply in a given situation. The main objective of this study is to empirically compare two existing risk analysis methods, failure mode and effect analysis (FMEA) and system theoretic process analysis (STPA). The study compares the results of and investigates the effectiveness of the well-established bottom-up FMEA and the rather new top-down STPA hazard analysis methods by performing a comparison of how a hazard analysis is conducted for the same system.

FMEA is a bottom-up analysis method that is used to identify potential failure modes with the causes for all the parts in system to find negative effects (I.E.C. 60812:2006 2006; MIL-STD-1629A 1980). The analysis starts with the lowest level components and proceeds up to the failure effect of the overall system. The main purpose of FMEA is to identify potential problems in the early design process of a system or product that can affect its safety and performance, and to introduce countermeasures to mitigate or minimize the effects of the identified potential problems (failure modes). On the other hand, STPA is a top-down method, just like the fault tree analysis (FTA) method. However, STPA uses a model of the system that consists of a functional control diagram instead of a physical component diagram used by traditional hazard analysis methods. It focuses on analyzing the dynamic behavior of the systems and is intended to provide advantages over traditional hazard analysis methods (Leveson et al. 2012). STPA is based on system theory unlike FMEA, which is based on reliability theory as explained in Sections 2.1 and 2.2. Moreover, STPA considers safety as a system’s control (constraint) problem rather than a component failure problem.

The results of the FMEA analysis yielded from this study are compared with the results of a previous study (Sulaman et al. 2014) that presents an STPA hazard analysis of a system. It should be noted that this study does not aim at comparing both methods quantitatively, but instead to understand the differences through a qualitative analysis. That is, we investigate both methods qualitatively by analyzing hazard analysis results gathered by applying both methods, FMEA and STPA, on a collision avoidance system. Furthermore, this study also evaluates the analysis process of both the methods by using a set of qualitative criteria, derived from the technology acceptance model (TAM) (Davis 1989; Davis et al. 1989).

The remainder of this paper is structured as follows. Section 2 provides a background on FMEA, STPA, and other risk and hazard analysis methods, as well as an overview of the forward collision avoidance system which is analyzed. Section 3 presents related work. Section 4 discusses the design of the case study and Section 4.5 presents the data collection procedure. Section 5 presents the results of the conducted analyses, Section 6 provides an analysis of the results, and Section 7 discusses the validity of the study. Section 8 discusses the results from the study, and Section 9 concludes the paper.

This section presents a brief description of the FMEA and STPA hazard analysis methods that are compared in this study. It also provides an overview of other existing risk and hazard analysis methods. In addition, this section presents the description of the selected system, forward collision avoidance system, on which both methods are applied.

The research objective of the current study is to compare the STPA and FMEA hazard analysis methods. There exist some studies that have compared different risk and hazard analysis methods. For example, Stȧlhane and Sindre (2007) performed a comparison of two safety analysis methods, misuse case (MUC) method and FMEA. The MUC method was originally proposed for eliciting security requirements (Sindre and Opdahl 2005), but it has also been used for safety analysis. The MUC method was developed by the software community as an alternative to FMEA and HAZOP. Both methods were compared in an experiment to investigate which method is better than the other for identifying failure modes and if one of the methods was easier to learn and to use. The authors concluded that when the system’s requirements are described as use cases, MUC is better than FMEA for analyzing failure modes related to user interactions. Furthermore, FMEA is better than MUC for analyzing failure modes related to the inner working of the system. The authors also concluded that MUC will create less confusion and in general be easier to use than FMEA.

Yu et al. (2011) compared and discussed three well-known risk analysis methods by applying them on a box fan: FMEA, advanced failure mode and effect analysis (AFMEA) (Eubanks et al. 1997), and FTA. The authors presented the advantages and disadvantages of these methods and concluded their study with an attempt of combining both deductive (top-down) and inductive (bottom-up) risk/safety analysis methods.

Ishimatsu et al. (2014) compared the STPA hazards analysis results with the FTA analysis results that were used to certify the H-II Transfer Vehicle (HTV). The HTV is an unmanned cargo transfer spacecraft that is launched from the Tanegashima Space Center aboard the H-IIB rocket and delivers supplies to the international space station (ISS). In the development of the HTV, the potential HTV hazards were analyzed using FTA and during the analysis, the NASA safety requirements were also considered. After comparison of the results, the authors concluded that STPA identified all the traditional causes of losses identified by FTA and FMEA, but it also identifies additional causes. The additional factors include those that cannot be identified using fault tree analysis, including software and system design as well as system integration.

Fleming et al. (2012, 2013) analyzed the NextGen In-Trail Procedures (ITP) application by using the STPA analysis method and compared its results with the official NextGen ITP application analysis (RTCA/DO-312 2008). NextGen is the next generation of air traffic management systems that contains In-Trail Procedures application. ITP is an application of Automatic Dependent Surveillance-Broadcast (ADS-B) that allows aircraft to change flight levels in areas where current radar separation standards would prevent desirable altitude changes (Haissig and Brandao 2012). To summarize, ITP helps to increase operational efficiency and throughput in oceanic airspace (Fleming et al. 2013). The authors concluded that STPA found more potential causes of the hazards considered (violation of separation requirements) than the traditional hazard analysis performed on ITP (RTCA/DO-312 2008). In the comparison, the authors identified 19 safety requirements that were not in either of the two official NextGen analysis documents.

Fleming et al. (2012, 2013) also compared STPA with bottom-up and other top-down analysis techniques. According to the authors, bottom-up analysis techniques, FMEA, start by identifying all possible failures. This list can be very long if there are a lot of components and all the permutations and combinations of component failures are considered. However, STPA only identifies the failures and other causes that can lead to a system hazard and does not start by identifying all possible failures. Moreover, in the top-down STPA analysis approach, the analyst can stop refining causes at the point where an effective mitigation can be identified and does not go down any further in detail. The analyst only has to continue refining causes if an acceptable mitigation cannot be designed. That is the major difference between STPA and FMEA (and any other bottom-up technique), which explains the differences in time and effort required (Fleming et al. 2012; 2013).

Furthermore, Nakao et al. (2011) evaluated STPA in a case study where it was applied on an operational crew-return vehicle design. The authors conclude that with STPA, it is possible to recognize safety requirements and constraints of the system before the detailed design.

Raspotnig and Opdahl (2013) compare risk identification techniques for safety and security requirements. From the safety field, the functional hazard assessment (FHA), the preliminary hazard analysis (PHA), HAZOP, and FMEA as well as FTA are considered. Each technique is assessed based on several quality criteria addressing the context, the application area, and the application method as well as advantages and disadvantages of utilizing the technique. The assessment is based on evidence reported in the literature. The authors conclude that risk identification techniques for safety are more mature than for security and that they have found a balance between creativity and formalism, which is needed for identification process.

As it can be noticed from the literature mentioned in this section, STPA is a quite new analysis technique as compared to other techniques (FMEA, FTA, etc.). Fleming et al. (2012, 2013) mention that traditional analysis methods (FMEA, FTA, etc.) are more than 50 years old and, while analyzing safety critical software-intensive systems, they cannot identify software faults or the errors pertaining to dynamic behavior of the system. SFMEA (Pries 1998) and especially also STPA (Leveson et al. 2012) have been developed to overcome the existing problems in traditional analysis methods. According to Leveson et al. (2012) and Fleming et al. (2012, 2013), STPA can find more component interaction, software, and human hazards than traditional methods. Therefore, according to the authors, STPA is more effective because it is developed by considering system thinking that considers whole system as a single unit and finds more hazards. Moreover, previously, STPA is compared and evaluated with bottom-up methods (e.g., FMEA) by the same authors who presented it or they were involved in its development. Several authors (Leveson 2012; Pereira et al. 2006; Thomas and Leveson 2011; Ishimatsu et al. 2010; Nakao et al. 2011; Fleming et al. 2013) reported positive outcomes from applying STPA on various systems. However, the traditional methods are still in use in practice even though they are more than 50 years old for the analysis of safety critical systems in early design, development, and operational phases. This means that there is a need for further investigation of effectiveness of the STPA method compared to other traditional safety analysis methods that are used in industry. If further investigations find STPA as an effective method, then these results can help industry to shift to this new analysis method.

To summarize, it is interesting to investigate what are the main differences in STPA and other traditional methods (in this case FMEA) and also the types of hazards identified by them.

This section presents the analysis of the results that encompasses the main comparison results of both methods, FMEA and STPA, and their evaluation results based on the developed criteria.

The validity of a study represents the trustworthiness of its results, which means, for example, that the results are not biased by the researcher’s own opinion or point of view (Runeson et al. 2012). Validity of this kind of study (a case study) can be assessed regarding construct validity, internal validity, external validity, and reliability (Runeson et al. 2012; Yin 2003).

Construct validity considers the studied artifacts and concerns if they represent what the researchers have in mind and also if the studied artifacts are investigated according to the research questions of the study. In this study, the collision avoidance system was analyzed to identify hazards. The analysis was done by two persons, the first and the second authors of this study. The hazard analysis performed by the first author is already published in a previous paper (Sulaman et al. 2014). In the current study, the second author analyzed the system using FMEA and all the documentation and information were available, which were used during the hazard analysis performed by the first author. There could be a risk of not understanding the analyzed system and its description by the authors. To decrease this threat, a simple system was selected and also its detailed description was acquired and made available to all the authors of this study. Moreover, there could be another risk of not understanding the investigated methods by the authors. To decrease this threat, the experienced persons were selected to apply methods and also the suitable method guidelines and instructions were followed during the analyses. Also, regular meetings were carried out to eliminate any existing ambiguities in understanding of the system and its description and investigated methods. This could also impact the evaluation of the analysis process. The effect was however limited by dividing the analysis into steps that were assumed to be known and understood.

Internal validity is important and mostly applicable in studies of causal relationships. In this study, there can be a chance of history internal validity threat. To decrease the chance of history threat, the following measures were taken. The second author of this study was selected to apply FMEA on the collision avoidance system. The first author already knows the existing hazards in the selected system because he has applied STPA on the selected system in the previous study (Sulaman et al. 2014). Therefore, in the current study, to improve research validity, it was decided that the first author would not apply the FMEA method on the selected system. Instead, another author did that. The second author of this study did not have access or review the previous study results (Sulaman et al. 2014). Furthermore, it was also considered that the same system information and description is available for the second author to analyze the system.

All stages of risk and hazard analysis process involve subjectivity (Redmill 2002). There is always a chance of uncertainty, the need for judgment, considerable scope for human bias, and inaccuracy. It is highly likely that the results obtained by one risk analyst are not the same to the results obtained by other risk analysts starting with the same information (Redmill 2002). In our case study, both the authors (first and second) analyzed the collision avoidance system independently by applying different hazard analysis methods (STPA and FMEA). Moreover, both the authors have sufficient level of experience of analyzing safety critical systems and it is believed that in this case study, there is a little or no chance of this threat. Here, in this study, the objective is not to compare hazard analysis methods based on just the numbers of identified hazards; instead, the objective is to compare them based on the types of hazards found.

During the study, the results, and the written formulations of the results, were studied and discussed by all authors in order to limit the risk that results from one method were treated and formulated more positively than the results of the other. There is always the risk that other factors than the one controlled affect the result without the researchers’ knowledge. We have tried to identify and remedy the most important known factors.

Reliability is concerned with to what extent the data and the analysis are dependent on the specific researchers. The reliability was addressed by conducting both the data collection and analysis as a group of researchers instead of one single researcher. In this study, there are less chances of this threat because the data used for the analysis is of third degree (Lethbridge et al. 2005), e.g., documentation, description, and published literature. Moreover, the first-degree data collected in this study is the hazard analysis results or identified hazards. To decrease the chances of reliability threats, guidelines for both methods were properly used. Special measures were taken during the hazard analysis process and continuously reviewed by the co-authors. For example, the first author of this study performed an initial comparison of the identified hazards yielded from the FMEA analysis and STPA analysis. The first author created a list of the common and unique hazards that were identified by both analysis methods. After this, the second author reviewed the initial comparison performed by the first author. The second author identified one more hazard (no. 18 in Table 4) as a common hazard identified by both analysis methods. Then, the first author classified all the identified hazards into five error categories: component interaction error, software error, human errors, component error, and system error. This classification was also reviewed by the second author of this study for the researcher triangulation.

The common identified hazards are classified as software error and component error type mainly, as shown in Fig. 4. Moreover, there are some common identified hazards classified as component interaction type hazards. From the 13 common identified hazards, it can be observed that both methods found software error type hazards covering the dynamic behavior of the system. In Ishimatsu et al. (2014); Thomas and Leveson (2011); Leveson et al. (2012); Nakao et al. (2011); Fleming et al. (2012, 2013), the authors have mentioned that the traditional analysis methods (FMEA, FTA, etc.) cannot identify software errors. However, FMEA is still used in many safety critical hardware software systems and was extended to detect software hazards (McDonald et al. 2008) as well as vulnerabilities (Schmittner et al. 2014), and has even been applied for security testing (Peischl et al. 2016). Another difference to STPA is that it does not start from an undesired state but from a malfunctioning hardware or software component. However, in our case study, there is no major difference between the types of identified hazards by both applied methods on the collision avoidance system.

Furthermore, both methods also found some hazards that are unique to them (identified by only one method, either FMEA or STPA). STPA identified 9 unique hazards that are not identified by FMEA of which majority of the identified hazards are of software and component failure type hazards. On contrary, FMEA identified 8 unique hazards that were not identified by STPA. Interestingly, the majority of the uniquely identified hazards by FMEA are also of software and component failure hazards like STPA.

Moreover, a small difference can be noticed in the unique identified hazards by the two analysis methods regarding component failure type hazards. That means, FMEA identified more component failure hazards as compared to STPA. This shows the basic philosophy behind both methods: FMEA focuses more on components, their failures, and risk mitigation measures, whereas STPA focuses on delivery of control commands and their feedbacks.

One more interesting factor in our case study is that STPA found fewer unique system error type hazards than FMEA. Because STPA is developed considering system engineering and thinking, which consider whole system as a single unit instead splitting it in several parts. One potential reason of finding few system type hazards by STPA can be that the analyzed system in this study does not have many system type hazards. On the other hand, FMEA identified 2 out of 8 hazards of system error type. Another interesting difference can be observed regarding the component interaction error type hazards; STPA identified 18% hazards of component interaction error type. On the other hand, FMEA did not identify any hazard of component interaction error type. This result corroborates with the results of the previous studies.

Fleming et al. (2012, 2013) mention that the main difference of STPA from bottom-up analysis methods like FMEA is that bottom-up analysis techniques start by identifying all possible failures. This can result in a very long list of potential failures if there are a lot of components to consider in the analysis. However, this long list is produced because FMEA takes the architecture and complexity of components into account (Stadler and Seidl 2013). Moreover, this long list of potential failures can be managed by introducing a hierarchical structure in FMEA. Furthermore, FMEA fosters propositions for the structure of a hardware and software system and generates preventive measures during development and operating (Stadler and Seidl 2013). However, in our case study, both methods were applied independently by the different authors on a collision avoidance system to find operational hazards of the system that yielded in almost the same number of identified hazards (21 by FMEA and 22 by STPA).

However, one clear difference where STPA seems to outperform FMEA is finding causal factors of identified hazards. According to Fleming et al. (2012, 2013) , STPA considers more types of hazard causes than the other traditional hazard analysis methods. Therefore, STPA is more complete than existing traditional hazard analysis methods (Fleming et al. 2012). In our case study, the results corroborate with the findings of Fleming et al. (2012, 2013) regarding STPA’s complete causal factor identification.

On the other hand, FMEA (which is based on reliability theory) is stronger with respect to risk assessment of software failures by calculation of a risk priority number based on the complexity of a component or system. In STPA (which is based on system theory), there is no corresponding process step. Also assigning the application function to each sub-component and sub-system is not covered in STPA. However, the steps of (a) system decomposition and acquisition, respectively, (b) identification of potential failures, their causes and effects, and (c) definition of countermeasures map to each other. Especially, the definition of countermeasures is according to the technology acceptance model hard to perform and requires experience.

In this paper, we present a qualitative comparison of the two hazard analysis methods, failure mode and effect analysis (FMEA) and system theoretic process analysis (STPA), using case study research methodology. Both methods have been applied on the same forward collision avoidance system to compare the effectiveness of FMEA and STPA. Moreover, the analysis process of both methods is also evaluated by applying a qualitative criteria derived from technology acceptance model (TAM) (Davis 1989; Davis et al. 1989).

It can be observed that almost all types of hazards that were identified in the study were found by both methods. That is, both methods found hazards classified as component interaction, software, component failure, and system type. With regard to component failure hazards, FMEA identified more component failure hazards than STPA. With regard to software error type hazards, STPA found more hazards than FMEA of unique hazards. With regard to component interaction error type hazards, STPA found some hazards; however, FMEA did not find any of unique hazards. Finally, with regard to system type error hazards, FMEA found slightly more hazards than STPA.

Both FMEA and STPA consider system decomposition (FMEA decomposes and STPA considers whole system for analysis), identification of potential failures, their causes and effects, and definition of countermeasures. But STPA does not consider risk assessment in terms of risk priority number calculation and assignment of the application function to each sub-system.

The methods have different focuses. FMEA especially takes the architecture and complexity of components into account, whereas STPA is stronger in finding causal factors of identified hazards.

It can be concluded that, in this study, there was no type of hazard that was not found by any of the methods, which means that it is not possible to point out any significant difference in the types of hazards found. However, it can be observed that none of the methods in this study was effective enough to find all identified hazards, which means that they complemented each other well in this study.

Further research, especially in terms of case studies and experiments, is needed in order to investigate differences, but also combinations of the methods and possible extensions of them. It would be possible to carry out the same kind of evaluation and comparison by having two or more teams consisting of three to four analysts each trying out one analysis method and, in that way, investigate also the effects of teamwork in the analysis. Moreover, by having more experienced analysts for the analysis, a comprehensive list of hazards can be created that later can be used as a benchmark to evaluate results of analysis. It would also be possible to extend the analysis with more cases and explicitly add a case that has more human input to see if there are some differences in these methods or not. It would also be possible to carry out experiments to gather more data points for the analysis and, in that way, focus more on quantitative comparison than in this study.

In addition, safety has been defined as an important risk driver for testing (Felderer and Schieferdecker 2014), but the number of risk-based testing approaches taking safety analysis into account is limited (Erdogan et al. 2014). Comparing different safety analysis methods like FMEA and STPA with respect to test planning, design, execution, and evaluation is another suggested topic for further research that could help to increase adoption of safety analysis methods for risk-based testing.