Evidence-based medical equipment management: a convenient implementation

The first step was to identify the classes of equipment as a target for the focused research analysis.

It was decided to give relevance to the most numerous and critical devices. The choice was restricted to classes of equipment with more than 40 units belonging to the two aforementioned departments. Table 1 shows the 13 selected classes with the device type in the first column and the quantity in the second. We analysed technical reports on CM and SM activities, including preventive maintenance, electrical safety tests and quality control. Table 1 also shows the number of CM and SM WOs in the third and fourth columns. The rightmost three columns show how many pieces of equipment are in use in surgery departments and intensive care units, as well as the related CM and SM work orders.

A total of 14.06% of the data were excluded from the analysis because the related reports lacked enough information for a proper classification.

The purpose of classifying the maintenance operations was to analyse and monitor the types of performed operations. The number of CM cases corresponds to the number of failures that occurred (except for false failures, NPF). The same codes used in [21] are used to identify each failure type summarized in Table 2. Each individual CM and SM technical report was carefully analysed so that the many failures that occur each year could be catalogued. In ambiguous cases, when there was a possible correspondence of two or more codes with the same failure, the most appropriate code was selected through careful analysis performed in cooperation with CE technicians and staff.

The UNI EN 15341:2007 standard [19] describes a system for managing KPIs to measure maintenance performance as influenced by key maintenance factors and to assess and improve efficiency and effectiveness. The standard is applicable to many industrial and technical sectors. The maintenance of medical devices must ensure equipment availability and reliability (linked to the safety of the device).

The standard suggests that the KPIs be structured into three groups to measure every aspect of the maintenance process. A thorough review of the literature led to the selection of the three groups below to match the CE department’s data and requirements. These KPIs are as follows:

Taking into account the criteria suggested by the above standard, as well as the analysed literature and our personal knowledge and needs from the field, we designed a set of 20 KPIs, which are thoroughly described in Table 3 below. The table summarizes the information on the chosen indicators: their name, the type of indicator (financial, organizational or technological), the mathematical definition and the rationale behind it. Moreover, the table indicates which activities are pertinent to each indicator, between CM and SM. Internal maintenance (IM) and/or external maintenance (EM) activities are indicated for each indicator as well. The identified indicators were calculated for each year from 2012 to 2016 for each of the 13 chosen equipment classes to obtain the overall behaviour and evolution of each indicator over time.

The set of 20 indicators in Table 3 not only come from the UNI EN 15341 standard but also are intended as a novel research result of our study.

To further clarify the indicator concepts of downtime and uptime, the European standard EN 13306:2010 was used as a reference [18]. Downtime is the time interval throughout which an item is not capable of performing its function. Uptime is the time interval throughout which an item is fully functional. The well-known mean time to restoration (MTTR) and mean time between failures (MTBF) are the average times to restoration of function and the average time between consecutive failures, respectively.

With regard to the financial indicators, the acquisition cost (used in KPI-15) was derived from the tables showing the purchase value estimates for each equipment class supplied by the CE department in 2014, increased by 20%. Furthermore, a 2% increase or decrease was estimated for the years subsequent to and prior to 2014, respectively. The acquisition costs reached for each class were multiplied by the annual number of each class. The decision to use purchase value estimates was justified, as indicated in the literature [8], by the fact that the purchase cost of each individual device represents only the initial portion of the total cost of ownership of that device. The total cost of ownership comprises several cost items, such as contracts, spare parts, accessories, consumables and instruments used to perform test measurements.

To calculate total maintenance costs, only the cost items clearly linked to each device were considered. Therefore, the total maintenance cost was calculated using these cost items:

The histograms in Fig. 2 show the distribution of failure codes obtained from CM and SM WOs, which were related to surgical tables, telemetry equipment, electro-surgery units and defibrillators. The graphs obtained through the analysis of CM WOs are in the first column. The graphs from SM WOs are in the second column. Details are sometimes shown for a better data comprehension.

The first five histogram bars for each type of failure represent the five investigated years (2012 to 2016). The rightmost one is the average value with error bars of ± 1 standard deviation (SD). The height of the bars represents the percentage of failures found in CM or SM WOs.

The CM values were corrected using the equipment type failure rate (ETFR), which is the percentage of units within a specific equipment type that failed each year [21]. This correction is necessary when a combination of CM and SM failure code distributions is required to provide a more complete view of the equipment fault history.

In fact, the CM WOs are only related to failed units and did not consider units that had not failed. Instead, the SM WOs refer to all units.

From the top left chart in Fig. 2 (CM distribution for defibrillators), some interesting information can be gleaned about the effectiveness of the most common types of maintenance procedures used in the category of defibrillators. It appears that the most significant failure category involves batteries (BATT). Specifically, in 2015, 60.40% of CM WOs were for battery failures, 8.91% were related to wear and 5.94% were unpreventable failures. In 2016, 93.33% of the SM WOs were NPF, 3.81% were for BATT and 2.86% were preventable failures (PF).

From a review of the literature [23], a comparison of maintenance strategies in different hospitals shows consistent differences. As expected, BATT failures were lower in cases where scheduled maintenance was performed more frequently. The paper confirms that a higher frequency of scheduled maintenance reduces certain types of failures (BATT). Therefore, the pattern from our case could be explained by a lower rate of scheduled maintenance coverage than the rate provided by the maintenance plan before 2016.

Another interesting fact emerged by comparing the CM and SM fault patterns for defibrillators (Fig. 2), the latter including preventive maintenance, electrical safety audits and quality control. The classes with potential maintenance omissions are PF and EF in SM and PPF in CM. The latter category is responsible for approximately 3% of all failures. Through the analysis of the technical reports, this circumstance may be ascribed to poor first-level maintenance by health personnel. This result is in line with the peculiarity of Careggi University Hospital, which is an extremely large healthcare facility with a very high rate of personnel turnover. This could affect staff accountability in asset management (the conflict between university and hospital property) reflected in the consistency of the USE failure class incidents that include accidental failures and failures due to the misuse of equipment. These data confirm the importance of staff training. Since there is no user-training programme beyond initial training during device testing, there is a high incidence of failures due to improper device management.

From the corrective maintenance pattern for electro-surgery units (Fig. 2), it is evident that the most affected category in this case is UPF. This type includes a broader range of failures, normally attributed to wear. The scheduled maintenance pattern shows a prevalence of procedures with positive outcomes (NPF). Nonetheless, there is a 1% potential failure rate, which might be related to issues that can be resolved by increasing the frequency of scheduled maintenance and by paying more attention to checking the correct function of components that are more prone to failure (pedal, plates and handpieces).

By analysing the CM procedures on surgical tables from 2012 to 2016 (Fig. 2), the most significant category of failures is ACC (failures of accessories). Some of the indirect actions the CE department could implement to reduce this type of failure belong to the procurement stage. Giving importance to the reliability of accessories and spare parts as well as analysing the brands in use with a higher failure rate in relation to the total number of units in the inventory could be effective in reducing these failures. The scheduled maintenance chart clearly shows that, every year, scheduled maintenance procedures on surgical tables take place with no negative results reported. The remaining 2% of the maintenance WOs were coded as evident failures (EF). To reduce this type of failure, personnel should be trained to immediately report failures and problems that are evident and can be identified with no special tools or measurements.

The telemetry corrective maintenance histogram for failures from 2012 to 2016 shows a failure history that includes three types of problems: UPF, USE and NPF. The peak of the UPF category for 2016 is in line with the list published by ECRI for the “Top 10 Health Technology Hazards” for 2016 [9]. This report states that telemetry failures and the resulting lack of monitoring of a patient’s vital signs are in 4th place among the hazards to patients from medical technology failures. This report mainly discusses the improper use of medical devices. Moreover, it highlights the importance of taking actions focused on reducing USE because this is the other category of failures that affects these devices. For example, among the indirect actions the CE department could implement, it is worth reiterating that better staff training can be effective in appropriate telemetry management.

From a review of the technical reports on telemetry, the high level of stress on these devices emerges, as they are constantly connected to patients being monitored. The purchase of more robust and reliable equipment that is able to better handle high technological stress levels could be a solution. The scheduled maintenance procedures on these devices have a 100% success rate with all positive outcomes (NPF).

This section presents some considerations on the values of the set of indicators identified in Section 2.3.

All the calculated KPIs for each equipment class, as well as the distribution of the equipment in classes of age, are shown in Tables 4, 5, 6a and b and 7 below.

Concerning the economic KPIs, the COSR (cost of service ratio) results (i.e. KPI 15) were compared with the values of the economic indicators proposed by the Procurement Unit of the Italian Public Administration CONSIP (i.e. Public Information Services Licensee). The value estimates were in line with the CONSIP values [7]. In this comparison, the electro-surgery unit class deserves further investigation because it has an average COSR value (1.45%) lower than that of CONSIP. Until 2014, there was a no-cost maintenance service policy in place for many of these devices (by contract, the cost was absorbed by the purchase of consumables); this fact strongly influenced this value. In addition, the range of the electro-surgery technology was highly variable, requiring extremely specialized equipment, with high initial purchase costs and very high consumable costs. CONSIP estimates put electro-surgery maintenance incidence at a medium-high level of 8%, so it seems clear that a policy with a service formula would be preferable to purchase.

The results related to KPI 16 (external maintenance cost) and KPI 17 (internal maintenance cost) for the class of electrocardiographs, which are characterized by both internal and external maintenance policies, should be highlighted due to the impact of this mixed policy on total maintenance costs. From 2012 to 2014, external maintenance costs accounted for an average of 93.03% of the total cost. Beginning in 2015 through 2016, the maintenance policy changed and internal technicians provided maintenance. Therefore, between 2015 and 2016, the impact of external maintenance costs dropped to an average of 69.48% of the total cost. This is an example of how evidence-based maintenance works. Because of the experience of internal technicians, starting from the evidence, it was possible to adjust the maintenance policy, leading to economic improvement.

A comparison of the cost patterns for KPI 18 (CM cost) and KPI 19 (SM cost) on a single class of equipment (telemetry) showed that preventive maintenance accounted for an average of 5.39% of the cost compared with corrective maintenance, which accounted for 94.61%. This difference can be explained by considering the 0.56 average coverage rate of scheduled maintenance for this class. Therefore, to improve this situation, the maintenance policy should be changed to guarantee that preventive maintenance will be performed on each device at least annually. This improvement in the maintenance schedule would probably reduce corrective maintenance costs.

For the first and the second equipment, downtime was affected significantly by the time required for spare parts to be delivered. Indeed, for these categories, failures of accessories (ACC) were significant. Any equipment downtime directly affects its availability or uptime (KPI 2). The uptime pattern for surgical tables with and without negligent maintenance service is shown in Fig. 3. Clearly, negligent maintenance service significantly affected uptime.

For the anaesthesia machine, it is also necessary to consider that the SM has a long duration for each intervention and, considering that this class has an average SM coverage rate of 1.09, the time dedicated to preventive maintenance affects the availability of the equipment (13.6%, on average) and, consequently, the uptime. Instead, the problem with telemetry could be linked to a higher incidence of negligent maintenance service. This leads to a hope for an organizational correction of the maintenance policy. Specifically, the implementation of an accurate monitoring system for service calls protracted over time, such as a dashboard, could reduce negligent maintenance service that prolongs downtime and delays the availability of the device.

The average annual uptime value, directly related to the downtime figure, for all equipment is better than 94%, which is the CONSIP figure that should be guaranteed each year.

It is no surprise then that the class with the highest average MTTR (KPI 3) values (approximately 4.5 days) is “surgical tables,” which, as mentioned above, are affected by negligent maintenance service. The time required to restore the correct function of the devices in this category is strictly linked to the time required for spare parts shipping. The surgical tables’ MTTR pattern in Fig. 4 shows a peak in 2014 for the MTTR with negligent maintenance service. Instead, the MTTR without negligent maintenance service is at the minimum uptime for that year.

Contrary to expectations, an analysis of the age failure rate (KPI 7) shows no correlation between failure rate and obsolescence. This could be due to the presence of no problem found (KPI 12, i.e. “fake faults”) that do not represent real failures of the equipment and represent a good 19% of the total corrective interventions (684 out of 3581). This category could introduce a distortive component (bias) that prevents a clear interpretation of the indicator performance. The calculation of this indicator was, therefore, repeated by removing the percentage attributable to NPF for some classes, but in general, a correlation between failure rate and age of the device did not appear evident. However, to calculate KPI 7, the equipment was divided into age classes, and this subdivision provided useful information on the age composition of the operating room and intensive care equipment (see Table 7). The data could then be compared with the average age from the literature. For example, among the analysed classes of equipment, it was found that defibrillators, ECGs, aspirators, ventilators and operating tables comprise more than 40% of devices that are older than 10 years. By consulting the data from the Biomedical Engineering Advisory Group (BEAG) [4] and the American Hospital Association (AHA) [26], it can be noted that the average age of operating tables and vacuums is equal to 15 years, that of ventilators and ECG ranges between 7 and 10 years, and that of defibrillators is 5–7 years. Therefore, in our data, defibrillators have an average higher age with respect to the values reported by BEAG and AHA. From the analysis of the failure pattern, however, the classes of equipment in which there is a high percentage of devices over 10 years of age do not always show a higher failure rate than the classes with newer devices.

The age indicator pattern suggests that the variability in age in terms of failure rate is probably much less than the variability of other factors, such as whether operators manage and use the devices properly or not. Therefore, age does not seem to be a significant parameter in current maintenance policies.

Negligent maintenance service (KPI 8) (corrective maintenance calls resolved in more than 30 days) are to be applied together with uptime (KPI 2), downtime (KPI 1) and MTTR (KPI 3). Compared with the total number of failures, the telemetry equipment and surgical table classes have the highest number of negligent maintenance service calls. Instead, the number of 1-day service calls (KPI 9) (corrective maintenance service requests resolved within 24 h) was found to be lower for telemetry equipment, which were more affected by negligent maintenance service. However, 64.49% of defibrillators were serviced within 24 h. This equipment also has KPI 1 and KPI 3.

No problem found (KPI 12) was more significant for aspirators and anaesthesia machines. However, if compared with the involved workload (approximately 5 days a year), it does not greatly affect the wasted time. In this case, investing in training to instruct staff to open corrective interventions in a better way would not be economically advantageous because the incidence of these “fake faults” does not justify the investment.

The SM coverage rate (KPI 11) is higher than 1 SM intervention per year for surgical lamps, ventilators and anaesthesia machines. For defibrillators, only in 2016 is the planned target of 2 preventive maintenance interventions per year reached because of the transition from internal service to external global service, due to a low internal service coverage rate in previous years. Other classes of equipment have a value of KPI 11 less than 1 mainly because devices that cannot be found or that are used continuously are unavailable for maintenance activities. Ceiling-mounted units from 2012 to 2015 have KPI 11 values less than 0.4. This low value is due to an SM frequency of 2 years, as specified in technical manuals. In Table 8, the yearly SM coverage rates for each equipment class are shown from 2012 to 2016.

By comparing internal maintenance and external service in terms of number of devices per technician (KPI 13) and time cost of the workforce (KPI 14) in CM and SM, it can be concluded that, in relation to the considered equipment, the internal technicians manage a higher number of devices than the service at the expense of more hours spent in maintenance. Despite a higher workload, the internal technicians mainly manage classes of equipment such as the operating room cabinets, telemetries and scialytic lamps. They are highly specialized in these types of equipment; hence, they are able to optimize their timing and manage a greater workload.

The combined use of the two approaches discussed thus far provides a tool for broad spectrum monitoring of the maintenance process. Indeed, the KPI values can be better investigated and understood by making use of the types of coded maintenance calls. Similarly, if an unusually high failure type is detected, the effects of this problem on the performance of the entire maintenance process may be observed. In this way, targeted corrective actions can be taken to improve maintenance service.

For example, if surgical tables have above-average downtime and MTTR, this can be investigated in greater depth by analysing the evidence, or rather, the types of failures. The spare parts cost indicator also shows a high impact on total maintenance costs for the surgical tables. From an analysis of the coded maintenance calls, one can find, that for this category of equipment, accessory failures have a greater incidence. The actions taken to improve the process can include monitoring the time needed for spare parts shipping and taking a survey to identify the most failure-prone accessories (models and brands) for consideration during procurement.

For example, on average, 34.39% of costs were for internal telemetry maintenance. More than half of all maintenance costs were for repairs that required sending the device back to the company for service. For this equipment class, the influence of negligent maintenance service (maintenance calls that last more than 30 days) affects not only equipment availability, which is a prolonged period out of service, but also maintenance costs, which represent the greater part of the cost items.

Another analysis procedure could be performed beginning with the classified service calls. Through a review of the categories with unusually high failure, the reasons for this singular behaviour may be investigated in more detail using KPIs. For example, electro-surgery units showed a peak of component failures in 2016. An analysis of the coverage rate indicator for scheduled maintenance calls showed that, in 2015 and 2016, SM coverage was not optimal. In addition, the analysis of the coded corrective maintenance calls in 2016 showed that 100% of the calls were classified as NPF. This seems to suggest that specific checks on the components that tend to fail most during scheduled maintenance calls should be included.

Another example concerns the defibrillators, where an analysis of the fault types revealed a prevalence of battery failures (BATT). An analysis of the scheduled maintenance indicator showed poor coverage for defibrillators, which improved in 2016, the year the equipment was transferred to a global service policy.

However, in 2016, the global failure rate for the defibrillator class was the highest of the 5 years analysed. This suggested that, despite adequate scheduled maintenance coverage, it might be useful to include specific battery status checks during the scheduled inspections. Given that 67% of the total maintenance costs for this class of equipment were represented by battery costs, appropriate battery management could also reduce maintenance costs.