The FUSE testbed: establishing a microgrid for smart grid security experiments

The current electricity infrastructure is evolving. Decentralized power generation demands more sophisticated control methods, Information and Communication Technology (ICT) components increase the risk for cyber-attacks and classical hierarchical top-down structures tend to be substituted by cell-based architectures, based on microgrids. A microgrid consists of several distributed generators as well as loads, storages and information and communication technology for monitoring and control operations. Through the Point of Common Coupling (PCC) it can connect to (grid connected, parallel mode) and disconnect from (islanding, autonomous operation) the main grid [7]. The ability of a microgrid to separate itself from the main grid and operate autonomously is very useful in case of faults and disturbances in the main grid.

In decentralized approaches system-parts, e.g. generators, loads, storages, microgrids and grid controllers need to communicate with each other and interact with each other. The next generation grid is being formed; the smart grid. Major challenges in ensuring a reliable and efficient energy supply include, but are not limited to: the growing complexity of the grid due to the increasing number of renewable and distributed energy sources, the necessary energy saving and demand response strategies to optimize the energy flow, as well as the various components, protocols and sensors involved in the data transfer, protection and control actions. Communication is a crucial component for a proper and reliable operation of a smart grid. However, this leads to new vulnerabilities due to possible attacks. In order to investigate attack scenarios and their impact on grid operation and control actions, an experimental microgrid facility is being established at the TU Wien for research on Future Self-Organizing Energy Networks (FUSE, https://​fuse.​project.​tuwien.​ac.​at/​).

The FUSE project’s main goal is to support the investigation of smart grid research questions by experiments in a real testbed with microgrid components. The project focuses on research questions on decentralized grid structures that consist of interconnections of microgrids. Microgrids are the key components in decentralized grid topologies and offer a broad range of opportunities for a flexible grid operation, which ideally uses self-organization to react to changing boundary conditions. Nevertheless, self-organization requires situational awareness. Internal microgrid operation as well as the controlled interconnection of microgrids is challenging regarding supervision and control. Expertise in both fields, i.e. in energy and communication, is required to address those challenges.

The FUSE project has its focus on the following key research aspects:

Future challenges in smart grid technologies require experts familiar with both, the energy and the communication domain. Therefore, besides the scientific objectives of the project it is also planned to exploit data and infrastructure in research-oriented teaching and establish research and teaching resources that can be shared with others (Fig. 1).

A microgrid, as already mentioned, is an energy system that is connected to the main grid through one or more Points of Common Coupling (PCC). It usually consists of several distributed generators as well as loads, storages and information and communication technology for monitoring and control operations. A microgrid can operate in two different modes: grid-connected (parallel mode) and islanded mode (autonomous operation) [7]. The ability of a microgrid to separate itself from the main grid and operate autonomously is very useful in case of faults and disturbances in the main grid. After normal operation of the main grid has been restored, the microgrid can be reconnected to it. While detailed simulation models of microgrid components exist and can be used for analysis and planning of microgrids, real infrastructure is required, not only to validate the simulation models, but also to improve them and to allow investigations of the complex interaction of smart grid components and ICT infrastructure [8], especially related to advanced methods of microgrid control [9‐12].

The communication network is a key component for a reliable and robust power grid. Important information about the status of the power grid, as well as failures or attacks can be extracted from measurement data, which is why their correct transmission through the communication network is of great importance. Measurement data recorded by metering equipment must not only be accurate and reliable but also provided in real-time in order to contribute to diagnostics and control actions. Every power system nowadays is equipped with SCADA (Supervisory Control and Data Acquisition) systems. Remote Terminal Units (RTUs) [1] support the reading of unsynchronized measurement values for a specific time interval. The measurement of complex power flows enables the calculation of bus voltages as a basis for state estimation.

Recently deployed Wide Area Measurement Systems (WAMS) use Phasor Measurement Units (PMUs) to measure voltage phasors directly. In theory such a WAMS infrastructure can offer an accurate snapshot of the network state at any point in time without state estimation. The deployment of PMUs on one hand increases demands with respect to the capacity and latency of communication networks [2]. On the other hand, the ability of PMUs to measure fine granular time-synchronized information is advantageous for grid management. PMUs sample the sinusoidal waveforms and estimate a synchrophasor equivalent of an alternate current waveform, while using the Global Positioning System (GPS) technology to accurately timestamp phasor measurements [3‐5]. The C37.118.1 standard addresses the measurement aspects of voltage phasors, frequency, rate of change of frequency, while the C37.118.2 standard addresses the communication aspects for data exchange. Analytics and diagnostic applications based on PMU data include phase angle deviations, oscillation monitoring, voltage stability monitoring, fast frequency decline, sudden change of active/reactive flows, cascading events, topology detection and others, whereas control applications may include microgrid controlling, intentional islanding, and Volt-Var optimization [1, 6].

However, in practice, the limited number and high cost of PMUs result in SCADA measurement data to be mixed with PMU data for state estimation purposes in today’s systems. In the FUSE network PMUs will be installed, among other sensors.

The FUSE microgrid experimental facility will consist of renewable generators, among which photovoltaic appliance and wind emulator, storages, controllable loads, sensors and actuators. The FUSE facility is established within a cooperation of three research groups and two Institutes of the Faculty of Electrical Engineering at TU Wien, Austria. Figure 2 depicts the components and structure of the FUSE microgrid.

The FUSE testbed provides a highly flexible environment and allows the possibility to perform a variety of experiments. We describe below three examples of experiments that are already planned in the project proposal and are significant for testbed requirements and the testbed design.

In this set of experiments various control algorithms are tested on failures or attacks that endanger grid stability. It is investigated how fast failures can be detected and how fast countermeasures (such as load shedding) can be invoked and if (and how fast) grid stability can be re-established.

In this experiment set collected data will be used for the development of models and training of model parameters for normal grid operation. These models will provide the basis for prediction methods and the detection of deviations from normal operation.

In this set of experiments cross layer anomaly detection will be tested. Anomalies can be caused by failures, misconfigurations or by malicious attacks. For this, data is gathered from sensors, log files, system data and network data from different layers in the FUSE testbed and will be analyzed and correlated. Although physical attacks are also possible, the FUSE project will first concentrate on attacks on IT components and communication. Different scenarios and use cases will be selected, including various IT infrastructure attacks, such as Denial of Service, eavesdropping, data injection and data modification. Since devices can always have undetected vulnerabilities, we also assume zero-day exploits in the scenarios. The detection quality and speed of different methods will be compared and evaluated. Combined models for the detection of anomalies both in communication network and electrical (micro)grids contribute significantly towards proactive planning and self-organizing energy networks.

In all three experiment sets, clock synchronization plays an essential role. It is needed to accurately monitor grid states and correlate distributed information to achieve situational awareness across multiple devices. In real environments, it is unlikely to have accurate time information available at all devices. In the FUSE testbed we equipped devices with GPS clock sync and distribute network time protocol information in the local network. This makes it possible to establish a fine grained view for situational awareness and to accurately correlate information from system logs, network measurements and grid sensors.

In the current testbed (as of December 2016) the following components have been already installed: a set of smart meters, a meter data collector, a phasor measurement unit (PMU), a phasor data collector (PDC), several GPS receivers with connections to servers and PMU, hardware and software components for the sensor data collection, including a storage solution and a data processing cluster. Further PMUs and the interconnection of the components to further equipment is currently in progress.

An experimental set up of a single PMU (Arbiter 1133A) has been installed in the Institute of Telecommunications’ Cyber-Physical Systems (CPS) laboratory. Initially, the communication between the corresponding PMU and a computer was tested through PMU model specific software. In the case of Ethernet connection, the properties of the communication can be alternated as far as the protocol is concerned (TCP/UDP), the number of frames per second, the device ID and others. Live visualization of the measurements (voltage and current magnitudes and angles) as well as data storage is possible. However, this serves only for the configuration, data visualization and storage of a single PMU device. For the scope of the FUSE testbed, a Phasor Data Concentrator (PDC) is necessary, which collects, visualizes, stores, aggregates and if needed sends the data from multiple PMUs to further concentrators (e.g., Super Phasor Data Concentrator, SPDC) or a central control center. Voltage and current phasors, frequency and statistical information (e.g., frames per second, time quality errors) are some of the data the PMU sends to the PDC. For this purpose, the open source software openPDC is deployed in the CPS laboratory. Data storage can be configured through historian adapters for various output streams that range from local comma separated value (CSV) files to remote SQL databases and other storage options. Power output calculation adapters are also provided by the software. Visualization of live measurements is also enabled, Fig. 3.

The data collected from the PMU devices and smart meters will be used as a basis for developing and investigating methods to detect failures and attacks. The ability of the PMUs to measure synchronized phasors with high resolution and accuracy enables not only the investigation and development of algorithms, which can detect slow evolving abnormal changes of metrics like, e.g., voltage magnitude or phase, frequency and others, but also the analysis and correlation of measurement data, which can only be realized in such a testbed. Preliminary tests of the WAN emulator clock synchronization performance on a 19″ HP Proliant DL120G6 with Celeron 2.2 GHz CPU running Linux confirm the expected, satisfactory accuracy as shown in Fig. 4. The results are based on the Network Time Protocol (NTP) daemon’s loopstats log file. Analysis of an exemplary chosen weekly NTP log file for week 40, 2016, confirms that the system clock is at any time within a window of ±20 μs to UTC. Moreover, the Cumulative Distribution Function (CDF) in Fig. 4, right, illustrates that 90 % of all system clock values stay within a narrow window of 15 μs (±7.5 μs) to UTC.

The experimental FUSE microgrid facility is described and first experimental installation of measurement equipment, data collecting and storing are presented. Based on the collected data from measurement equipment, methods and algorithms to detect anomalies both in the electrical and the communication networks will be developed and tested in the experimental facility. Security, timely data transfer and reliability of communications will be evaluated in the testbed in order to enhance situational-awareness as a step forward in evolving grids into self-organizing networks.