1 Introduction

1.1 What is demand response

Demand response is defined by the U.S. Federal Energy Regulatory Commission as follows [1].

Changes in electric usage by end-use customers from their normal consumption patterns in response to changes in the price of electricity over time, or to incentive payments designed to induce lower electricity use at times of high wholesale market prices or when system reliability is jeopardized.

Demand response contributes to the economy and reliability of a power system. From an economic point of view, demand response can shift energy use from high-cost to low-cost periods, thus reducing the costs of generation. From a power system reliability point of view, demand response can help maintain the system frequency and supply-demand balance.

Demand response programs can be categorized into incentive-based and price-based programs. They differ in what drives customers to change their consumption behaviors, i.e., incentive payments or time-varying prices. Incentive-based programs take a variety of forms. A popular form is direct load control (DLC), in which customers receive incentives and allow power companies to control some of their loads at certain times. Incentive-based programs were initially implemented in 1968 [2], when Detroit Edison, a power utility, implemented a DLC program. Price-based programs expose customers to prices that vary within 24 hours a day. They gain popularity as a result of installation of the smart meter technology [3]. Traditional meters accumulate energy usage over time, and customers are billed typically on a monthly basis. In contrast, smart meters can record energy usage on a more frequent basis, e.g., every 10 minutes, making time-varying pricing tariffs feasible.

1.2 What is transactive energy

Since the first demand response program was implemented, power systems have become much more complex. Distributed resources increasingly penetrate the grid, and generation has become more variable [4]. Also, intelligent devices such as smart thermostats are more accessible [5]. According to [5], networks of smart devices take complexity to “a scale where we cannot manage things centrally.” Furthermore, these new resources are mostly controlled by end users. It is a challenging task to monitor and manage these devices in real time. Traditional demand response programs have to adapt in the new environment.

Transactive energy is associated with “democratization of electricity”, “eBay of electricity”, and “internet of things” [6]. According to GridWise Architecture Council, transactive energy is “a set of economic and control mechanisms that allow the dynamic balance of supply and demand across the entire electrical infrastructure using value as a key operational parameter” [7]. The term “value” here basically equates to prices.

Under the framework of transactive energy, distributed flexible resources are directly controlled by their owners. Transaction mechanisms are designed to align individual behaviors with the system’s interests. Similar to existing demand response programs, transactive energy is concerned with creating incentives to ensure all resources are generating/consuming electricity in a system friendly manner. However, transactive energy extends the concept of demand response to both the supply side and demand side, and aims to balance supply and demand in a real-time, autonomous, and decentralized manner.

1.3 Comparison

An example is given to illustrate existing demand response programs and potential transactive energy programs. Consider a scenario with a photovoltaic (PV) and some flexible loads. Due to forecasting errors, PV generation is lower than expected. To re-balance power supply and demand, different methods will be used in transactive energy programs and conventional demand response programs.

In a price-based program, the market will temporarily raise electricity prices, expecting that consumers will reduce their load.

In a DLC program, the control center managing the area will remotely control and curtail some load.

In a transactive energy pilot in the U.S. Pacific Northwest, all flexible loads are represented by one agent [8, 9]. The load balancing authority is represented by another agent. The balancing authority agent negotiates with the load agent, requesting it to lower consumption.

In an extended form of transactive energy scheme, the PV will negotiate with the flexible loads. The PV will “buy” from the flexible loads the difference between its forecasted generation and actual generation. Table 1 compares the above schemes from three aspects.

Table 1 Comparison of demand response and transactive energy
Full size table

Section 2 and Section 3 will review the industry practice and research on demand response. Section 4 gives an overview of transactive energy, including its features, applications, and concerns.

2 Demand response: industry practice

2.1 Classifications of demand response programs

In incentive-based programs, users are offered monetary incentives and agree to reduce load to help maintain system reliability or to avoid high generation costs. DLC, interruptible load, and load as a capacity resource are common incentive-based programs [1].

In a DLC program, a power company is allowed to remotely control participants’ appliances such as heating, ventilating, and air conditioning (HVAC), water heaters, or pool pumps. For example, HVAC can be controlled to be cycled on and off via a switch on the compressor, or by adjusting room temperature set points via a smart thermostat.

In an interruptible load program, participants are subject to load interruption during system contingencies.

In a program where load serves as a capacity resource, participants commit to load reduction by pre-specified levels when system contingencies arise.

Price-based demand response provides time-varying price signals to induce consumers to reduce energy usage during high-price hours. According to [1, 3, 10, 11], time-varying tariffs typically include time-of-use (TOU) tariffs, critical peak pricing (CPP) tariffs, critical peak rebate (CPR) tariffs, and real time pricing (RTP) tariffs.

TOU is a tariff where electricity prices vary by time periods, each period being a block of hours. A 24-hour day is typically divided into peak hours and off-peak hours. In summer, for example, peak hours can include 6 hours in weekday afternoons, whereas off-peak hours include all other hours in the week. Prices are pre-determined at the beginning of a tariff cycle and kept constant until the end of the cycle (e.g., a season).

CPP is similar to a TOU, except that power companies are entitled to call critical events during a time period of high wholesale market prices and/or system emergency conditions. A critical event lasts for a limited number of hours, within which electricity prices increase substantially to incentivize users to reduce energy usage. When critical events are called, the time and duration of the price increase can either be pre-determined or vary based on how much load needs to be reduced in the events.

CPR is analogous to CPP, except that during a critical event, electricity prices remain the same while a user is refunded for a pre-determined rebate. The billing factor is a user’s usage reduction relative to what the power company’s expectation.

RTP is a tariff where the retail prices track wholesale market prices. As a result, it typically fluctuates hourly or more often.

Electricity rates in time-varying tariffs reflect the time-varying energy costs. The following cost components should be included in these tariffs [12].

  1. 1)

    Monthly fixed charge per customer to recover the costs that vary with the number of customers but do not vary with electricity usage.

  2. 2)

    Distribution facility charge per kW of peak demand to recover the operation and maintenance costs of local distribution facilities.

  3. 3)

    Location-specific and time-varying charge per kWh of energy usage to recover the marginal costs of electricity generation.

2.2 Findings of demand response pilots and programs

2.2.1 Overall performances and challenges

Starting from 2006, each year the U.S. Federal Energy Regulatory Commission publishes an Assessment of Demand Response and Advanced Metering Staff Report [1, 13, 14]. These reports document the latest progress of U.S. demand response programs, lessons learned, and trends. For example, one can find in these reports the peak load reduction by customer class in the U.S. In Fig. 1 (source: 2012 Assessment of Demand Response and Advanced Metering Staff Report), the largest portion of peak load reduction is attributable to commercial & industrial customers and wholesale market participants. The growth of peak load reduction is mainly driven by those customers. Relatively, residential customers make a modest contribution to peak load reduction.

Fig. 1
figure 1

Potential peak reduction by customer class

Full size image

These reports also identified some barriers to demand response as follows.

  1. 1)

    Customers have not been fully engaged. It is challenging to expect a large number of consumers to actively participate for monetary incentives. They need to be informed about the significance and opportunities of demand response.

  2. 2)

    Uniform standards for demand response pricing and incentives have not been established. Incentives, prices, and information exchange protocols are typically designed on a company-specific basis.

  3. 3)

    The measurement and cost-effectiveness of demand reductions continue to be an issue. It is crucial yet unsolved how to recover the costs of deploying demand response technologies and implementing programs.

2.2.2 Findings of price-based programs

Reference [15] reviewed 15 time-varying pricing pilots in the U.S. designed for households. It is found that the magnitude of peak load reduction depends on several factors, including the presence of central air conditioning, magnitude of the price increase, and availability of enabling technologies (such as programmable communicating thermostats). Across various pilots, TOU induces a peak demand reduction ranging from 3% to 6%. CPP achieves a peak demand reduction ranging from 13% to 20%.

The promises of price-based programs are as follows.

Enabling technologies are vital to the success of time-varying pricing. They free customers from manual response to price changes. According to [15], CPP with enabling technologies achieved a peak load reduction as high as 51% in the California’s Advanced Demand Response System program. This reveals the great potential of CPP in alleviating the pressure of peak load on a power system.

However, some challenges also arise from this type of programs.

  1. 1)

    The number of retail customers on price-based programs is limited. A possible reason is that retail customers’ savings in electricity bill resulting from a change in consumption behavior are trivial relative to the efforts to make such changes [16]. Another possible reason is that most retail customers are risk-averse and are not in favor of price variations [17].

  2. 2)

    Price-based programs may fail without technologies that enable automated response. According to [18], consumers, especially residential consumers, may not self-respond to prices in a meaningful manner. The largest TOU pricing pilot in the U.S. operated by Puget Sound Energy was discontinued. It turned out that a number of consumers had to pay more than they do under flat tariffs [5]. The lesson is that one can lose consumers’ time and attention by asking them to manually self-adjust their usage behaviors.

2.2.3 Findings of incentive-based programs

As opposed to price-based program participants, participants in DLC are not expected to self-change consumption behaviors. Reference [19] surveyed a number of DLC programs. The recruitment incentives in the programs include free installation of enabling equipment, one-time payments, and/or a recurring annual payment. It is reported that an average peak load reduction between 0.8 and 1.5 kW is achievable per residential participant. Small commercial and industrial participants can reach an average reduction between 2 and 4 kW.

The promises of DLC programs include the following.

In a DLC program, a participant can receive incentive payments without additional efforts beyond enrollment. Therefore, peak load reduction can be more significant and controllable compared to time-varying pricing [20]. In a Federal Energy Regulatory Commission survey, DLC ranked first in total peak load reduction potential [1].

Some associated challenges are as follows.

  1. 1)

    Based on the survey in [19], only 43% of customers expressed an interest in DLC while 74% of customers were interested in time-varying pricing. Recent pilots have shown that the inconvenience associated with the mandatory electricity interruption from DLC can lead to potential reluctance among consumers [17].

  2. 2)

    Privacy and equity issues also arise from DLC programs. How can one convince customers that DLC will not jeopardize their privacy? Who should pay the incentives? How should program benefits be shared? These are important decisions for a successful DLC program.

3 Demand response: research

Research on time-varying pricing can be categorized into two types: how to characterize consumers’ behaviors under a time-varying electricity tariff and how to design a time-varying tariff that fully exploits users’ demand response potentials.

Models characterizing consumer behaviors fall into two categories: price elasticity and utility functions. The price elasticity of electricity demand, including own price elasticity and elasticity of substitution, is studied in [21–26]. Own price elasticity refers to the percentage change in electricity demand in response to a percentage change in price of that same time period [27]. Elasticity of substitution refers to the elasticity of the ratio of demand in two different time periods with respect to the ratio of prices in those two time periods [27]. It indicates how easy it is for consumers to substitute demand in one period for another. The work of [28–33] models consumer behaviors via utility functions, including quadratic function, logarithm function, and power function. Consumers are assumed to be rational and, therefore, determine their load patterns by maximizing utility.

Approaches for designing time-varying tariffs include deterministic programming [22], stochastic programming [21, 23], and game theory [28, 34]. A single-level optimization problem is formulated by both deterministic and stochastic programming approaches. The decision variables are prices, and consumers’ elasticity is used to depict their responses to prices. The stochastic programming approach differs from the deterministic one in that the former captures uncertainties associated with consumer behaviors. Game-theory-based approaches feature a two-level optimization problem. In the upper level, a tariff designer, e.g., a power company, acts as a leader to set time-varying prices. In the lower level, consumers act as followers and behave to maximize their utility.

The above work does not rely on knowledge about specific load models. On the other hand, DLC programs allow power companies to access specific loads owned by consumers. This gives rise to another research direction, i.e., to develop demand response strategies for specific loads [35], such as HVAC loads, batteries, electric vehicles (EVs), data centers, and computer servers.

Among these flexible loads, HVAC draws most attention and is normally considered flexible for two reasons. First, minor temperature changes in customers’ buildings may not adversely impact human comfort [36]. Then, heat can be stored in buildings because of building thermal insulation, which further enables buildings to “store” electricity. That is, buildings can be overheated or overcooled when electricity prices are low, so HVAC devices can be turned off when electricity prices go up. In the work of [20, 37–39], HVAC loads are scheduled to minimize consumer electricity bills. These studies capture the thermodynamics of buildings and temperature constraints set by consumers. It is shown that the HVAC control scheme has a dramatic effect on both system-wide peak load and consumer electricity bills.

A battery is another promising resource for demand response. Indeed, a battery can arbitrage using price differences among time periods. That is, a battery can be charged when prices are low and discharged when prices are high. References [40, 41] derive the charging/discharging schedules for batteries, taking into account changes in batteries’ state of charge, charging/discharging rate limits, energy capacities, and impact of charging/discharging on battery life.

An EV, irrigation pump, or water heater can be responsive to price variations because they are deferrable loads. That is, they need to consume a certain amount of kWh energy within a certain time window, but it is flexible regarding how much kW load they need to consume at each instant. References [42, 43] deal with the design load schedules for such deferrable devices.

4 Transactive energy: overview

4.1 Characteristics of transactive energy

Transactive energy is designed to maintain the real-time balance of supply and demand in an environment where the number of distributed and self-controlled generation/load resources is rapidly increasing. It highlights the following features.

  1. 1)

    Distributed intelligent devices are controlled in real-time. Transactive energy can take place at time scales from fractions of a second to hours, whereas typical demand response takes place at time scales of hours or days [44].

  2. 2)

    These devices are “controlled” based on economic incentives rather than centralized commands. The participation of devices in balancing supply and demand is voluntary.

  3. 3)

    These devices exchange information and make transactions in a decentralized way to ensure the scalability of the control system.

  4. 4)

    These devices are managed under human supervision rather than human-in-the-loop operation. That is, these devices should be automated to enable real-time transactions and control.

  5. 5)

    These devices are controlled by their owners rather than power companies to ensure autonomy and protect customer privacy.

  6. 6)

    Transactive energy provides joint market and control functionality.

  7. 7)

    Both supply-side resources and demand-side resources are coordinated.

As a generalization of demand response, transactive energy exploits the flexibility of distributed generation and load resources to balance supply and demand.

There are also commonalities between the idea of transactive energy and that of smart grid. However, transactive energy highlights additional characteristics [5].

  1. 1)

    Transactive energy allows for faster transmission of information, including supply and demand quantities and prices, across the grid.

  2. 2)

    Transactive energy accommodates new generation assets using a functional decentralized supply model.

  3. 3)

    Transactive energy accommodates two-way power flows.

  4. 4)

    Transactive energy uses transactions at the retail level.

  5. 5)

    Transactive energy envisions that end users will have energy management systems (EMSs).

4.2 State of the art of transactive energy

The Pacific Northwest Smart Grid Demonstration is a $179 million transactive energy pilot project initiated in 2010 and lasting for five years [8, 9]. The project partitioned the Pacific Northwest power grid into 27 sub-regions that can exchange information with one another. Each sub-region had a local balancing authority. The authority estimated the cost of electricity delivered to neighboring sub-regions. The cost was dependent on an estimate of the quantity of power to be exported. The authority communicated the cost information with neighboring sub-regions. Its neighbors in turn fed back the quantity of power they would like to import. If the quantity in their feedback matched the quantity in the authority’s estimate, no additional information would be exchanged. If there was a disagreement, say, neighbors wanted to import more electricity than estimated, the authority would update (in this case raise) the cost of power export and neighbors would update (in this case lower) the quantity of power import. This process iterated until the cost and quantity of power export matched to the authority. The power exchange scheme above is analogous to centralized economic dispatch, but features decentralized and autonomous decision-making by sub-regions.

A transactive energy scheme is introduced in [45], where during a high-price event, flexible loads and less flexible loads negotiate to reach a consensus to reduce demand. Flexible loads receive compensation from less flexible loads because the former can avoid a price spike by reducing load. This transactive scheme features decentralized and autonomous decision-making between the two consumers. However, the scalability of this negotiation scheme should be further studied when hundreds and thousands of loads are involved.

A transactive energy scheme is envisioned by Pacific Northwest National Laboratory (PNNL) [46], where individual loads communicate with neighborhoods and determine respective energy consumption schedules in order to smooth their aggregate load curves. This transactive scheme also features decentralized and automated decision-making. However, it is unclear how consumers are incentivized to alter their energy usage patterns. Nor does [46] estimate the monetary benefits of smoothing load curves vs. the comfort degradation arising from user behavior changes.

Reference [47] demonstrates a residential transactive energy scheme, where a retail electricity market is run on a distribution feeder every 5 minutes. Each building sends a demand bid to an operation center located at a substation. The operation center assembles bids from all buildings on the same feeder to form an aggregate demand bidding curve. Supposing that the supply bidding curve in the feeder’s area is known, the clearing price can be found at the point where the supply and demand curves intersect. This clearing price is broadcast to all participants so each smart device knows how to behave. This approach features automated and autonomous decision-making. Information exchange is limited to only a feeder to avoid the scalability issue when centralizing all information in the operation center. By finding the local instead of global market equilibrium, however, the optimality of the outcome may be compromised.

Reference [48] argues that existing bulk power markets are also examples of transactive energy systems, because prices and economic signals are used in these markets to drive economic efficiency and balance supply and demand. Wholesale power markets in California, Texas, and New York maintain the real-time balance in power supply and demand through largely centralized operations, such as security constrained economic dispatch (SCED) [49–51]. From the authors’ perspective, however, wholesale market mechanisms such as SCED may not be directly copied to distribution systems where the number of players can get extremely large. Transactions and control functionalities need to be decentralized and automated in a transactive energy ecosystem.

4.3 Potential transactive energy scheme

This paper also envisions an extended transactive energy scheme. The scheme requires that each transactive node in the power system provide and execute a generation/load schedule. A transactive node is a node equipped with an agent that communicates with other agents and makes automated decisions. The generation/load schedules can be determined either day-ahead or hour-ahead, and derive from either self-schedules or pool-market clearing. Any deviation of a schedule would result in a deviation charge.

If one transactive node cannot follow its schedule in real-time, it will publish a request regarding the MW energy deviation it needs to eliminate. The recipients are its “qualified trading partners”, i.e., nodes that the transactive node trusts and prefers to make transactions with. A partner should also be physically close to the transactive node to localize the transactions and avoid significant network flow changes. Upon receipt of the request, the partners respond by returning offers that specify the MW electricity they are willing to generate/consume and the prices of their offers. The transactive node can then decide whether to accept one or more of these offers. Once an offer is accepted, the corresponding partner alters its schedule to offset the transactive node’s deviation, and the transactive node avoids penalty charges.

Figure 2 illustrates one application of this scheme. A transactive node with a PV has to generate 1 MWh less electricity than is scheduled in the next 5 minutes due to weather forecast errors. To avoid deviation charges, this node sends a request to four partners, i.e., three buildings and one battery. The partners have schedules to follow (the red dotted lines in Fig. 2) and flexible loads that allow them to change consumption patterns. By estimating the potential loss (e.g., comfort loss arising from 1 MWh usage reduction), each partner returns an offer to the PV node. The PV node then makes a transaction with building 1 who offers 1 MWh usage reduction with the lowest price. As a result, both the PV node and building 1 change the schedules (from original red dotted lines to black solid lines), passively and actively, respectively. The changes offset each other, so the rest of nodes can still stick to schedules while the power balance is still maintained.

Fig. 2
figure 2

Illustration of one potential transactive energy scheme

Full size image

Under this mechanism, supply and demand are balanced as long as all resources track their schedules. In case some node is to deviate from its schedule in real-time, localized and decentralized transactions allow the node to fix the deviation with its partners, while the rest of nodes are immune to changes.

Several research topics regarding this mechanism are identified as follows.

  1. 1)

    To ensure the global optimality of this mechanism, a transactive node’s qualified trading partners should include every other transactive node in the power grid. To ensure real-time communicational and computational tractability, a transactive node’s qualified trading partners should be limited to a small number of its neighboring nodes. It is practical to define the qualified trading partners by making tradeoffs between optimality and practicability.

  2. 2)

    The transactive node launches an auction when it needs partners’ help to offset a deviation. There are four major auction types [52], i.e., open ascending price auction, open descending price auction, sealed-bid first-price auction, and sealed-bid second-price auction. The differences of the four auctions are to be illustrated using Fig. 2. Table 2 shows the payments to each player under the four auctions.

    Table 2 Comparison of auction types
    Full size table

The sealed-bid second-price auction is strategy-proof and ensures that the optimal strategy for each player is to bid his/her true valuation. Therefore, by a sealed-bid second-price auction one can infer that the four bidders’ valuations of 1 MWh electricity are $90, $100, $110, and $120. By contrast, the first three auctions incentivize players to strategize. If the four bidders’ valuations are $90, $100, $110, and $120, Player 1 would not just bid $90 and receive $90 under the first three auctions. Player 1 would try to estimate the second lowest price among all bids and set his/her bid just a bit lower than the second lowest price. In the first three auctions, Player 1 can bid $99.99 (a bit lower than $100). By doing so, Player 1 still wins and maximizes his/her payoff.

It is important to study which auction type is more appropriate for the transactive scheme. Among them a sealed-bid second-price auction may be most suitable, because it is strategy-proof and improves the efficiency of the auction process. In case the transactive node needs to divide its MW deviation into multiple pieces and buy/sell these pieces to multiple partners, a Vickrey–Clarke–Groves (VCG) auction may be used [53]. It is a generalization of a sealed-bid second-price auction with multiple items.

  1. 3)

    It is important to identify the bidding strategy for the partners. A partner may lower its utility by altering its schedule to offset the transactive node’s deviation. The partner should quantify its degradation of utility arising from schedule changes. This utility degradation can serve as a basis of bidding prices.

4.4 Promises and challenges of transactive energy

Transactive energy is motivated by the need to manage a complex system consisting of a large number of distributed and self-controlled generation/load resources. Those controllable and flexible resources, either on the supply side or demand side, are incentivized to cooperate with those noncontrollable and variable ones. Some potential benefits that transactive energy is expected to deliver include: (a) it optimizes the use of distributed energy resources; (b) it improves power system efficiency and reliability; (c) it reduces the requirements for capacities and spinning reserves to address generation/load uncertainties; (d) it creates a fair and transparent platform that allows all resources to transact.

However, there are also challenges that stakeholders should consider [5].

  1. 1)

    Technology. What is the current level of automation of energy management devices and appliances? Are they ready for deployment, reliable and affordable?

  2. 2)

    Scalability. A distributed platform in the transactive world is expected to scale well. Can the platform function well when the number of smart devices in the distribution system increases significantly?

  3. 3)

    System management. As a highly centralized control system moves toward a more decentralized system, who will oversee and govern such a platform? The emerging technology of blockchain is perceived as a promising platform for transactive energy due to its decentralized, cyber-attack proof, and transparent feature [54]. Can a blockchain platform manage problems such as congestion, power quality, and reliability?

  4. 4)

    Consumer behavior. Transactive energy is based on the vision that “individual customers understand their needs best [55].” Is transactive energy empowering consumers or making their lives more complicated? How can one prepare consumers for this new concept? How can consumers derive enough values from this platform so they are willing to participate?

All these questions are crucial for the transition from conventional demand response to transactive energy, and should be further studied by the industry and academia.

5 Conclusion

Although significant progress has been made in demand response, there are outstanding barriers to overcome to further its performance. In addition, the power grids are undergoing a transformative change that requires demand response to adapt. Transactive energy has been proposed as a promising solution that goes beyond demand response. It is expected to maintain the dynamic balance of supply and demand by enabling real-time, decentralized, automated, and autonomous transactions among distributed generation and load resources. In this emerging field, much research is needed. One can identify the value proposition of transactive energy, design the mechanics of how it should function, develop enabling platforms, and derive strategies for individual participants.