Abstract:This paper establishes a two-layer data-driven robust scheduling method to deal with the significant computational complexity and uncertainties in scheduling industrial heat loads. First, a two-layer deterministic scheduling model is proposed to address the computational burden of utilizing flexibility from a large number of bitumen tanks (BTs). The key feature of this model is the capability to reduce the number of control variables through analyzing and modeling the clustered temperature transfer of BTs. Second, to tackle the uncertainties in the scheduling problem, historical data regarding BTs are collected and analyzed, and a data-driven piecewise linear Kernel-based support vector clustering technique is employed to construct the uncertainty set with convex boundaries and adjustable conservatism, based on which robust optimization can be conducted. The case results indicate that the proposed method enables the utilization of flexibility in BTs, improving the level of onsite photovoltaic consumption and reducing the aggregated load fluctuation.

Abstract:Contingencies, such as behavior shifts of microgrid operators (MGOs) and abrupt weather fluctuations, significantly impact the economic operations of multi-microgrids (MMGs). To address these contingencies and enhance the economic and autonomous performance of MGOs, a self-organizing energy management modeling approach is proposed. A second-order stochastic dynamical equation (SDE) is developed to accurately characterize the self-organizing evolution of the operating cost of MGO incurred by contingencies. Firstly, an operating model of MMG relying on two random graph-driven information matrices is constructed and the order parameters are introduced to extract the probabilistic properties of variations in operating cost. Subsequently, these order parameters, which assist individuals in effectively capturing system correlations and updating state information, are incorporated as inputs into second-order SDE. The second-order SDE is then solved by using the finite difference method (FDM) within a loop-structured solution framework. Case studies conducted within a practical area in China validate that the proposed self-organizing energy management model (SEMM) demonstrates spontaneous improvements in economic performance compared with conventional models.

Abstract:The optimal dispatch of energy storage systems (ESSs) in distribution networks poses significant challenges, primarily due to uncertainties of dynamic pricing, fluctuating demand, and the variability inherent in renewable energy sources. By exploiting the generalization capabilities of deep neural networks (DNNs), the deep reinforcement learning (DRL) algorithms can learn good-quality control models that adapt to the stochastic nature of distribution networks. Nevertheless, the practical deployment of DRL algorithms is often hampered by their limited capacity for satisfying operational constraints in real time, which is a crucial requirement for ensuring the reliability and feasibility of control actions during online operations. This paper introduces an innovative framework, named mixed-integer programming based deep reinforcement learning (MIP-DRL), to overcome these limitations. The proposed MIP-DRL framework can rigorously enforce operational constraints for the optimal dispatch of ESSs during the online execution. This framework involves training a Q-function with DNNs, which is subsequently represented in a mixed-integer programming (MIP) formulation. This unique combination allows for the seamless integration of operational constraints into the decision-making process. The effectiveness of the proposed MIP-DRL framework is validated through numerical simulations, demonstrating its superior capability to enforce all operational constraints and achieve high-quality dispatch decisions and showing its advantage over existing DRL algorithms.

Abstract:The spot flexibility markets are before the real-time energy exchange, allowing demand-side management to reduce energy consumption during peak periods. In these markets, demand aggregators must quickly choose the customers ’reduction bids that fulfill grid requirements. This clearing procedure is challenging due to the computational complexity of selecting the optimal bids. Therefore, developing a clearing mechanism that avoids searching the entire flexibility bid space while respecting grid constraints is essential for the smooth operation of the spot flexibility market. This paper presents a clearing mechanism with reduced computational complexity of the winner determination problem in spot flexibility market for demand aggregators carrying out reductions in energy consumption. The proposed approach transforms customers’flexibility bids into a reward-based function. Afterward, the gradient-based optimization solves the bid selection problem. This approach helps demand aggregators achieve satisfactory energy reductions within an appropriate delay for spot flexibility markets. A comparative study presents the effectiveness of the proposed approach against commonly used approaches: hybrid particle swarm optimization genetic algorithm and combinatorial search.

Abstract:This paper proposes a novel parallel hybrid deep reinforcement learning (DRL) approach to address the real-time energy management problem for microgrid (MG). As the proposed approach can directly approximate a discrete-continuous hybrid policy, it does not require the discretization of continuous actions like regular DRL approaches, which avoids accuracy degradation and the curse of dimensionality. In addition, a novel experience-sharing-based parallel technique is further developed for the proposed approach to accelerate the training speed and enhance the training robustness. Finally, a safety projection technique is introduced and incorporated into the proposed approach to improve the decision feasibility. Comparative numerical simulations with several existing MG real-time energy management approaches (i.e., myopic policy, model predictive control, and regular DRL approaches) demonstrate the effectiveness and superiority of the proposed approach.

Abstract:Over the past decade, bidding in electricity markets has attracted widespread attention. Reinforcement learning (RL) has been widely used for electricity market bidding as a powerful artificial intelligence (AI) tool to make decisions under real-world uncertainties. However, current RL-based bidding methods mostly employ low-dimensional bids (LDBs), which significantly diverge from the N price-power pairs commonly used in current electricity markets. The N-pair bid format is denoted as high-dimensional bid (HDB) format, which has not been fully integrated into the existing RL-based bidding methods. The loss of flexibility of current RL-based bidding methods could greatly limit the bidding profits and make it difficult to address the increasing uncertainties caused by renewable energy generation. In this paper, we propose a framework for fully utilizing HDBs in RL-based bidding methods. First, we employ a special type of neural network called the neural network supply function (NNSF) to generate HDBs in the form of N price-power pairs. Second, we embed the NNSF into a Markov decision process (MDP) to make it compatible with most existing RL algorithms. Finally, the experiments on energy storage systems (ESSs) in the Pennsylvania-New Jersey-Maryland (PJM) real-time electricity market show that the proposed bidding method with HDBs can increase the bidding flexibility, thereby increasing the profits of state-of-the-art RL-based bidding methods.

Abstract:Driven by increasing penetration of intermittent renewable energy generation, modern power systems are promoting the integration of energy storage (ES) and advocating high-resolution dynamic security constrained optimal power flow (DSCOPF) models to exploit ES time-shifting flexibility against contingencies and respond promptly to more frequent variations in the system operating status. While pioneering research works explore different methods to solve security constrained optimal power flow (SCOPF) problems at individual time steps, real-time implementation of DSCOPF still faces challenges associated with uncertainty adaptation, complex constraint satisfaction， and computational efficiency. This paper proposes a physics-guided safe policy learning method, featuring an analytical evaluation model to provide both accurate safety and cost-efficiency evaluations. A primal-dual-based learning procedure is developed to guide policy learning, fostering prompt convergence. A spatial-temporal graph neural network is constructed to enhance perception on the spatial-temporal uncertainties and leverage policy generalization. Case studies validate the effectiveness and scalability of the proposed method in safety, cost-efficiency, and computational performance and highlight the value of enhanced perception on IEEE 39-bus and 118-bus test systems.

Abstract:Active distribution network (ADN), as a typically cyber-physical system, develops with the evolution of Internet of Things (IoTs), which makes the network vulnerable to cybersecurity threats. In this paper, the eavesdropping attacks that lead to privacy breaches are addressed for the IoT-enabled ADN. A privacy-preserving energy management system (EMS) is proposed and empowered by secure data exchange protocols based on the homomorphic cryptosystem. During the information transmission among distributed generators and load customers in the EMS, private information including power usage and electricity bidding price can be effectively protected against eavesdropping attacks. The correctness of the final solutions, e.g., optimal market clearing price and unified power utilization ratio, can be deterministically guaranteed. The simulation results demonstrate the effectiveness and the computational efficiency of the proposed homomorphically encrypted EMS.

Abstract:The increasing integration of renewable energy sources (RESs) presents significant challenges for the safe and economical operation of power grids. Addressing the critical need to assess the effect of RES uncertainties on optimal scheduling schemes (OSSs), this paper introduces a convex hull based economic operating region (CH-EOR) for power grids. The CH-EOR is mathematically defined to delineate the impact of RES uncertainties on power grid operations. We propose a novel approach for generating the CH-EOR, enhanced by a big-M preprocessing method to improve the computational efficiency. Performed on four test systems, the proposed big-M preprocessing method demonstrates notable advancements: a reduction in average operating costs by over 10% compared with the box-constrained operating region (BC-OR) derived from robust optimization. Furthermore, the CH-EOR occupies less than 11.79% of the generators ’ adjustable region (GAR). Most significantly, after applying the proposed big-M preprocessing method, the computational efficiency is improved over 17 times compared with the traditional big-M method.

Abstract:With the goal of low-carbon energy utilization, electric vehicles (EVs) and EV charging stations (EVCSs) are becoming increasingly popular. The economical operation strategy is always a primary concern for EVCSs, while users ’ behavior and operating data leakage problems in EVCSs have not been taken seriously. Herein, federated deep reinforcement learning, a privacy-preserving method, is applied to learn the optimal strategy for multiple EVCSs. However, it is prone to Byzantine attacks. It is urgent to achieve an economical operation strategy while preserving data privacy and defending against Byzantine attacks. Therefore, this paper proposes a Byzantine-resilient federated deep reinforcement learning (BR-FDRL) method to address these problems. First, the distributed EVCS data are utilized by the federated deep reinforcement learning to train an economical operation strategy while preserving privacy by only transmitting gradients. The sampling efficiency is enhanced by both federated learning and stochastically controlled stochastic gradient. Then, the Byzantine-resilient gradient filter (BRGF) designs two distance rules to keep malicious gradients out. The case study verifies the effectiveness of the proposed BRGF in resisting Byzantine attacks and the effectiveness of federated deep reinforcement learning in improving convergence speed and reward and preserving privacy. The resluts show that the BR-FDRL method minimizes the operation cost by an average of 35% compared with the rule-based method while meeting the state of charge demand as much as possible.