Abstract
With the deteriorating effects resulting from global warming in many areas, geographically distributed data centers contribute greatly to carbon emissions, because the major energy supply is fossil fuels. Considering this issue, many geographically distributed data centers are attempting to use clean energy as their energy supply, such as fuel cells and renewable energy sources. However, not all workloads can be powered by a single power sources, since different workloads exhibit different characteristics. In this paper, we propose a fine-grained heterogeneous power distribution model with an objective of minimizing the total energy costs and the sum of the energy gap generated by the geographically distributed data centers powered by multiple types of energy resources. In order to achieve these two goals, we design a two-stage online algorithm to leverage the power supply of each energy source. In each time slot, we also consider a chance-constraint problem and use the Bernstein approximation to solve the problem. Finally, simulation results based on real-world traces illustrate that the proposed algorithm can achieve satisfactory performance.
GEOGRAPHICALLY distributed data centers contribute greatly to high energy consumption and raise concerns about their environmental impact. For example, Google and Microsoft [
Given the increasing pressure from energy limitations and the deterioration of the climate, a growing number of green data centers have been deployed to mitigate the above challenges in two ways. One approach is to reduce power costs by increasing energy efficiency [
Though using heterogeneous energy supplies may result in substantial energy efficiency and significant environmental benefits, some challenges remain in architecture design, capacity planning, and energy management strategies, which have been investigated in many researches [
In order to solve the above problems, a well-grained heterogeneous power distribution model is proposed to leverage the different characteristics of multiple types of energy resources to dynamically supply power to the geographically distributed data centers. In order to obtain a better balance between cost and performance, we further propose an online power management algorithm, which includes two phases: ① energy management in a single time slot; ② energy management in a long period. In a single time slot, multiple types of energy sources are distributed in the energy manage phase by considering the demand from data centers as a random variable. A chance-constrained optimization technique is applied, which requires little information about the demand. In a long period, we propose an online greedy distribution algorithm to leverage the power supply of each energy source in every time slot in the energy management phase.
The major contributions of our work are summarized as follows.
1) We formulate a fine-grained heterogeneous power distribution model with an objective of minimizing the total energy costs and the sum of the energy gap generated by the geographically distributed data centers powered by multiple types of energy resources. To guarantee better operation performance, we introduce an opportunity constraint to ensure the probability that the workload demands exceed the energy supplies within a small threshold.
2) We present a two-stage online algorithm to efficiently solve the formulated problem to distribute the different power sources to various workloads.
3) An evaluation simulation with real-world data center traces validates the effectiveness and feasibility of our proposal and shows that our proposed algorithm can achieve good performance.
The remainder of this paper is organized as follows. Section II presents the background and motivation. In Section III, we introduce the system model and propose an online algorithm. The evaluation methodology and experimental results are given in Sections IV and V. Finally, Section VI concludes the paper.
The energy cost for data centers has been increasingly concerned in recent years. Various green energy sources are investigated and applied to data center systems as alternative energy supplies [

Fig. 1 Characteristics of different energy resources. (a) Power grid. (b) Renewable energies. (c) Storage. (d) Fuel cells.
Renewable energy sources provide a priority for Internet technology (IT) corporations to power rapidly expanding data center infrastructures.
Energy storage devices are used to reduce the peak power demand in data centers. They can charge and discharge energy whenever there is a demand for different types of energy, as shown in
Fuel cells have emerged as a promising energy source for data centers due to their advantages of high energy efficiency, high reliability, and low carbon dioxide emission [
Modern data centers often provide a variety of web services, and the power usage patterns of different services are also heterogeneous, as shown in [
Various energy sources have produced substantial concerns in power systems, and the use of multiple energy resources has been applied in many fields. Reference [
For the energy cost minimization of data centers, many studies have been carried out by different approaches. Reference [
In this section, we consider an energy management problem for multi-energy-source data centers in a single time slot. The energy management model is shown in

Fig. 2 Energy management model of multi-energy-source data center.
There are many different kinds of workloads in data centers, some of which are delay-sensitive, such as web services. The others are delay-tolerant, such as simulations and MapReduce jobs. These delay-tolerant workloads can be scheduled to run at any time as long as the jobs are completed before a maximum completion time, i.e., there is a maximum completion time which may be 1 hour. Therefore, in this paper, we consider the delay-tolerant workloads. In addition, the effect of workload migration is ignored, because this paper concentrates on the energy cost of mismatched demand and supply. The cost produced by workload migration is relatively small, so it is ignored.
The amount of input workload requested in data center is denoted by The proportion of the incoming requests transmitted from data center to data center is . The number of input requests loaded from data center to data center is given as:
(1) |
where is a non-decreasing function; and J is the number of data centers. In this paper, we consider a linear function that has been adopted by existing works [
After receiving the input workload, the data center should provide adequate energy supply to conduct data processing. Four kinds of energy sources can be used in this model, which exhibit different energy response curves, as analyzed in Section II. Then, we define (t), (t), (t), and (t) as the amounts of energy for the power grid, battery, fuel cell, and renewable energy sources, respectively. For each energy source i, we can obtain:
(2) |
The above constraints mean that each data center can be powered by four kinds of energy sources, and each data center can be powered by one or more energy sources in the same time slot.
Different from the conventional data center system, there are multiple types of energy sources. Therefore, a single electricity pricing rule cannot be applied in this case. For example, a renewable energy source has the lowest electricity price, but the price is not stable. Because the renewable energy resource price changes over time. We denote as the electricity price for renewable energy sources in data center in time slot . A lot of studies concentrate on the prediction of electricity pricing for renewable energy sources [
(3) |
For the power grid and battery system, we adopt the TOU electricity pricing policy [
(4) |
where is the TOU electricity price of the power grid.
Let be the energy level of the battery at data center in time slot t. Then, is bounded by its maximum capacity , i.e.,
(5) |
The dynamics of can be expressed by
(6) |
where and are the charging and discharging energy, respectively, which are bounded by (7) and (8).
(7) |
(8) |
where and are the maximum charging and discharging energy, respectively.
The charging energy of the batteries is the energy production. Therefore, the cost of the battery system in each time slot can be expressed by
(9) |
where is the charging/discharging rate of the battery system.
For fuel cells, we use as the price of fuel cells. (t) is the energy supply from fuel cells, and the cost of fuel cell generation can be represented as
(10) |
As [
(11) |
Our ideal objective is to match the energy demand and supply in each time slot. In fact, it is hard to accurately match the energy consumption and supply when a job arrives, because the energy demand cannot be known in advance. In addition, renewable energy is uncertain in each time slot, and the price of this kind of energy source also strongly fluctuates. Therefore, the matching between the energy demand and supply in each time slot is just an ideal objective. In order to address this problem, we model both the energy demand and the cost of renewable generation as random variables whose expectation can be acquired through an analysis of historical information. The chance constraint can be written as
(12) |
where is the probability function; and is the arbitrarily small value.
By considering the following constraints, the minimization problem (abbreviated as P1) is described by (2) to (4), (7),and (9) to (16).
(13) |
s.t.
(14) |
(15) |
(16) |
where I is the number of energy sources; and and are the maximum amounts of energy from power grid and fuel cells, respectively.
Our objective is then to minimize the total energy cost of the multi-energy-source supply system. As the existing convex optimization solutions cannot solve the above constraint (12) directly, the above non-convex optimization problem is a nondeterministic polynomial (NP)-hard problem.
In this subsection, we address the challenge of the chance constraint (12) of P1 and consider (t) as a random variable. Suppose that the distribution of (t) is bounded within and the distribution of is bounded within . By defining /2 and /2, (t) can be normalized within as follows.
(17) |
Similarly, by defining and , can be normalized within as follows.
(18) |
Additionally, let and . The chance constraint can be equivalently written as
(19) |
According to the Bernstein approximation, the constraint can be approximated by
(20) |
where is the weight.
(21) |
where can be expressed as (22).
(22) |
According to [
(23) |
The in the above constraint can be removed by substituting , so that (23) can be equivalently written as
(24) |
Finally, a mixed-integer linear programming formulation for the optimization problem (abbreviated as P2) is described by (2) to (4), (7), (9) to (13), (15), and (24).
In this subsection, we will introduce the algorithm design for a long period. In Section III, the proportion of the incoming requests are obtained through
The energy supplies of both fuel cells and batteries are related to the energy supplies in the last time slot based on the limitation of the charging/discharging rate and slow flowing behaviour. The dynamics of can be expressed as
(25) |
where is the energy supply changes of the fuel cells. The lower bound and upper bound of are denoted as and , respectively, i.e.,
(26) |
For is obtained by
(27) |
(28) |
Let denote the energy demand by the workloads of data center in time slot . The relation between and can be expressed as
(29) |
The objective of
(30) |
s.t.
(31) |
(32) |
(33) |
(34) |
(35) |
where is the sum of the energy supplies from the four kinds of energy sources. Constraints (32) and (33) stand for the energy gap when the energy demand exceeds the energy supply, and the energy gap when the energy supply exceeds the energy demand, respectively. means that the cost of the case in which demand exceeds supply is more significant than that of another case. Similarly, means that supply exceeding demand is more important. As a result, the objective function and constraints are all linear, and the P3 is easy to solve offline. However, the future energy demand cannot be obtained in advance. Therefore, we design an online algorithm to solve this problem, as shown in
In Section III, we have proposed two minimization problems to realize two different objectives: ① the minimization of the total energy consumption with a minimized gap between supply and demand; ② the minimization of the total electricity cost, as shown in P1 and P3. According to the two formulations, the results in P3 are related to the time slots while the results in P1 are unrelated to the time slots. Therefore, the operations in P1 occur in single time slot while the operations in P3 are done from one time slot to next time slot. In addition, we also have found that some input variables of

Fig. 3 Relationship between two algorithms.
Suppose that we have observed a group of initial values at time slot : . Then we perform
Carbon usage effectiveness (CUE) [
(36) |
where is the electricity generated from energy type ; and is the carbon emission rate of energy type .
In the next section, we carry out several sensitivity analyses for the key parameters of our proposed model to investigate the relationship between different parameters and their effects on the final results. In our proposed model, we have two different optimization objectives, including the energy cost and the gap between demand and supply. In order to better obtain a clear comparison between the different pairs of parameters, we introduce a new optimization objective named Tcost, which is:
(37) |
where gap is a type of energy loss. The factor 17 in (37) means that, as the gap between demand and supply is a type of energy loss, we can use the maximum electricity price as the price of the energy gap.
The other key experimental settings are given in
We use five different kinds of workload traces collected from the Wiki data center, which are shown in

Fig. 4 Simulation dataset of workload. (a) Workload A. (b) Workload B. (c) Workload C. (d) Workload D. (e) Workload E.
In the experiments, the length of each time slot is set to be 1 hour. The five traces show different characteristics. We also use solar energy generation and wind energy production as our renewable energy traces, which are shown in

Fig. 5 Simulation dataset of renewable sources. (a) Electricity price of renewable energy for a month. (b) Solar energy production for a month. (c) Wind energy production for a month. (d) Electricity price for a day.
1) GSEr: including the power grid, battery, fuel cells, and renewable energy.
2) GSE: including the power grid, battery, and fuel cells.
3) GEr: including the power grid, fuel cells, and renewable energy.
4) GSr: including the power grid, battery, and renewable energy.
5) GS: including the power grid and the battery.
6) G: only including the power grid.
There are various important parameters in the proposed system model. As discussed in Section III, different amounts of power are generated from the four types of energy sources at each site. Different values of these parameters will have a significant effect on the optimization results. For instance, expanding the maximum value of the battery capacity or fuel cell capacity may decrease or increase the energy gap, because of the slow following behaviours of the charging or discharging rate of the battery and the power changing rate of the fuel cells. In addition, different energy compositions will also have a large influence on the final optimization results, because they will affect the proportion of energy sources and then influence the energy cost. In this section, we analyze the impact of and on different energy composition, study how the two parameters impact the energy cost and try to find the optimal composition of parameters.
For these experiments, we consider a configuration with five data centers executing a hybrid workload. The experiments analyze the total cost of the system energy with different parameter compositions for the maximum capacities of the battery and fuel cells. Other implementation-related factors are ignored, because this study only focuses on sensitivity analysis.
In the following subsections, we present sensitivity analysis of our proposed system from three aspects: the relationship between and , the relationship between and , and the relationship between and .
The maximum capacities of the battery and fuel cells are sampled from 10 kWh to 40 kWh. Furthermore, we consider a simulation of GSRr for each set of values and calculate Tcost as shown in
As discussed in Sections IV and V, the charging or discharging rate of the battery we are considering in the proposed system model has an immediate impact on the energy cost. The workload at a particular location changes over time by a large margin, and it can even reach hundreds of times. is set to determine the capacity to adapt to the workload changes of the battery. at different values has different energy costs as shown in
and are sampled from two value ranges, where is set as 20 kWh to 50 kWh and is set as 1 kW to 10 kW. When is small (), the variation of has no effect on the Tcost. When is 3 kWh and 4 kWh, the limited load following is unable to adapt to significant changes in energy demand. When is larger than 4 kWh, The changes of the energy supply for fuel cells are enough to cope with the changes of energy demand. If we fix the value of , a minimum value of Tcost can be obtained when is set as 30 kWh to 50 kWh.
As discussed in Sections IV and V, the charging or discharging rate of the battery has an immediate impact on the energy cost. The workload at a particular site changes over time by a large margin, and it can even reach hundreds of times. is set to determine the capacity to adapt to workload changes of the battery. at different values has different energy costs as shown in
In this subsection, we evaluate the benefits of green data centers powered by multi-composition energy sources. To be more specific, we compare the energy cost among six kinds of power compositions, which are summarized in Section VI and shown in

Fig. 6 Experimental results of three performance metrics. (a) Comparison of energy cost. (b) Comparison of Tcost. (c) Comparison of carbon emission rate.
When the renewable energy is wind energy, the relationship among the values of the energy cost is . The major difference between the two relationships is the value of GEr; the energy generation of wind energy is much higher than that of solar energy, which is shown in
When the renewable energy is solar energy, the data centers always need to be powered by the power grid and fuel cells. However, the energy variation is limited by the fuel cells. The main energy supply should be the power grid. As shown in

Fig. 7 Energy proportions. (a) Solar-GSEr. (b) Solar-GSE. (c) Solar-GSr. (d) Solar-GEr. (e) Solar-GS. (f)Wind-GSEr. (g) Wind-GSE. (h) Wind-GSr. (i) Wind-GEr. (j) Wind-GS.
In summary, our power management policies can achieve the lowest energy cost. The evaluation results show that our management policy, compared with the GSE policy, GEr policy, GSr policy, and GS policy, can improve the energy cost on average by 4.77%, 118.87%, 21.49%, and 120.87%, respectively.
The energy gap is a necessary evaluation index in several power management policies. In this subsection, we use to estimate the effect on the energy gap, which is shown in
In summary, our power management policies can achieve the lowest . When we do not use renewable energy in GSE, there is little impact on the energy cost. However, renewable energy has a much larger impact on . On the contrary, when we do not use fuel cells in GSr, there is little impact on . However, the fuel cells have a much larger impact on the energy cost. The evaluation results show that our management policies, compared with the GSE policy, GEr policy, GSr policy, and GS policy, can improve the energy cost on average by 33.55%, 93.39%, 89.85%, and 4.61%, respectively.
We further compare the carbon emission rate based on our proposed hybrid power management policies with other baseline power management strategies.
In summary, our power management policies can achieve the lowest carbon emission rate. The experimental results show that our power management policies, compared to the GSE, GEr, GSr, and GS policies, can improve the carbon emission rate on average by 32.80%, 39.45%, 18.59%, and 83.02%, respectively.
In this paper, we first establish a system model of minimizing the energy cost of data centers powered by heterogenous energy resources, such as power grid, fuel cells, energy storage devices, and renewable energy sources. Then, we formulate a problem in a single time slot to minimize the total energy cost of data centers powered by four types of energy sources. Moreover, we also formulate another problem in a long period to mitigate the energy gap between the workload and energy supply. To solve a chance-constraint problem in the former formulated problem, we design an online control algorithm by using the Bernstein approximation. We also design a greedy online control algorithm to solve the latter formulated problem. Finally, by using two realistic traces, we conduct several sensitivity analyses of the impacts on various parameters of the system model and compare three key characteristics of different energy supplies powered by different energy sources compositions. It is observed that the proposed heterogeneous energy supply model can achieve similar results to other compositions.
References
A. Qureshi, “Power-demand routing in massive geo-distributed systems,” Ph.D. dissertation, Massachusetts Institute of Technology, Cambridge, 2010. [百度学术]
C. Ren, D. Wang, B. Urgaonkar et al., “Carbon- aware energy capacity planning for datacenters,” in Proceedings of IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, Washington DC, USA, Aug. 2012, pp. 391-400. [百度学术]
W. Deng, F. Liu, H. Jin et al., “Harnessing renewable energy in cloud datacenters: opportunities and challenges,” IEEE Network, vol. 28, no. 1, pp. 48-55, Jan. 2014. [百度学术]
L. Zhao, J. Brouwer, S. James et al., “Servers powered by a 10 kW in-rack proton exchange membrane fuel cell system,” in Proceedings of International Conference on Asme International Conference on Fuel Cell Science, Boston, USA, Jun.-Jul. 2014, pp. 1-8. [百度学术]
R. Tripathi, S. Vignesh, and V. Tamarapalli, “Optimizing green energy, cost, and availability in distributed data centers,” IEEE Communications Letters, vol. 21, pp. 500-503, Mar. 2017. [百度学术]
X. Hu, P. Li, K. Wang et al., “Energy management of data centers powered by fuel cells and heterogeneous energy storage,” in Proceedings of 2018 IEEE International Conference on Communications (ICC), Kansas City, USA, May 2018, pp. 1-6. [百度学术]
K. Wang, J. Yu, Y. Yu et al., “A survey on energy internet: architecture, approach and emerging technologies,” IEEE Systems Journal, vol. 12, no. 3, pp. 2403-2416, Sept. 2018. [百度学术]
L. Liu, H. Sun, C. Li et al., “Exploring customizable heterogeneous power distribution and management for datacenter,” IEEE Transactions on Parallel and Distributed Systems, vol. 29, pp. 2798-2813, Dec. 2018. [百度学术]
F. Koushanfar and A. Mirhoseini, “Hybrid heterogeneous energy supply networks,” in Proceedings of IEEE International Symposium on Circuits and Systems, Rio de Janeiro, Brazil, May 2011, pp. 2489-2492. [百度学术]
C. Xu, K. Wang, P. Li et al., “Renewable energy-aware big data analytics in geo-distributed data centers with reinforcement learning,”IEEE Transactions on Network Science and Engineering, vol. 7, no. 1, pp. 205-215, Jan. 2020. [百度学术]
A. Energiewende. (2018, Dec.). Power prices, generation and consumption. [Online]. Available: https://www.agora-energiewende.de/en/service/recent-electricity-data/chart/power generation price/04.08.2018/03.09.2018/ [百度学术]
M. A. Islam, X. Ren, S. Ren et al., “A spot capacity market to increase power infrastructure utilization in multi-tenant data centers,” in Proceedings of the 2017 ACM SIGMETRICS/International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS ’17, New York, USA, Jun. 2017, pp. 19-20. [百度学术]
Q. Wu, Q. Deng, L. Ganesh et al., “Dynamo: Facebook’s data center-wide power management system,” in Proceedings of 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), Seoul, South Korea, Jun. 2016, pp. 469-480. [百度学术]
B. Aksanli, “Data center peak power management with energy storage devices,” IEEE Internet Computing, vol. 21, no. 4, pp. 26-33, Jul. 2017. [百度学术]
N. Kalhoff, “Integration of fuel cell applications into the power supply for information and telecommunications technology,” in Proceedings of 29th International Telecommunications Energy Conference, Rome, Italy, Sept.-Oct. 2007, pp. 444-448. [百度学术]
H. Hsu, Q. Deng, J. Mars et al., “Smooth operator: reducing power fragmentation and improving power utilization in large-scale data centers,” in Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems, New York, USA, Mar. 2018, pp. 535-548. [百度学术]
H. Chen, P. Xuan, Y. Wang et al., “Key technologies for integration of multitype renewable energy sources research on multi-time frame robust scheduling/dispatch,” IEEE Transactions on Smart Grid, vol. 7, no. 2, pp. 471-480, Jan. 2016. [百度学术]
S. Clegg and P. Mancarella, “Integrated electrical and gas network flexibility assessment in low-carbon multi-energy systems,” IEEE Transactions on Sustainable Energy, vol. 7, no.2, pp. 718-731, Apr. 2016. [百度学术]
A. Eladl, M. El-Afifi, and M. El-Saadawi, “Optimal power dispatch of multiple energy sources in energy hubs,” in Proceedings of 2017 Nineteenth International Middle East Power Systems Conference (MEPCON), Cairo, Egypt, Dec. 2017, pp. 1053-1058. [百度学术]
H. Zhang, Y. Li, D. W. Gao et al., “Distributed optimal energy management for Energy Internet,” IEEE Transactions on Industrial Informatics, vol. 13, pp. 3081-3097, Dec. 2017. [百度学术]
Y. Guo, Y. Gong, Y. Fang et al., “Energy and network aware workload management for sustainable data centers with thermal storage,”IEEE Transactions on Parallel and Distributed Systems, vol. 25, no. 8, pp. 2030-2042, Aug. 2014. [百度学术]
M. Jawad, M. Qureshi, U. Khan et al., “A robust optimization technique for energy cost minimization of cloud data centers,” IEEE Transactions on Cloud Computing. DOI: 10.1109/TCC.2018.2879948 [百度学术]
S. Kwon, L. Ntaimo, and N. Gautam, “Demand response in data centers: integration of server provisioning and power procurement,” IEEE Transactions on Smart Grid, vol. 10, no. 5, pp. 4928-4938, Sept. 2019. [百度学术]
N. Hogade, S. Pasricha, H. Siegel et al., “Minimizing energy costs for geographically distributed heterogeneous data centers,” IEEE Transactions on Sustainable Computing, vol. 3, no. 1, pp. 318-331, Oct. 2018. [百度学术]
L. Yu, T. Jiang, Y. Cao et al., “Carbon-aware energy cost minimization for distributed internet data centers in smart microgrids,” IEEE Internet of Things Journal, vol. 1, no. 3, pp. 255-264, Jun. 2014. [百度学术]
H. Wu, M. Shahidehpour, A. Alabdulwahab et al., “Demand response exchange in the stochastic day-ahead scheduling with variable renewable generation,” IEEE Transactions on Sustainable Energy, vol. 6, no. 1, pp. 516-525, Apr. 2015. [百度学术]
K. Wang, L. Gu, S. Guo et al., “Distributed energy management for vehicle-to-grid networks,” IEEE Network, vol. 31, no. 2, pp. 22-28, Mar. 2017. [百度学术]
K. Wang, H. Li, and Y. Feng, “Big data analytics for system stability evaluation strategy in the Energy Internet,” IEEE Transactions on Industrial Informatics, vol. 13, no. 4, pp. 1969-1978, Aug. 2017. [百度学术]
A. Nemirovski and A. Shapiro, “Convex approximations of chance constrained programs,” SIAM Journal on Optimization, vol. 17, no. 4, pp. 969-996, Dec. 2006. [百度学术]
X. Gao, A. Curtis, B. Wong et al., “It’s not easy being green,” ACM SIGCOMM Computer Communication Review, vol. 42, no. 1, pp. 211-222, Aug. 2012. [百度学术]
C. Ren, D. Wang, B. Urgaonkar et al., “Carbon-aware energy capacity planning for datacenters,” in Proceedings of 2012 IEEE 20th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, Washington DC, USA, Aug. 2012, pp. 391-400. [百度学术]
Wiki. (2018,Dec.). Wiki dump data. [Online]. Available: http://dumps.wikimedia.org/other/pagecounts-raw [百度学术]
Agora Enegiewende. (2018, Dec.). Source code: piceang-for-hres (MATLAB) optimal design of hybrid renewable energy systems using multi-objective evolutionary algorithm. [Online]. Available: http://ruiwangnudt.gotoip3.com/optimization.htmltdsourcetag=spcqqaiomsg [百度学术]