Journal of Modern Power Systems and Clean Energy

ISSN 2196-5625 CN 32-1884/TK

网刊加载中。。。

使用Chrome浏览器效果最佳,继续浏览,你可能不会看到最佳的展示效果,

确定继续浏览么?

复制成功,请在其他浏览器进行阅读

Minimizing Energy Cost for Green Data Center by Exploring Heterogeneous Energy Resource  PDF

  • Xiaoxuan Hu
  • Peng Li
  • Yanfei Sun
School of Automation, Nanjing University of Posts and Telecommunications, Nanjing 210003, China; School of Computer Science and Engineering, University of Aizu, Aizuwakamatsu 965-8580, Japan; School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing 210003, China

Updated:2021-01-19

DOI:10.35833/MPCE.2019.000052

  • Full Text
  • Figs & Tabs
  • References
  • Authors
  • About
OUTLINE

Abstract

With the deteriorating effects resulting from global warming in many areas, geographically distributed data centers contribute greatly to carbon emissions, because the major energy supply is fossil fuels. Considering this issue, many geographically distributed data centers are attempting to use clean energy as their energy supply, such as fuel cells and renewable energy sources. However, not all workloads can be powered by a single power sources, since different workloads exhibit different characteristics. In this paper, we propose a fine-grained heterogeneous power distribution model with an objective of minimizing the total energy costs and the sum of the energy gap generated by the geographically distributed data centers powered by multiple types of energy resources. In order to achieve these two goals, we design a two-stage online algorithm to leverage the power supply of each energy source. In each time slot, we also consider a chance-constraint problem and use the Bernstein approximation to solve the problem. Finally, simulation results based on real-world traces illustrate that the proposed algorithm can achieve satisfactory performance.

I. Introduction

GEOGRAPHICALLY distributed data centers contribute greatly to high energy consumption and raise concerns about their environmental impact. For example, Google and Microsoft [

1] have paid tens of millions of dollars for electricity, and 50 tons of carbon dioxide a year have been produced [2] due to the high power consumption.

Given the increasing pressure from energy limitations and the deterioration of the climate, a growing number of green data centers have been deployed to mitigate the above challenges in two ways. One approach is to reduce power costs by increasing energy efficiency [

3], e.g., dynamic voltage and frequency scaling. However, this approach only can reduce energy consumption and carbon emissions within a limited range. Another approach is to use alternative green energy resources, such as solar, wind, and fuel cells [4]. However, the intermittent nature of renewable energy and the high temperature of fuel cells have hindered their widespread use. In combination with the above two approaches, several recent efforts have been made to try to achieve a balance between steady performance and environmental protection for geographically distributed data centers [5], [6]. Nevertheless, only using one or two energy sources is not ideal in practice. For instance, Table I [8]-[10] shows the different characteristics of four energy resources as power supplies. No single type of energy source can satisfy all the operating performance requirements, and these energy sources complement each other. Therefore, the limitations of efficiency, reliability, economics, and environmental friendliness of a specific type of energy source can be resolved by effectively exploring heterogeneous energy resources.

Table I Characteristics of Energy Resources
Energy resourceCostResponse timeGreenCapacityStability
Power grid High High Low High High
Battery Low High Middle Low High
Fuel cell Middle Low High Middle High
Renewable energy Low, free Middle High Middle Low

Though using heterogeneous energy supplies may result in substantial energy efficiency and significant environmental benefits, some challenges remain in architecture design, capacity planning, and energy management strategies, which have been investigated in many researches [

7]. In addition, heterogeneous workloads and energy supplies make it difficult to match the demands and supplies. Recently, few work has been investigated in this field. A distributed energy system was designed for data centers to deliver an appropriate power supply to different workloads [8]. In this paper, an artificial neural network (ANN) was used to classify four energy sources by the demanded performance metrics. The proposed system provides a promising option for exploring heterogeneous energy for data centers, but it still requires much information about the performance metrics of the incoming workloads, such as utility power cost, renewable utilization, and workload performance, which is the total completion time of the workloads. In fact, information on the workloads cannot be predicted in advance in many cases. In addition, the workloads and power supplies are often changed over time, and it is difficult to match them well in each time slot.

In order to solve the above problems, a well-grained heterogeneous power distribution model is proposed to leverage the different characteristics of multiple types of energy resources to dynamically supply power to the geographically distributed data centers. In order to obtain a better balance between cost and performance, we further propose an online power management algorithm, which includes two phases: ① energy management in a single time slot; ② energy management in a long period. In a single time slot, multiple types of energy sources are distributed in the energy manage phase by considering the demand from data centers as a random variable. A chance-constrained optimization technique is applied, which requires little information about the demand. In a long period, we propose an online greedy distribution algorithm to leverage the power supply of each energy source in every time slot in the energy management phase.

The major contributions of our work are summarized as follows.

1) We formulate a fine-grained heterogeneous power distribution model with an objective of minimizing the total energy costs and the sum of the energy gap generated by the geographically distributed data centers powered by multiple types of energy resources. To guarantee better operation performance, we introduce an opportunity constraint to ensure the probability that the workload demands exceed the energy supplies within a small threshold.

2) We present a two-stage online algorithm to efficiently solve the formulated problem to distribute the different power sources to various workloads.

3) An evaluation simulation with real-world data center traces validates the effectiveness and feasibility of our proposal and shows that our proposed algorithm can achieve good performance.

The remainder of this paper is organized as follows. Section II presents the background and motivation. In Section III, we introduce the system model and propose an online algorithm. The evaluation methodology and experimental results are given in Sections IV and V. Finally, Section VI concludes the paper.

II. Motivation and Related Work

A. Motivation

The energy cost for data centers has been increasingly concerned in recent years. Various green energy sources are investigated and applied to data center systems as alternative energy supplies [

9], [10]. Different kinds of energy sources possess different characteristics. We have divided these characteristics into four major types and each type of characteristics can be represented as one or more energy sources. Figure 1 shows the characteristics of four kinds of energy sources.

Fig. 1 Characteristics of different energy resources. (a) Power grid. (b) Renewable energies. (c) Storage. (d) Fuel cells.

Figure 1(a) shows the energy production from conventional power plants, which are aggregated by hard coal, lignite, and gas cogeneration power plants. This production data, which was collected during the year 2016 from conventional power plants, depends on the months of the year [

11]. The lowest power is used in April, while the highest power is used in February. Although the amount of energy production in each month differs, the energy production from the power grid can be considered as a fixed value in a certain period of time (one week or more). When the data centers are powered by a conventional grid, the maximum workload cannot exceed the capacity of the power plants. Due to the stable energy supply from the power grid, the power infrastructures of geographically distributed data centers remain largely under-utilized [12]. Only 85% of the capacity on average can be used, such as in the case of Facebook [13]. The reason for this under-utilization is that the power infrastructure of data center is configured to maintain a high power supply to avoid frequent emergencies caused by overloading that may be harmful to the reliability of data centers.

Renewable energy sources provide a priority for Internet technology (IT) corporations to power rapidly expanding data center infrastructures. Figure 1(b) includes wind onshore fleet electricity, wind offshore fleet electricity, and solar fleet electricity. The energy produced from renewable sources changes over time and cannot achieve regularity. Therefore, renewable energy supplies are suitable for delay-sensitive non-flexible applications (e.g., web browsing) and delay-tolerant flexible applications (e.g., scientific computing). These IT workloads are sent to geographical data centers to benefit from the location diversity of the different characteristics of renewable energy sources. Moreover, the flexibility of IT workloads can solve the problem of intermittent renewable energy through workload migration.

Energy storage devices are used to reduce the peak power demand in data centers. They can charge and discharge energy whenever there is a demand for different types of energy, as shown in Fig. 1(c). The power consumption of geographically distributed data centers may have a variety of characteristics due to the changing nature of the workload requests. In this case, energy storage is needed to satisfy the different peak requirements. For instance, some energy storage devices, such as long-term flywheels and high-energy capacitors, are suitable for a short peak duration. Because they have a high power density, a fast recharging time, and almost unlimited charging and discharging cycles. Some energy storage devices, such as batteries, can sustain power longer, which are appropriate for a longer peak duration. These devices are used as standby uninterruptible power supplies (UPSs), which can maintain the power supply of the whole system for a period of time when needed [

14].

Fuel cells have emerged as a promising energy source for data centers due to their advantages of high energy efficiency, high reliability, and low carbon dioxide emission [

4]. Although fuel cells show many advantages, they are slow in changing the output power, i.e., it may take several seconds or even minutes to reach the desired demand, as shown in Fig. 1(d). Therefore, fuel cells are useful as a second redundant energy source for relatively longer peak intervals. If a malfunction or maintenance occurs, the redundant units supply the energy needed for assuring uninterrupted operation [15].

Modern data centers often provide a variety of web services, and the power usage patterns of different services are also heterogeneous, as shown in [

16]. As mentioned above, different workloads can be matched with different types of energy supplies, and each energy has its advantages and disadvantages. For example, despite the fact that the renewable power sources are cheap and green, the nature of intermittent make them unsuitable for delay-sensitive tasks. However, they may be applied to power delay-tolerant jobs. Thus, only one type of energy supply cannot provide a “one-size-fits-all” solution. Recognizing such opportunities, we propose a geographically distributed data center network powered by heterogeneous energy sources in order to mitigate the power budget problem with high environmentally friendliness. The design of the system is introduced in Section III.

B. Multi-source Generation

Various energy sources have produced substantial concerns in power systems, and the use of multiple energy resources has been applied in many fields. Reference [

17] proposed a new architecture of a multiple time slot dispatch system by using multiple types of renewable energy sources in smart grid. Reference [18] proposed a method to quantify the flexibility of a gas network considering the constraints of gas network and different heating schemes. Based on the proposed quantitative methods, a holistic multi-energy system was assessed to explore the impact of the gas network infrastructure on power system generation. Reference [19] proposed a general modelling method and an optimization solution for energy dispatching and conversion in power systems with various energy carriers. Reference [20] presented a novel energy management framework for energy Internet with various energy sources. Researchers seldom pay more attention to the energy management of data centers powered by heterogeneous energy resources. However, the specific workload could not be well matched with a particular energy source. Therefore, we focus on fine-grained energy management for data centers powered by heterogenous energy resources. In each time slot, the incoming workload can be matched well with an optimal composition of different energy supplies.

C. Energy Cost Minimization in Data Centers

For the energy cost minimization of data centers, many studies have been carried out by different approaches. Reference [

21] proposed an online control algorithm based on the Lyapunov optimization technique to reduce the cost of energy usage considering geographical workload balance, delay-tolerant workloads scheduling, and thermal storage management in data centers. Reference [22] presented an intelligent energy management system based on a robust energy cost optimization algorithm for data centers. Power demand, electricity price, and renewable power generation effects were considered for optimizing the total power consumption cost. Reference [23] considered the co-optimization of server operation and power procurement, then proposed a new holistic approach to implement an efficient demand response. Reference [24] presented a group of energy management techniques to achieve the energy minimization for geographically distributed data centers. Detailed cooling power, co-location interference, and time-of-use (TOU) electricity pricing of data centers were considered in modelling of this problem.

III. System Model and Problem Formulation

In this section, we consider an energy management problem for multi-energy-source data centers in a single time slot. The energy management model is shown in Fig. 2, which is divided into six parts: a power grid module, a battery module, a fuel cell module, a renewable energy module, data center module, and a task execution module. When a request is submitted to the system, the scheduler on the request side dispatches the incoming requests to different data centers. Meanwhile, the scheduler on the energy supply side is responsible for distributing the different amounts of energy for powering the data center. We have two objectives. One objective is to minimize the total energy consumption with a minimized gap between supply and demand. The other objective is to minimize the total electricity cost. It is difficult to realize these two objectives at the same time. Therefore, we decide to use two algorithms to realize the above two objectives and the two algorithms are performed alternately. These two algorithms are: ① realizing the minimization of the gap between supply and demand in a single time slot; ② realizing the minimization of the total energy cost in a long time period.

Fig. 2 Energy management model of multi-energy-source data center.

A. Problem Formulation in Single Time Slot

1) Model for Data Center

There are many different kinds of workloads in data centers, some of which are delay-sensitive, such as web services. The others are delay-tolerant, such as simulations and MapReduce jobs. These delay-tolerant workloads can be scheduled to run at any time as long as the jobs are completed before a maximum completion time, i.e., there is a maximum completion time which may be 1 hour. Therefore, in this paper, we consider the delay-tolerant workloads. In addition, the effect of workload migration is ignored, because this paper concentrates on the energy cost of mismatched demand and supply. The cost produced by workload migration is relatively small, so it is ignored.

The amount of input workload requested in data center j is denoted by Dj. The proportion of the incoming requests transmitted from data center j to data center i is xij. The number of input requests loaded from data center j to data center i is given as:

jj=0JDjxiji (1)

where j() is a non-decreasing function; and J is the number of data centers. In this paper, we consider a linear function that has been adopted by existing works [

21], [25].

After receiving the input workload, the data center should provide adequate energy supply to conduct data processing. Four kinds of energy sources can be used in this model, which exhibit different energy response curves, as analyzed in Section II. Then, we define Gi(t), Si(t), Ei(t), and ri(t) as the amounts of energy for the power grid, battery, fuel cell, and renewable energy sources, respectively. For each energy source i, we can obtain:

Gi(t)0iSi(t)0iEi(t)0iri(t)0i (2)

The above constraints mean that each data center can be powered by four kinds of energy sources, and each data center can be powered by one or more energy sources in the same time slot.

2) Electricity Pricing

Different from the conventional data center system, there are multiple types of energy sources. Therefore, a single electricity pricing rule cannot be applied in this case. For example, a renewable energy source has the lowest electricity price, but the price is not stable. Because the renewable energy resource price changes over time. We denote pirenew(t) as the electricity price for renewable energy sources in data center i in time slot t. A lot of studies concentrate on the prediction of electricity pricing for renewable energy sources [

26]. Therefore, in this paper, pirenew(t) is known in advance. ri(t) is the energy supply from renewable generation, and the cost of renewable generation Ri can be represented as:

Ri=ri(t)pirenew(t)i (3)

For the power grid and battery system, we adopt the TOU electricity pricing policy [

27]. We denote three levels in this TOU pricing policy: the prices of valley, flat, and peak loads are piTOU,low(t), piTOU,middle(t), and piTOU,high(t), respectively. Assuming that the battery is always charged at a lower electricity price, we regard piTOU,low(t) as the pricing policy for the battery system. The cost of the power grid can be represented as:

Pi=Gi(t)piTOU(t)i (4)

where piTOU(t) is the TOU electricity price of the power grid.

Let Si(t) be the energy level of the battery at data center i in time slot t. Then, Si(t) is bounded by its maximum capacity Simax, i.e.,

0Si(t)Simaxi,t (5)

The dynamics of Si(t) can be expressed by

Si(t+1)=Si(t)+Sicharge(t)-Sidischarge(t)i,t (6)

where Sicharge(t) and Sidischarge(t) are the charging and discharging energy, respectively, which are bounded by (7) and (8).

10 kWhSicharge(t)Sicharge,maxi (7)
0Sidischarge(t)Sidischarge,maxi (8)

where Sicharge,max and Sidischarge,max are the maximum charging and discharging energy, respectively.

The charging energy of the batteries is the energy production. Therefore, the cost of the battery system in each time slot can be expressed by

Bi=Sicharge(t)μpiTOU,low(t)i (9)

where μ is the charging/discharging rate of the battery system.

For fuel cells, we use pifuel(t) as the price of fuel cells. Ei(t) is the energy supply from fuel cells, and the cost of fuel cell generation Fi can be represented as

Fi=Ei(t)pifuel(t)i (10)

As [

27], [28] mentioned, these prices have the following relationship:

piTOU,high(t)piTOU,middle(t)piTOU,low(t)pifuel(t)pirenew(t)i (11)

3) Chance-constrained Problem

Our ideal objective is to match the energy demand and supply in each time slot. In fact, it is hard to accurately match the energy consumption and supply when a job arrives, because the energy demand Dj cannot be known in advance. In addition, renewable energy is uncertain in each time slot, and the price of this kind of energy source also strongly fluctuates. Therefore, the matching between the energy demand and supply in each time slot is just an ideal objective. In order to address this problem, we model both the energy demand Dj and the cost of renewable generation Ri as random variables whose expectation can be acquired through an analysis of historical information. The chance constraint can be written as

PrGi(t)+Si(t)+Ei(t)+ri(t)-j=0JDjxij01-ϵi (12)

where Pr() is the probability function; and ϵ is the arbitrarily small value.

By considering the following constraints, the minimization problem (abbreviated as P1) is described by (2) to (4), (7),and (9) to (16).

mini=0ICosti (13)

s.t.

Costi=Pi+Bi+Fi+Ri  i (14)
0Gi(t)Gimaxi (15)
0Ei(t)Eimaxi (16)

where I is the number of energy sources; and Gimax and Eimax are the maximum amounts of energy from power grid and fuel cells, respectively.

Our objective is then to minimize the total energy cost of the multi-energy-source supply system. As the existing convex optimization solutions cannot solve the above constraint (12) directly, the above non-convex optimization problem is a nondeterministic polynomial (NP)-hard problem.

4) Algorithm Design

In this subsection, we address the challenge of the chance constraint (12) of P1 and consider ri(t) as a random variable. Suppose that the distribution of ri(t) is bounded within [ari,bri] and the distribution of Dj is bounded within [adj,bdj]. By defining αri=(bri-ari)/2 and βri=(bri+ari)/2, ri(t) can be normalized within [-1,1] as follows.

oiri-βriαri[-1,1]i (17)

Similarly, by defining αdj=(bdj-adj)/2 and βdj=(bdj+adj)/2, Dj can be normalized within [-1,1] as follows.

qjDj-βdjαdj[-1,1]i (18)

Additionally, let Xi=Gi(t)+Si(t)+Ei(t)+βri+j=0Jβdjxij and Yij=-αdjxij. The chance constraint can be equivalently written as

PrXi+αrioi+j=0JqjYij01-ϵi (19)

According to the Bernstein approximation, the constraint can be approximated by

infχ>0χlgexpχ-1Xi+αrioi+j=0JqjYij+χlg1ϵ0 (20)

where χ is the weight.

Formula (20) can be briefly written by

infχ>0Xi+χΩ(χ-1(oi,Yij))+χlg1ϵ0i (21)

where Ω(χ-1(oi,Yij)) can be expressed as (22).

Ω(χ-1(oi,Yij))=lgexpχ-1αrioi+j=0JqjYij (22)

According to [

29], Ω(w)max(ν-w,ν+w)+σ2w2/2, where -1ν-ν+1 and σ0 are constants that depend on the given probability distribution, and w is a variable. Accordingly, the constraint (20) can be approximated by

infχ>0Xi+ν+αrioi+j=0JqjYij+σ22χαrioi+j=0JqjYij2+χlog1ϵ0 (23)

The inf() in the above constraint can be removed by substituting χ=σαripi+j=0JqjYij/2lg ϵ-1, so that (23) can be equivalently written as

Xi+ν+αrioi+j=0Jqj(t)Yij+2lg ϵ-1σαrioi+j=0JqjYij0 (24)

Finally, a mixed-integer linear programming formulation for the optimization problem (abbreviated as P2) is described by (2) to (4), (7), (9) to (13), (15), and (24).

B. Algorithm Design for Long Period

In this subsection, we will introduce the algorithm design for a long period. In Section III, the proportion of the incoming requests xij are obtained through Algorithm 1. However, the energy amounts of the four energy resources in each time slot Gi(t), Si(t), Ei, and ri(t) need to be obtained in advance. In addition, Algorithm 1 only realize one of the objectives: the minimization of the total energy cost. In this case, we will also minimize the gap between energy demand and supply in all data centers. Therefore, we design another algorithm to obtain the other variables.

Algorithm 1  : online control algorithm in single time slot

1: Input: set ri(t), Gi(t), Ei(t), Si(t), I, J, piTOU, pirenew, pfuel, ν+, σ, αri, βri

2: Output: xij, Costi

3: begin

4: at the beginning of each time slot t, do

5: observe the system states: ri(t), Gi(t), Ei(t), Si(t)

6: find the optimal solution of P2

7: end

The energy supplies of both fuel cells and batteries are related to the energy supplies in the last time slot based on the limitation of the charging/discharging rate and slow flowing behaviour. The dynamics of Ei(t) can be expressed as

Ei(t)=Ei(t-1)+Δfi(t-1)i,t (25)

where Δfi(t-1) is the energy supply changes of the fuel cells. The lower bound and upper bound of Δfi(t-1) are denoted as -fimin and fimax, respectively, i.e.,

1-fiminΔfi(t-1)fimaxi,t (26)

For xij(t) is obtained by Algorithm 1, we also have

j=1Jxij(t)=1i,t (27)
xij(t)0i,j,t (28)

Let Hi(t) denote the energy demand by the workloads of data center j in time slot t. The relation between Hi(t) and Dj can be expressed as

Hi(t)=Hi j=1Jxij(t)Dji,t (29)

The objective of Algorithm 2 is to minimize the gap between the energy demand and supply in all data centers. There are two situations involved in this gap: one is an oversupply, the other is a short supply. Let mi and ni denote the weights of the two cases, respectively. The problem (abbreviated as P3) is described by (6) to (8), (25) to (28), and (30) to (35).

mint=1Ti=1J(Li(t)+Zi(t)) (30)

s.t.

Wi(t)=Gi(t)+Si(t)+Ei(t)+ri(t)i,t (31)
Li(t)=mimax(Hi(t)-Wi(t),0)i,t (32)
Zi(t)=nimax(Wi(t)-Hi(t),0)i,t (33)
mi+ni=1i (34)
mi>0ni>0 (35)

where Wi(t) is the sum of the energy supplies from the four kinds of energy sources. Constraints (32) and (33) stand for the energy gap when the energy demand exceeds the energy supply, and the energy gap when the energy supply exceeds the energy demand, respectively. mi>ni means that the cost of the case in which demand exceeds supply is more significant than that of another case. Similarly, mi<ni means that supply exceeding demand is more important. As a result, the objective function and constraints are all linear, and the P3 is easy to solve offline. However, the future energy demand cannot be obtained in advance. Therefore, we design an online algorithm to solve this problem, as shown in Algorithm 2.

Algorithm 2  : online algorithm in long period

1: Input: set I, J, Dj, Sicharge(t), Sidischarge(t), fi(t), xij(t)

2: Output: ri(t),Si(t),Gi(t),Ei(t)

3: Wi(t)=0

4: for t=0; tT do

5:  for i=0; iI do

6:  observe the states demands: Dj(t), xij(t)

7:  increase the energy supply following the order of ri(t), Ei(t), Gi(t),       Si(t)

8:  obtain the objective value of P2

9:  end

10: end

In Section III, we have proposed two minimization problems to realize two different objectives: ① the minimization of the total energy consumption with a minimized gap between supply and demand; ② the minimization of the total electricity cost, as shown in P1 and P3. According to the two formulations, the results in P3 are related to the time slots while the results in P1 are unrelated to the time slots. Therefore, the operations in P1 occur in single time slot while the operations in P3 are done from one time slot to next time slot. In addition, we also have found that some input variables of Algorithm 1 are the output values of Algorithm 2. Likewise, some input variables of Algorithm 2 are the output values of Algorithm 1. Hence, the two algorithms are performed alternately. In order to better interpret the executed time logic of these two algorithm, we consider the following simple example, as shown in Fig. 3.

Fig. 3 Relationship between two algorithms.

Suppose that we have observed a group of initial values at time slot T0: ri(T0),Gi(T0),Ei(T0,Si(T0)). Then we perform Algorithm 1 before next time slot T1 by using the initial values. After performing the Algorithm 1, we perform Algorithm 2 at next time slot T1 by using both the initial values at time slot T0 and some output values of Algorithm 1 (xij(T0)). Then, the Algorithm 1 will be performed again by using some values (ri(T1),Gi(T1),Ei(T1,Si(T1))) obtained from Algorithm 2. Therefore, the complete algorithm combining the two algorithms is shown in Algorithm 3.

Algorithm 3  : hybrid online control algorithm

1: Begin

2: in each slot t, do

3: at the beginning of slot t, observe the system states: ri(t), Gi(t), Ei(t),     and Si(t)

4: find the optimal solution of P1

5: observe the system state: xij

6: find the optimal solution of P2

7: end

IV. Evaluation Methodology

A. Measurement Metric and Experimental Settings

Carbon usage effectiveness (CUE) [

30], [31] is defined as a sustainability metric to measure the carbon emission associated with data centers. A comparison of the carbon emission rate from the most commonly-used energy sources is shown in Table II [30]. Then, we use the average carbon emission rate to estimate the green level of each energy composition. The average carbon emission rate (CER) of a composition is found by summing the weighted contributions from each type of energy source, which is defined as:

CER=eiγiei (36)
Table II Carbon Emission Rate of Major Energy Sources
SourceCarbon emission rate (g/kWh)SourceCarbon emission rate (g/kWh)
Nuclear 150.0 Hydro 13.5
Coal 9680.0 Wind 22.5
Gas 4400.0 Solar 53.0
Oil 8900.0

where ei is the electricity generated from energy type i; and γi is the carbon emission rate of energy type i.

In the next section, we carry out several sensitivity analyses for the key parameters of our proposed model to investigate the relationship between different parameters and their effects on the final results. In our proposed model, we have two different optimization objectives, including the energy cost and the gap between demand and supply. In order to better obtain a clear comparison between the different pairs of parameters, we introduce a new optimization objective named Tcost, which is:

Tcost=gap×17 (37)

where gap is a type of energy loss. The factor 17 in (37) means that, as the gap between demand and supply is a type of energy loss, we can use the maximum electricity price as the price of the energy gap.

The other key experimental settings are given in Table III.

Table III Experimental Parameter Settings
ParameterValueParameterValue
T (hour) 744 Smax,Emax (kWh) 50
I,J 5 Gmax (kWh) 500
ΔEmin,ΔSmin (kWh) 5 pfuel (cent/kWh) 5
ΔEmax,ΔSmax(kWh) 10 μ 0.95
S0,E0 (kWh) 5 ν+ 0
G0 (kWh) 200 σ 0.95

B. Methodology

We use five different kinds of workload traces collected from the Wiki data center, which are shown in Figs. 4 (a)-(e) [

32].

Fig. 4 Simulation dataset of workload. (a) Workload A. (b) Workload B. (c) Workload C. (d) Workload D. (e) Workload E.

In the experiments, the length of each time slot is set to be 1 hour. The five traces show different characteristics. We also use solar energy generation and wind energy production as our renewable energy traces, which are shown in Fig. 5 [

33]. The energy prices of renewable energy sources are also shown in Fig. 5(d). We compare our proposed methods with the following schemes.

Fig. 5 Simulation dataset of renewable sources. (a) Electricity price of renewable energy for a month. (b) Solar energy production for a month. (c) Wind energy production for a month. (d) Electricity price for a day.

1) GSEr: including the power grid, battery, fuel cells, and renewable energy.

2) GSE: including the power grid, battery, and fuel cells.

3) GEr: including the power grid, fuel cells, and renewable energy.

4) GSr: including the power grid, battery, and renewable energy.

5) GS: including the power grid and the battery.

6) G: only including the power grid.

V. Experimental Results

A. Sensitivity Analysis

There are various important parameters in the proposed system model. As discussed in Section III, different amounts of power are generated from the four types of energy sources at each site. Different values of these parameters will have a significant effect on the optimization results. For instance, expanding the maximum value of the battery capacity or fuel cell capacity may decrease or increase the energy gap, because of the slow following behaviours of the charging or discharging rate of the battery and the power changing rate of the fuel cells. In addition, different energy compositions will also have a large influence on the final optimization results, because they will affect the proportion of energy sources and then influence the energy cost. In this section, we analyze the impact of Smax and Emax on different energy composition, study how the two parameters impact the energy cost and try to find the optimal composition of parameters.

For these experiments, we consider a configuration with five data centers executing a hybrid workload. The experiments analyze the total cost of the system energy with different parameter compositions for the maximum capacities of the battery and fuel cells. Other implementation-related factors are ignored, because this study only focuses on sensitivity analysis.

In the following subsections, we present sensitivity analysis of our proposed system from three aspects: the relationship between Smax and Emax, the relationship between ΔE(t) and Emax, and the relationship between Smax and ΔS(t).

1) Sensitivity Analysis of Smax and Emax

The maximum capacities of the battery and fuel cells are sampled from 10 kWh to 40 kWh. Furthermore, we consider a simulation of GSRr for each set of values and calculate Tcost as shown in Table IV. In the table, when the Smax is 10 kWh, the value of Tcost increases as Emax increases. Because when the energy demand grows significantly, the energy supply of fuel cells and batteries will increase. However, due to the effect of limited load following, the energy supply will be slow in reducing its output when the energy supply quickly drops in next time slot, which incurs large energy gap. When Smax is larger, a part of extra energy output will be stored in batteries, leading to less waste. When both Smax and Emax are larger than 20 kWh, the value of Tcost is kept on hold. On one hand, when Smax is set to be 10 kWh, the value of Tcost increases as Emax increases, while when Smax is larger than 20 kWh, the value of Tcost decreases as Emax increases. On the other hand, when Smax is lower than 30 kWh, the values of Tcost decreases as Smax increases. When Smax is not lower than 30 kWh, the value of Tcost is kept on hold. From the whole table, we can see that when Smax is higher than 30 kWh and Emax is larger than 20 kWh, Tcost can obtain a minimum value of $11569.63.

Table IV Comparison of Tcost Between Different Compositions of Smax and Emax for Sensitivity Analysis
Smax(kWh)Tcost ($)
Emax=10 kWhEmax=20 kWhEmax=30 kWhEmax=40 kWh
10 120713.1 120775.9 120775.9 120775.9
20 116921.8 116196.8 116196.8 116196.8
30 116190.2 115569.6 115569.6 115569.6
40 116190.2 115569.6 115569.6 115569.6

2) Sensitivity Analysis of ΔE(t) and Emax

As discussed in Sections IV and V, the charging or discharging rate of the battery we are considering in the proposed system model has an immediate impact on the energy cost. The workload at a particular location changes over time by a large margin, and it can even reach hundreds of times. ΔE(t) is set to determine the capacity to adapt to the workload changes of the battery. ΔE(t) at different values has different energy costs as shown in Table V.

Table V Comparison of Tcost Between Different Compositions of ΔE(t) and Emax for Sensitivity Analysis
ΔE(t) (kWh)Tcost ($)
Emax=20 kWhEmax=30 kWhEmax=40 kWhEmax=50 kWh
1 91410.2 91410.2 91410.2 91410.2
2 91895.9 91895.9 91895.9 91895.9
3 11674.0 116755.9 116755.9 116755.9
4 117013.8 117136.4 117136.4 117136.4
5 116190.2 115569.6 115569.6 115569.6
6 116089.1 115290.5 115290.5 115290.5
7 115843.9 114967.9 114967.9 114967.9
8 89180.4 88025.7 88025.7 88025.7
9 89016.4 87896.4 87896.4 87896.4
10 88689.2 63013.7 63013.7 63013.7

Emax and ΔE(t) are sampled from two value ranges, where Emax is set as 20 kWh to 50 kWh and ΔE(t) is set as 1 kW to 10 kW. When ΔE(t) is small (ΔE(t)=1  kWh), the variation of Emax has no effect on the Tcost. When ΔE(t) is 3 kWh and 4 kWh, the limited load following is unable to adapt to significant changes in energy demand. When ΔE(t) is larger than 4 kWh, The changes of the energy supply for fuel cells are enough to cope with the changes of energy demand. If we fix the value of ΔE(t), a minimum value of Tcost can be obtained when Emax is set as 30 kWh to 50 kWh.

3) Sensitivity Analysis of Smax and ΔS(t)

As discussed in Sections IV and V, the charging or discharging rate of the battery has an immediate impact on the energy cost. The workload at a particular site changes over time by a large margin, and it can even reach hundreds of times. ΔS(t) is set to determine the capacity to adapt to workload changes of the battery. ΔS(t) at different values has different energy costs as shown in Table VI. Smax and ΔS(t) are sampled from two value ranges, where Smax is set as 20 kWh to 50 kWh and ΔS(t) is set as 10 kW to 20 kW. When ΔS(t) is fixed, Tcost does not change appreciably when Smax changes. If we fix the value of Smax, Tcost can obtain a minimum value when Smax is set as 20 kWh or 30 kWh. When ΔSmax is 15 kWh, the Tcost increases as Smax increases from 30 kWh to 40 kWh. There is a similar explanation to Table IV as the characteristic of battery is similar to the fuel cells.

Table VI Comparison of Tcost Between Different Compositions of ΔS(t) and Smax for Sensitivity Analysis
ΔS(t) (kWh)Tcost ($)
Smax=20 kWhSmax=30 kWhSmax=40 kWhSmax=50 kWh
10 116196.9 116196.9 115569.6 115569.6
11 113217.1 113217.1 113049.0 113049.0
12 115698.9 115698.9 115397.8 115397.8
13 115612.6 115612.6 115463.7 115463.7
14 115608.0 115608.0 115529.5 115529.6
15 92259.8 92259.8 114045.2 114045.2
16 118423.1 118423.1 114244.3 114244.3
17 118850.6 118850.6 112267.3 112267.4
18 118923.3 118923.3 118923.2 118923.3
19 118923.3 118923.3 118923.2 118923.3
20 118923.3 118923.3 118923.2 118923.3

B. Energy Cost

In this subsection, we evaluate the benefits of green data centers powered by multi-composition energy sources. To be more specific, we compare the energy cost among six kinds of power compositions, which are summarized in Section VI and shown in Fig. 6. For the three kinds of energy compositions including renewable energy, GSEr, GEr, and GSr, the differences between solar energy and wind energy are 0.61%, 1.56%, and 3.03%, respectively. The total energy costs of GEr, GS, and G are much higher than the other three kinds of energy compositions. We can see that the six kinds of energy compositions have different values in their total energy cost. When the renewable energy is solar energy, the relationship among the values of the energy cost is TcostGEr>TcostGS=TcostG>TcostGSr>TcostGSE>TcostGSEr.

Fig. 6 Experimental results of three performance metrics. (a) Comparison of energy cost. (b) Comparison of Tcost. (c) Comparison of carbon emission rate.

When the renewable energy is wind energy, the relationship among the values of the energy cost is TcostGS=TcostG>TcostGEr>TcostGSr>TcostGSE>TcostGSEr. The major difference between the two relationships is the value of GEr; the energy generation of wind energy is much higher than that of solar energy, which is shown in Fig. 6(b) and (c).

When the renewable energy is solar energy, the data centers always need to be powered by the power grid and fuel cells. However, the energy variation is limited by the fuel cells. The main energy supply should be the power grid. As shown in Fig. 7(d) and (i), in both pie charts, the proportion of fuel cells is only around 2%-3%. In Fig. 7(d), the proportion of solar energy is only 0.15%. However, in Fig. 7(i), the proportion of wind energy is 36%. Although the proportion of the power grid with GS and G is 100%, the energy cost of GEr is higher than those of GS and G. Therefore, only the use of a limited energy supply may have a bad effect on the economic efficiency. Comparing the energy costs of GSE and GSr, the energy cost of GSE is lower than that of GSr in both cases of renewable energy sources. Because of the capacities of the battery and fuel cells in GSr, the limited energy cannot amount to full energy supply in each time slot. Therefore, the total energy generation of GSE is much higher than that of GS.

Fig. 7 Energy proportions. (a) Solar-GSEr. (b) Solar-GSE. (c) Solar-GSr. (d) Solar-GEr. (e) Solar-GS. (f)Wind-GSEr. (g) Wind-GSE. (h) Wind-GSr. (i) Wind-GEr. (j) Wind-GS.

In summary, our power management policies can achieve the lowest energy cost. The evaluation results show that our management policy, compared with the GSE policy, GEr policy, GSr policy, and GS policy, can improve the energy cost on average by 4.77%, 118.87%, 21.49%, and 120.87%, respectively.

C. Energy Gap

The energy gap is a necessary evaluation index in several power management policies. In this subsection, we use Tcost to estimate the effect on the energy gap, which is shown in Fig. 6(b). When the renewable energy is solar energy or wind energy, the relationship of the values of Tcost is TcostGEr  TcostG > TcostGSr > TcostGSE >  TcostGSEr. Compared with the results of the energy cost in Fig. 6(a), the main differences are the relationships between the GSE policy and GSr policy and between the GEr policy and GS policy. Tcost in the GSE policy is higher than that in the GSr policy. As mentioned in the previous subsection, GSE cannot achieve a full energy supply in powering the data centers, which lead to a large energy gap in this policy. As there is no energy gap in the GS policy and G policy, the difference in the energy cost between GEr and GS is expanded when we use Tcost as the evaluation index.

In summary, our power management policies can achieve the lowest Tcost. When we do not use renewable energy in GSE, there is little impact on the energy cost. However, renewable energy has a much larger impact on Tcost. On the contrary, when we do not use fuel cells in GSr, there is little impact on Tcost. However, the fuel cells have a much larger impact on the energy cost. The evaluation results show that our management policies, compared with the GSE policy, GEr policy, GSr policy, and GS policy, can improve the energy cost on average by 33.55%, 93.39%, 89.85%, and 4.61%, respectively.

D. Carbon Emission Rate

We further compare the carbon emission rate based on our proposed hybrid power management policies with other baseline power management strategies. Figure 6(c) shows a comparison of different power management policies in terms of carbon emission rate. Obviously, GSEr leads to the lowest carbon emission rate because its first priority is to use clean energy, and the proportion of the power grid is the lowest. When we use solar energy, the carbon emission rate of GSEr is close to that of GSE. As shown in Fig. 7(a) and (b), the proportions of the power grid of GSEr and GSE are 70% and 72%, respectively, and the proportions of fuel cells are 20% and 28%, respectively. The proportions of the main energy supply for the two policies are similar, which is the reason why GSEr and GSE have similar results for the carbon emission rate. When we use renewable energy in the three policies of GESr, GEr, and GSr, wind energy always achieve better performance than solar energy. Because the total energy generation of wind energy is always higher than that of solar energy, the proportion of wind energy is always twice that of wind energy, which is shown in Fig. 7(a), (c), (d), (f), (h), (i).

In summary, our power management policies can achieve the lowest carbon emission rate. The experimental results show that our power management policies, compared to the GSE, GEr, GSr, and GS policies, can improve the carbon emission rate on average by 32.80%, 39.45%, 18.59%, and 83.02%, respectively.

VI. Conclusion

In this paper, we first establish a system model of minimizing the energy cost of data centers powered by heterogenous energy resources, such as power grid, fuel cells, energy storage devices, and renewable energy sources. Then, we formulate a problem in a single time slot to minimize the total energy cost of data centers powered by four types of energy sources. Moreover, we also formulate another problem in a long period to mitigate the energy gap between the workload and energy supply. To solve a chance-constraint problem in the former formulated problem, we design an online control algorithm by using the Bernstein approximation. We also design a greedy online control algorithm to solve the latter formulated problem. Finally, by using two realistic traces, we conduct several sensitivity analyses of the impacts on various parameters of the system model and compare three key characteristics of different energy supplies powered by different energy sources compositions. It is observed that the proposed heterogeneous energy supply model can achieve similar results to other compositions.

References

1

A. Qureshi, “Power-demand routing in massive geo-distributed systems,” Ph.D. dissertation, Massachusetts Institute of Technology, Cambridge, 2010. [百度学术

2

C. Ren, D. Wang, B. Urgaonkar et al., “Carbon- aware energy capacity planning for datacenters,” in Proceedings of IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, Washington DC, USA, Aug. 2012, pp. 391-400. [百度学术

3

W. Deng, F. Liu, H. Jin et al., “Harnessing renewable energy in cloud datacenters: opportunities and challenges,” IEEE Network, vol. 28, no. 1, pp. 48-55, Jan. 2014. [百度学术

4

L. Zhao, J. Brouwer, S. James et al., “Servers powered by a 10 kW in-rack proton exchange membrane fuel cell system,” in Proceedings of International Conference on Asme International Conference on Fuel Cell Science, Boston, USA, Jun.-Jul. 2014, pp. 1-8. [百度学术

5

R. Tripathi, S. Vignesh, and V. Tamarapalli, “Optimizing green energy, cost, and availability in distributed data centers,” IEEE Communications Letters, vol. 21, pp. 500-503, Mar. 2017. [百度学术

6

X. Hu, P. Li, K. Wang et al., “Energy management of data centers powered by fuel cells and heterogeneous energy storage,” in Proceedings of 2018 IEEE International Conference on Communications (ICC), Kansas City, USA, May 2018, pp. 1-6. [百度学术

7

K. Wang, J. Yu, Y. Yu et al., “A survey on energy internet: architecture, approach and emerging technologies,” IEEE Systems Journal, vol. 12, no. 3, pp. 2403-2416, Sept. 2018. [百度学术

8

L. Liu, H. Sun, C. Li et al., “Exploring customizable heterogeneous power distribution and management for datacenter,” IEEE Transactions on Parallel and Distributed Systems, vol. 29, pp. 2798-2813, Dec. 2018. [百度学术

9

F. Koushanfar and A. Mirhoseini, “Hybrid heterogeneous energy supply networks,” in Proceedings of IEEE International Symposium on Circuits and Systems, Rio de Janeiro, Brazil, May 2011, pp. 2489-2492. [百度学术

10

C. Xu, K. Wang, P. Li et al., “Renewable energy-aware big data analytics in geo-distributed data centers with reinforcement learning,”IEEE Transactions on Network Science and Engineering, vol. 7, no. 1, pp. 205-215, Jan. 2020. [百度学术

11

A. Energiewende. (2018, Dec.). Power prices, generation and consumption. [Online]. Available: https://www.agora-energiewende.de/en/service/recent-electricity-data/chart/power generation price/04.08.2018/03.09.2018/ [百度学术

12

M. A. Islam, X. Ren, S. Ren et al., “A spot capacity market to increase power infrastructure utilization in multi-tenant data centers,” in Proceedings of the 2017 ACM SIGMETRICS/International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS ’17, New York, USA, Jun. 2017, pp. 19-20. [百度学术

13

Q. Wu, Q. Deng, L. Ganesh et al., “Dynamo: Facebook’s data center-wide power management system,” in Proceedings of 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), Seoul, South Korea, Jun. 2016, pp. 469-480. [百度学术

14

B. Aksanli, “Data center peak power management with energy storage devices,” IEEE Internet Computing, vol. 21, no. 4, pp. 26-33, Jul. 2017. [百度学术

15

N. Kalhoff, “Integration of fuel cell applications into the power supply for information and telecommunications technology,” in Proceedings of 29th International Telecommunications Energy Conference, Rome, Italy, Sept.-Oct. 2007, pp. 444-448. [百度学术

16

H. Hsu, Q. Deng, J. Mars et al., “Smooth operator: reducing power fragmentation and improving power utilization in large-scale data centers,” in Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems, New York, USA, Mar. 2018, pp. 535-548. [百度学术

17

H. Chen, P. Xuan, Y. Wang et al., “Key technologies for integration of multitype renewable energy sources research on multi-time frame robust scheduling/dispatch,” IEEE Transactions on Smart Grid, vol. 7, no. 2, pp. 471-480, Jan. 2016. [百度学术

18

S. Clegg and P. Mancarella, “Integrated electrical and gas network flexibility assessment in low-carbon multi-energy systems,” IEEE Transactions on Sustainable Energy, vol. 7, no.2, pp. 718-731, Apr. 2016. [百度学术

19

A. Eladl, M. El-Afifi, and M. El-Saadawi, “Optimal power dispatch of multiple energy sources in energy hubs,” in Proceedings of 2017 Nineteenth International Middle East Power Systems Conference (MEPCON), Cairo, Egypt, Dec. 2017, pp. 1053-1058. [百度学术

20

H. Zhang, Y. Li, D. W. Gao et al., “Distributed optimal energy management for Energy Internet,” IEEE Transactions on Industrial Informatics, vol. 13, pp. 3081-3097, Dec. 2017. [百度学术

21

Y. Guo, Y. Gong, Y. Fang et al., “Energy and network aware workload management for sustainable data centers with thermal storage,”IEEE Transactions on Parallel and Distributed Systems, vol. 25, no. 8, pp. 2030-2042, Aug. 2014. [百度学术

22

M. Jawad, M. Qureshi, U. Khan et al., “A robust optimization technique for energy cost minimization of cloud data centers,” IEEE Transactions on Cloud Computing. DOI: 10.1109/TCC.2018.2879948 [百度学术

23

S. Kwon, L. Ntaimo, and N. Gautam, “Demand response in data centers: integration of server provisioning and power procurement,” IEEE Transactions on Smart Grid, vol. 10, no. 5, pp. 4928-4938, Sept. 2019. [百度学术

24

N. Hogade, S. Pasricha, H. Siegel et al., “Minimizing energy costs for geographically distributed heterogeneous data centers,” IEEE Transactions on Sustainable Computing, vol. 3, no. 1, pp. 318-331, Oct. 2018. [百度学术

25

L. Yu, T. Jiang, Y. Cao et al., “Carbon-aware energy cost minimization for distributed internet data centers in smart microgrids,” IEEE Internet of Things Journal, vol. 1, no. 3, pp. 255-264, Jun. 2014. [百度学术

26

H. Wu, M. Shahidehpour, A. Alabdulwahab et al., “Demand response exchange in the stochastic day-ahead scheduling with variable renewable generation,” IEEE Transactions on Sustainable Energy, vol. 6, no. 1, pp. 516-525, Apr. 2015. [百度学术

27

K. Wang, L. Gu, S. Guo et al., “Distributed energy management for vehicle-to-grid networks,” IEEE Network, vol. 31, no. 2, pp. 22-28, Mar. 2017. [百度学术

28

K. Wang, H. Li, and Y. Feng, “Big data analytics for system stability evaluation strategy in the Energy Internet,” IEEE Transactions on Industrial Informatics, vol. 13, no. 4, pp. 1969-1978, Aug. 2017. [百度学术

29

A. Nemirovski and A. Shapiro, “Convex approximations of chance constrained programs,” SIAM Journal on Optimization, vol. 17, no. 4, pp. 969-996, Dec. 2006. [百度学术

30

X. Gao, A. Curtis, B. Wong et al., “It’s not easy being green,” ACM SIGCOMM Computer Communication Review, vol. 42, no. 1, pp. 211-222, Aug. 2012. [百度学术

31

C. Ren, D. Wang, B. Urgaonkar et al., “Carbon-aware energy capacity planning for datacenters,” in Proceedings of 2012 IEEE 20th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, Washington DC, USA, Aug. 2012, pp. 391-400. [百度学术

32

Wiki. (2018,Dec.). Wiki dump data. [Online]. Available: http://dumps.wikimedia.org/other/pagecounts-raw [百度学术

33

Agora Enegiewende. (2018, Dec.). Source code: piceang-for-hres (MATLAB) optimal design of hybrid renewable energy systems using multi-objective evolutionary algorithm. [Online]. Available: http://ruiwangnudt.gotoip3.com/optimization.htmltdsourcetag=spcqqaiomsg [百度学术