1 Introduction

Increasing the penetration of variable renewables, such as wind and solar energy, is regarded as a critical pathway for the transition to a deep de-carbonized power system. Over the last decade, the penetration of renewables in the power sector has increased from 27% to 57% in Denmark, from 10% to 26 % in Germany, from 15% to 40% in Spain, and from 16% to 44% in Italy [1]. The United States has increased its renewable energy relatively steadily from 9% in 2004 to 13% in 2014 [1], with several important states leading the change. The penetration of renewables in California, for example, increased to 33% in 2017 [2]. The global investment in renewable energy for the year of 2016 surpassed, for the first time, that for the fossil fuel sector.

Aggressive plans have been made across the world to promote further use of renewables to mitigate the impact of changing climate. In the United States, New York State has targeted to have 50% of its power supplied by renewables by 2030; Hawaii has committed to have 100% of renewable supply by 2045; California has committed to a 33% renewable portfolio standard (RPS) by 2020, 50% by 2030 [3, 4]. China committed in the Paris COP to supply 20% of its primary energy consumption by renewables by 2030, equivalent to approximately 40% of renewables in the power sector.

It has been shown that renewables can provide more than enough energy compared to the current needs of human society. Reference [5] argued that the economically available potential of wind power in China is 18 times larger than its current total energy consumption. Reference [6] showed that the wind power on a global scale could provide 35 times total current electricity consumption. Globally exploitable hydropower potential was comparable to total electricity demand in 2005 [7]. The potential for the combination of hydro and solar power is about 4-10 times current global energy consumption [8].

However, the transition towards higher penetration of renewables is confronting significant challenges in different counties. In China, grid integration of renewables is one of the greatest challenges. Financial loss due to wind curtailment reached 20 billion RMB in 2016. Europe provided subsidies too early when the cost of renewable was relatively high. In Germany, about 36% of solar power was installed before 2010, when the capital cost for solar power was greater than 6000 $/kW. As a result, the price for retail electricity increased by 10% in Germany in 2009. The United States faces a number of pricing distortions in current power markets at both wholesale and retail levels. As a result, profits for fossil generators (especially coal-fired units) and utilities are shrinking.

In the long-term, capital-intensive transition, the challenges are not only technological, but involve increasingly economical, political and environmental dimensions. Attention to interactions between energy systems, the atmospheric environment and social economics is lacking. References [9] and [10] have demonstrated the importance of considering cyber-physical interaction and restrictions on primary energy supply. However, a broader scope and larger vision is needed to deal with these interlinked challenges. From the technological perspective, the coordination between power sector and other energy sectors needs special attention. Also, coordination is needed between short term operations and long-term planning.

The environment/climate system and the power system interact strongly under elevated renewable penetration. Production of wind power is affected by the state of the climate system and wind conditions. The available solar power depends not only on the technical characteristics of solar panels, but also critically on strength of incoming solar radiation, cloud cover and the optical characteristics of air pollutants. Hydro power production depends heavily on regional precipitation. Almost all of the major hydro resources in China rely on precipitation on the Tibetan Plateau, vulnerable particularly to properties of a changing climate.

The interactions between the energy systems and socio-economic systems are also of importance. Energy policy will influence the economics for different energy technologies, and will determine thus the generation mix for the power sector and even the overall energy structure. The current structure of the power market is designed to adapt to a generation mix with high operational costs, and the increased investment in renewables has resulted in a significant pricing distortion in wholesale markets [11].

This paper provides an overview of the interlinked challenges related to power systems, environmental systems and socio-economic system when transitioning towards a higher penetration of renewables. The modeling framework to simulate the coupled energy-environment-economic systems will be discussed in the final section.

2 Technical challenges

To better deal with the variability and intermittency of renewables, coordination over different time horizons and different energy sectors is required.

2.1 Demand of flexibility to integrate variable renewables

Sufficient flexibility in the power system is essential for an effective integration of variable inputs from renewables. The existing levels of flexibility for generation fleets differ from region to region. Figure 1 summarizes the generation mixes for several regions in China as compared with the generation mix for countries, such as Germany, Spain and the United States. The flexible resources, mainly gas and hydro units, account for half of Spain’s generation mix. In contrast, coal-fired units are responsible for 80% of the generation mix in North China, imposing significant challenges for the integration of renewables.

Fig. 1
figure 1

Generation mix for different countries and regions as in 2014

The role of flexibility is under-appreciated in traditional power system planning. With a higher contribution from renewables, flexibility is critically important in determining operational costs, curtailment of wind power and the rate of utilization of thermal units. Ignoring the flexibility will lead to a 24% difference in generation mix [12]. Reference [13] indicates that when the renewable penetration is higher than 50%, large scale (≥ 600 MW) units will fall out of the optimal generation mix in China, due to expensive startup costs. Omitting flexibility is no longer acceptable for power system planning at higher penetration levels of variable renewables.

However, the evaluation of flexibility in the capacity expansion framework is computationally intensive. Incorporating the mixed integer linear programming (MILP) formulation of the unit commitment problem for the whole year is computationally inefficient. A common approach is to choose a set of selective days to represent the whole year, but such selection is subjective and arbitrary. Weekly and seasonal variations in renewable power supply and power demand are often ignored in such treatments. On a macro level, an alternative approach has been proposed to employ the unit clustering and grouping technique, and the computational speed can improve by an approximate factor of 2000. With further linearization [13], the computational speed can improve by as much as 20000 times compared with the conventional UC formulation, making feasible a simulation for the entire year.

2.2 Coordination with different energy sectors

For many regional energy systems with significant wind penetration, combined heat and power (CHP) units account for the largest fraction of thermal units. In Denmark, more than 80% of the district heating is cogenerated with electricity. About 50%–70% of thermal generation in north and northeastern China is by CHP units [14]. During the heating season, the operational flexibility of CHP units is reduced significantly. Driven by the strict demand for heat, the CHP units are often must-run units during the heating season, and their range of power outputs narrows down accordingly.

Several technologies have proven to be effective in bridging the otherwise curtailed wind power to satisfy the demand for space heating. The most cost-effective solution is to install electrical boiler in parallel with CHP units [13]. When wind power is facing potential curtailment, electric boilers can not only generate heat through otherwise curtailed wind power, but could also reduce reliance on CHP units for heat supply, leading to higher operational flexibility of CHP units [1516]. The use of heat storage tanks is another potential option to relieve the link between power and heat production for CHP units. The investment cost for large scale heat storage tank is about 0.0037 million $/MWh, orders of magnitude lower than the cost for electrical storage systems [16].

On the consumption side, electrical thermal storage uses resistance heating combined with ceramic bricks, storing heat at a temperature of thousands of degrees. Heat pumps provide the most efficient means for heating. The coefficient of performance (COP) for a heat pump defines the amount of heat transferred from ambient environment to the indoor area per unit of electricity. The COP of air-source heat pump ranges from 3 to 5 depending on the ambient temperature [17]. The use of a heat pump provides thus an efficient way to harness the benefit of wind curtailment. The passive storage embedded in the building thermal inertia is also important for mitigating the variability of renewables [17].

Charging of electric vehicles could significantly influence the integration of renewables. The charging behavior depends on the type of vehicles, the driving pattern of the vehicle owners and the incentives provided by tariff for use of electrical chargers. Reference [18] has shown that fast charging is counterproductive for wind integration and leads to an increase in CO2 emissions. Slow-charging vehicles, in contrast, are beneficial for the renewable integration. A coherent tariff design is important to incentivize the use of slow charging rather than relying on the fast charging option. Private light duty vehicles, taxi and public buses have different emission intensities, driving behavior and annual miles traveled; different types of electric vehicles can have different impacts on air quality [18].

3 Influence of environment and climate on power system planning and operation

3.1 Variability of renewable sources and its connection with climate systems

With an increasing contribution of electricity from wind and solar, system operations, market pricing and long-term planning of the power sector are strongly influenced by variability in the climate system.

The potential for wind power in China has declined significantly over the past 37 years, responding to the changing climate [19]. On top of that, the long-term decline is punctuated by significant inter-annual variability. For some locations in Inner Mongolia, the variability in annual production from wind power was larger than 40% [19]. Figure 2 illustrates the inter-annual variability of wind power potential for Inner Mongolia as a whole. The decline of wind power potential is more than 15% over the less than 4 decades.

Fig. 2
figure 2

Inter-annual variations of wind capacity factor for Western Inner Mongolia

The significance of the inter-annual variability is currently under appreciated because: ① the wind industry is relatively young and the time duration for most of the existing wind projects is less than 10 years; ② the penetration of wind power until very recently has been relatively low for most areas.

The inter-annual variation of wind power will influence not only the revenue stream for the wind producer, but will have an impact also on fuel consumption, and on revenue and utilization rates for thermal units. The capacity credit for wind power is assessed currently based on records of wind power production from experience of generally less than three years. The analysis depends thus largely on the choice of years for which these data are applied, leading to potential bias in investments for projected future thermal capacity. With better understanding of inter-annual variability, the uncertainty associated with the climate impact can be reduced.

3.2 Influence of haze pollution on solar power

In contrast to the influence of climate on wind power, solar power is affected strongly by the levels of pollution in the atmospheric environment. During haze events, penetration of sunlight is affected by properties of the fine particles in the atmosphere, with two implications: ① reduction of direct radiation; ② increase of indirect radiation. The overall effects on the solar source of haze pollution depends on both the severity of pollution and the optical property of involved pollutants. The pollutants can be reflective (white) or absorbing (black). A preliminary study has shown that the haze pollution in several regions in China could reduce solar power by 20% on an annual basis [20]. Dust accumulation on the surface of solar photovoltaic (PV) systems could lead to an additional reduction in power output. Further studies are needed to better forecast the potential and economic costs for solar power under conditions of severe haze pollution.

It is noteworthy also that power system dispatch can in turn influence air quality, and such effect was demonstrated firstly in [21]. References [21] and [22] presented a novel dispatch framework to minimize the air pollution level by scheduling power units optimally according to geographical locations and meteorological conditions.

3.3 Influence of changing climate on power consumption

With increased penetration of wind and solar power, the role of fossil fuel units switches from providing energy to the provision of reliability and flexibility. As the capacity credit for wind and solar power is relatively low, the total capacity for fossil units is linked almost directly with annual peak power demand. The power demand peaks in summer for most locations, correlated with the hottest days of the year due to use of air-conditioners. With changing climate, the extreme high temperatures may be expected to rise more rapidly than global average surface temperature. The anticipated increase in greenhouse gas concentration may lead to a sharp increase in extreme hot days. Temperatures for 65 days would surpass the 4 hottest days currently in ERCOT if the CO2 concentration increased to 450 ppm [23]. Similar impacts could be observed in Europe [24].

Such changes will lead to a higher level of requirements for installed fossil generation capacity, resulting in higher investment costs and lower utilization rates for thermal units. Scarcity pricing will be triggered more often during extreme hot days. This situation could be exacerbated by the decrease of thermal capacity due to restrictions on available cooling water.

4 Socio-economical challenges

Transitioning towards a low-carbon energy future requires policy interventions, such as a carbon tax, a carbon cap-and trade regime, renewable portfolio standards, feed-in tariffs or production tax credits. Policy interventions in turn influence wholesale and retail markets. Understanding the influence of renewable investments and renewable policy interventions on the wholesale and retail market price is critical in ensuring revenue sufficiency for both renewable and fossil fuel generators, arrangements that are critical for the low-carbon transition.

4.1 Power market with high penetration of renewables

A liberalized power market is designed on the basis of an auction-bidding process. Each generator submits offers for available generation capacity and associated prices (based ideally on operational costs). All submitted offers determine the supply curve (merit order curve) of how much power could be generated at what price. The market clearing price is determined by the intersection of power demand and merit order/supply curves. The market clearing price will apply then to all generators bidding below this level. As the marginal cost for production of wind power is zero, an increase in wind power will shift the merit order curve to the right. All else equal, the market clearing price will fall with increasing penetration of renewables. In other words, higher penetration of renewables will lead to a decrease in revenue stream for both renewables and fossil units [25].

Figure 3 illustrates the merit order curves for mid-continent independent system operator (MISO) region at 0 GW and 20 GW contribution of wind capacity respectively. The merit order curves are based on capacities and bids for more than 2000 power generation units within the MISO region in 2013 [25]. Assuming the power demand represented by the dashed curve, the market clearing prices would be about 60 $/MWh at 0 GW wind penetration and about 30 $/MWh at 20 GW wind: an increase of 20 GW wind power could cut the market clearing price in half.

Fig. 3
figure 3

Illustrative merit order curves based on individual power plants for MISO region in 2013

Along with the decrease of market clearing price, revenue from the wholesale market decreases for all generators. The deficit of revenue for the fossil units is compensated either by a revenue stream from a capacity market or by scarcity pricing in the absence of a capacity market. The presence of a capacity market and scarcity pricing in major ISOs of the United States is summarized in Table 1 [11].

Table 1 Capacity payment for different regional systems

An alternative revenue stream for the fossil units could be supplied through the ancillary service market, including participation in frequency regulation and provision of spinning reserve. The total revenue from frequency regulation is limited as the market size is relatively small compared with energy balancing; possible income by contributing to spinning reserve is larger but depends in turn on the design of capacity payments. Usually, if the power market is implemented with strong incentives in the capacity market, the price of the spinning reserve will be depressed, as the system reserve margin is sufficient. As indicated above, the different markets are strongly coupled. Changes for wholesale market signals will influence also other market prices.

On the other hand, depressing the market clearing price will influence also the revenue sufficiency for renewables. However, variable renewables will not benefit much from a capacity market, and their revenue stream can be offset only with external policy support.

4.2 Low carbon policies

  1. 1)

    Feed-in tariff

A feed-in tariff for renewables is a long-term contract for renewables offering a fixed purchasing price for every kilowatt hour of electricity generated. Such a policy is designed to guarantee the long-term return and minimize the risk of investment. It is also easy to implement. The major concern for a feed-in tariff is that it fails to differentiate the temporal and geographical characteristics associated with the electricity price, and the investment thus is guided only by natural resources. Also, once the investment is settled, the price will be locked in for 20 years. European countries suffered from the fact that too much solar power was installed prior to 2012, when capital costs for solar PV were about 20 €-cts/kWh. As a result, the subsidy imposes a significant fiscal burden, which will be carried for decades to come.

  1. 2)

    Production/investment tax credit

Production/investment tax credits have been adopted in the United States since 1992. They provide tax rebates for renewable investors based on the electricity produced (kWh basis) or capacity installed (kW basis), respectively. The level of subsidy decreased gradually in the United States as the investment cost for renewables continues to drop. One of the differences between production tax credit (PTC) and investment tax credit (ITC) is that PTC incentivizes the increase of power production from renewables; whereas PTC could lead to negative pricing as wind farm owners are willing to bid below zero in the wholesale market in order to claim for the PTC. With a higher PTC subsidy, the market clearing price could go negative when facing possible curtailment [25].

  1. 3)

    Renewable portfolio standards [4]

A renewable portfolio standard (RPS) refers to legislation that creates a market for tradable renewable or green electricity certificates. Electricity generators or wholesale purchasers of electricity are required to source a specified percentage of their electricity (portfolio) from renewables. 38 of the 50 states in the United States have implemented the RPS standards. California is considered the most successful example of using the RPS option to promote renewables.

  1. 4)

    Carbon tax

Taxing carbon is regarded as an economically efficient way to internalize the negative impacts of CO2 emissions in a market environment. The emission factors for coal, oil and natural gas units are 97, 71.3, 53 kg/MMBTU respectively. Elevated carbon tax will differentially increase the operational costs of coal and natural gas units. Further, the carbon taxes will result in an increase in the wholesale market clearing price, making investment on renewables more profitable.

  1. 5)

    Net metering

Under net metering programs, distributed renewable sources, such as solar, sell electricity back to the grid at retail prices, a payment sometimes twice the price in the wholesale market. Such a policy has incentivized the rapid installation of solar power in the US in recent years, stimulated by decreasing costs for solar PV. However, transmission and distribution costs for residential consumers are charged based on electricity consumed (per kWh) in the United States. Residential consumers equipped with a significant capacity of roof-top solar PV can avoid the transmission and distribution payment, leading to a shrinking revenue stream for utility companies. Proper correction for the retail tariff structure (mainly allocation of transmission and distribution costs) is urgent for the power industry at elevated solar penetration.

5 Modeling framework for coupled energy-environment-economic systems

To address the coupled technological, environmental and socio-economic challenges, an integrated modeling for coupled energy-environmental-economic systems is required. The simulation framework is illustrated as in Fig. 4.

Fig. 4
figure 4

Modeling framework for coupled energy-environment-economic systems

For the environmental simulation, global general circulation models can be used to capture the inter-annual variations in the climate system on a global scale. Meteorological fields observational and reanalysis data are critical to understand the potential of renewable energy resources. A higher resolution analysis for solar power would require an atmospheric chemistry model to determine the level and properties of air-pollution.

For the energy system simulation modeling, the coupling of optimal power system investment and operational simulation is needed to design the optimal pathway for deep decarbonized power systems. Interactions between heating systems and electrified transportation are also important for modeling simulations. Assessments of resources and costs for renewables are interlinked through environmental simulation models.

For economic assessments, power market simulations should emphasize the interaction between wholesale, capacity and ancillary markets. The impact of diverse low-carbon policies should not be ignored in conducting market simulations.

6 Conclusion

Transition towards higher penetration of variable renewables encounters interlinked challenges from environmental, technical and social-economical aspects. An integrated energy-environmental-economic system framework is proposed in this paper, but further efforts are required to advance understanding and improve solutions.