Carbon-Efficient Virtual Machine Placement Based on Dynamic Voltage Frequency Scaling in Geo-Distributed Cloud Data Centers

Abstract

The tremendous growth of big data analysis and IoT (Internet of Things) has made cloud computing an integral part of society. The prominent problem associated with data centers is the growing energy consumption, which results in environmental pollution. Data centers can reduce their carbon emissions through efficient management of server power consumption for a given workload. Dynamic voltage frequency scaling (DVFS) can be applied to control the operating frequencies of the servers based on the workloads assigned to them, as this approach has a cubic increment relationship with power consumption. This research work proposes two DVFS-enabled host selection algorithms for virtual machine (VM) placement with a cluster selection strategy, namely the carbon and power-efficient optimal frequency (C-PEF) algorithm and the carbon-aware first-fit optimal frequency (C-FFF) algorithm.The main aims of the proposed algorithms are to balance the load among the servers and dynamically tune the cooling load based on the current workload. The cluster selection strategy is based on static and dynamic power usage effectiveness (PUE) values and the carbon footprint rate (CFR). The cluster selection is also extended to non-DVFS host selection policies, namely the carbon- and power-efficient (C-PE) algorithm, carbon-aware first-fit (C-FF) algorithm, and carbon-aware first-fit least-empty (C-FFLE) algorithm. The results show that C-FFF achieves 2% more power reduction than C-PEF and C-PE, and demonstrates itself as a power-efficient algorithm for CO₂ reduction, retaining the same quality of service (QoS) as its counterparts with lower computational overheads.

1. Introduction

Datacenters are critical infrastructures that amalgamate vast computing and storage resources, offering online computing as and when needed. Virtualization techniques embedded in grid computing platforms aid data centers in providing computing resources as a service to customers [1]. The growing energy consumption is a significant problem in data centers. The consumption of energy is increasing by about 10%–12% per year [2]. Synchronized power and resource management are essential to assist data centers in conserving energy while providing the required quality of service (QoS) for hosted applications [3]. It is very much advantageous to maximize server utilization to lower energy consumption [4]. Virtual machine (VM) consolidation is performed to accomplish auto scaling, resulting in reduced energy consumption [5,6].It is essential to maintain the maximum number of servers possible in a running state to satisfy service level agreements (SLAs), which account for more than 80% of information technology (IT) budgets. The power consumption of an idle server is two-thirds of its energy consumption with 100% utilization at full load [7,8,9]. It is noteworthy that idle power and dynamic power consumption utilization levels vary based on different power models of physical servers. The energy reduction achieved by shrinking the number of existing resources through VM consolidation may result in lower resource availability, jeopardizing the credibility of the provider. Utilization of servers at high voltage results in high temperatures and shorter lifetimes. Resource utilization should be optimized based on the computing capacities of the servers in order to reduce idle and active server power consumption [10]. Considering the above, minimum power consumption is achieved through optimum central processing unit (CPU) utilization of the servers with our proposed algorithms.

2. Related Works

Complications in workload allotment in servers with reduced power consumption mean that optimal power management is required, which is dependent on the arrival rate of the tasks and the processor’s power-to-frequency relationship. Thus, it is vital to perform a quantitative analysis of the association between dynamic voltage frequency scaling (DVFS) and power consumption to optimize the use of servers [11]. The worst-fit decreasing (WFD) strategy has been proposed as a load balancing approach for task allocation and energy consumption reduction [12], where by DVFS-based fixed discrete CPU utilization levels were considered and 34% power reduction was achieved [13]. A polynomial complexity algorithm was presented, with the assumption that the energy consumption of servers with lower workloads is comparably less than higher workloads [14]. When optimizing power consumption, most of the research work has focused on optimizing the CPU and cooling devices, as they are the components that consume the most power. The CPU consumes 46% and the cooling device consumes 15% of the total power in a data center [15]. The DVFS-based approach can be used when there is a lower workload and no need to run the servers at their maximum performance level [16,17,18]. The job scheduling approach was used for workload management to achieve maximum utilization of servers with reduced energy consumption [19,20,21]. Many heuristic methods for VM placement have been used with constrained combinatorial optimization problems for different objectives, such as to identify energy-efficient hosts for VMs [22,23,24], to reduce the number of migrations [25], and to increase the number of idle hosts [26,27]. In heterogeneous environments, heuristic techniques generally cannot guarantee optimal long-term solutions [28]. Reducing power consumption through the VM migration approach involves limiting the number of powered servers operating at highest utilization level.This approach is not energy free, rather it is dependent on VM size and bandwidth [29]. The energy requirement for the live migration of idle VMs is estimated using their proposed power model [30]. The consecutive sequential migration of several VMs also has an energy impact [31]. DVFS is mainly applied to non-critical workloads to improve energy-efficient scheduling of idle servers or light-loaded servers [32]. The genetic algorithm-based model was proposed for VM placement to minimize energy consumption [33]. A multiobjective model of the VM placement problem was proposed to maximize resource utilization and minimize energy consumption and network traffic, with VM placement formulated as a bin-packing problem and the network traffic reduction formulated as a quadratic assignment problem, with resources constraints [34]. An optimization model with multiobjective formulation was considered to maximize server utilization and minimize the number of active servers with memory and CPU resource constraints [35]. The algorithm was designed in order to form an optimal initial population and to reduce the search space, and was evaluated on a small-scale data center. The author of [36] disagreed with the work performed in [35], insisting on the need for an exhaustive approach in order to arrive at an optimal solution for difficult NPproblems. Approaches for bin-packing-based modified best fit decreasing (BFD) placement and dynamic placement of VMs were considered to reduce operation costs and environmental impacts [37]. In the modified BFD approach, the VMs with the best utilization were placed in the physical machine (PM) with the least energy consumption. A multi dimension space partition model was used to balance resource utilization and energy consumption. Energy efficient virtual machine placement algorithm with balanced resource utilization was proposed to reduce power consumption by reducing the number of PMs [38]. A multi objective problem was formulated to reduce energy consumption by forecasting CPU and memory utilization in the forthcoming slot based on the previous history [39]. The result outcomes were compared with a grey forecasting model. A constrained optimization problem was formulated with virtual machine (VM) and physical machine (PM) profiles. New heuristic information was embedded in an ant colony algorithm to achieve energy efficiency [40].

A mixed-integer non-linear programming model was proposed for systematical allocation of workloads, considering the electricity price diversity of geo-distributed data centers and DVFS of servers without violating QoS requirements [41,42]. A tradeoff between energy consumption and cost was achieved by exploiting the electricity price, data center location, and energy source in the context of internet services sensitive to response time [43]. Reinforcement-learning-based resource management was optimized for information storage with QoS and power consumption as reward function [44]. The revenue from different tasks was considered as a QoS parameter and VM migration and network communication were considered for power consumption. Geo-distributed resource allocation was performed based on two heuristic force-directed load distribution (FDLD) methods, namely task-aware over provisioning and simple over provisioning with co-location interference, in order to estimate the co-location effects of different task execution rates [45]. Genetic-algorithm-based co-location-aware load distribution was also performed and compared with FDLD, with the result showing energy cost reductions with over provisioning elimination, making this an optimal choice. Regarding data center power efficiency measurement, PUE and carbon usage effectiveness metrics play vital roles. Most of the energy consumption in data centers is caused by the cooling load [46].

This work considers all three triangular dependent parameters, namely the power dissipation of the processor regarding the operating frequency, cooling device power consumption, and dynamic PUE. The carbon-aware power-efficient optimal frequency (C-PEF) and carbon conscious first-fit optimal frequency (C-FFF) algorithms proposed in this work distribute the load and maintain the lowest possible utilization level in all servers for the current workload. The placement algorithm considers the following factors:

• Selection of data centers and clusters is performed based on the PUE and CO₂ emission rate, aiming to reduce the overall carbon footprints of the data centers;
• Load balancing is done by identifying a feasible server with a minimal operating frequency for the current workload with the required quality of service, aiming to reduce hot spots in CPU heat dissipation, which have a direct impact on hardware lifetime and performance;
• The impacts of static and dynamic power usage effectiveness (PUE) on placement decisions are analyzed, along with cooling load power impacts.

The rest of this paper is organized as follows. In Section 1, the general facts about power consumption in data centers are outlined. Section 2 surveys several closely associated research approaches related to this work. Section 3 and Section 4 detail the system models and the research problem formulation. Subsequently, Section 5 elaborates on the algorithms proposed in this research work for solving the formulated stochastic problem. Then, the experimental set-up is presented in Section 6. Section 7 presents the simulation results and discussions about the significance of load balancing using the optimal frequency and the dynamic cooling load. Finally, Section 8 presents the findings of this research work.

3. System Overview

3.1. Power Model

The power consumption of the processor is directly proportional to the frequency. Hardware-based solutions to the problem of power consumption have reached a saturation point. The energy efficiency of multi core systems is entirely dependent on the workload. Cores that have no activity will experience static power loss, which is directly related to the supply voltage. Running a core at maximal workload means it will use the highest frequency and voltage, resulting in high power consumption. By distributing the workload among the processors, the work is completed in the same amount of time with less power consumption. In multi core systems, the only way to address this problem is to maintain the optimal CPU frequency with the minimum energy consumption ratio by distributing the workload. Operating the processor at minimum frequency is a sensible and more reasonable model for achieving minimum power requirements. There is a collective impact on P-states and workload activity on processor temperature [47]. A linear relationship exists between power consumption and the temperature of a processor in a well-cooled environment. DVFS is used to scale the supply voltage and frequency to prevent power wastage. As DVFS has a direct influence on the power consumption and temperature, it can be used as a workable thermal and power control mechanism.

3.2. System Model

Figure 1 presents the overall system model. The description and functionalities of every component are detailed below:

• Resource Management System: The resource management system contains information about the cluster list, PUE, CFR, total utility power, current IT load, and other metadata information related to the data centers.
• Management Node (MN): The Resource Allocation Management (RAM) algorithm is a daemon that is executed in the management node. It is updated with the cluster list, host list, PUE, carbon footprint rate, and other information related to the clusters in the data center. It activates the scheduling algorithm for VM-to-PM mapping and the resource deallocation algorithm to perform resource recovery, and updates target virtual machine queue (TargetVMQ) with VM-to-PM mapping information.
• Cluster Manager (CM): A node in the cluster is nominated as the head node to function as the cluster manager. The overall utilization of the cluster, number of machines in on and off states, maximum and minimum utilization, number of VMs operating in the cluster, and power consumption are maintained by the cluster manager and updated by the management node.
• Physical Machine Manager (PMM): The PM details related to available memory and CPU capacity, current operating frequency, power consumption, percentage of CPU utilization, number of active VMs, and other PM-related information are maintained by the PMM and updated by the CM in the head node.
• Virtual Machine Manager (VMM): The VMM is a daemon that is executed in each PM. It is responsible for maintaining the VMs executing in PM. VM resource utilization, percentage of CPU time utilized, submission time, placement time, active and idle states, remaining execution time, power consumption, and other VM details are maintained by the VMM.

4. Problem Formulation

The VM request is in the form of a triplet (f, r, e), where f ϵ F represents the reserved frequency, r ϵ R represents the resource requirement, and e ϵ I represents the execution interval. Consider M heterogeneous servers, with each containing discrete frequencies (f₀, f₁, f₂, f₃, f₄,…f_k) with utilization (U₀, U₁, U₂, U₃…, U_k),where U₀= 0% (idle), U_k= 100%, and fixed dynamic power consumption (P₀,P₁, P₂ P₃ P₄…, P_k). Here, U₀ is considered as the idle state, with power consumption P₀. Let S = {S₁, S₂, S₃….S_M} represent M servers for each S_j, where jε [1,M] with utilization (U_j,0, U_j,1, U_j,2, U_j,3…, U_j,k), and power consumption (P_j,0, P_j,1, P_j,2, P_j,3, P_j,4…, P_j,k) can be characterized as a triplet (CU_j, CP_j, C_j); where CU_j is the current utilization of server S_j, CP_j is the power consumption of server S_j with utilization state CU_j, and C_j is the total processing capacity of Sj.

The relation R between the j^th PM and i^th VM indicates whether VM_i is placed in PM_j, as below:

$$R_{j,i}=\left\{\begin{array}{c}1 V{M}_{i }is allocated to P{M}_j\\ 0 otherwise\end{array}\right\}$$

(1)

The service level agreement (SLA) is measured using the ratio of virtual machine acceptance (RVA), calculated as:

$$RVA \left(V\right)=T\left(R\right)/N$$

(2)

where N represents the total number of VM requests submitted and T(R) represents the total number of VM requests accepted and mapped to available PMs. This is derived as:

$$T\left(R\right)={\displaystyle \sum }_{j=1}^M{\displaystyle \sum }_{i=1}^N{R}_{j,i}$$

(3)

4.1. Objective Function

4.1.1. Server Power

The VM request of the ith VM in request Queue (ReqQ), (f_i, r_i, e_i), remains constant throughout the execution. Here, e_i is the total number of intervals reserved by the VM for the resource (f_i × r_i). The power consumption of the jth physical machine with utilization U_l at time t is represented as $P_{j,l}\left(t\right)$ and derived as [48]:

$$P_{j,l}\left(t\right)=\left(\frac{({\mathrm{CU}}_{\mathrm{j}}\left(\mathrm{t}\right))-(U_{j,l})}{(U_{j,l+1})-(U_{j,l})}\times \left(\left(P_{j,l+1}\right)-\left(P_{j,l}\right)\right)+\left(P_{j,l}\right)\right)$$

(4)

where U_j,l ˂ CU_j(t) ˂ U_j,l+1, 0 ≤ l ≤ k, where l represents the operating frequency, CU_j(t) is the current utilization of the j^th server at time t, and k is the number of discrete frequencies. The energy consumption of the j^th PM with utilization u within interval [0, I] can be calculated as:

$${{\displaystyle \int }}_0^I{\displaystyle \sum }_{l=0}^k{P}_{j,l}\left(t\right)dt$$

(5)

The total energy consumption of the M number of PMs within a reservation interval [0, I] can be calculated as:

$${\displaystyle \sum }_{j=1}^M{{\displaystyle \int }}_0^I{\displaystyle \sum }_{l=0}^k{P}_{j,l}\left(t\right)dt$$

(6)

4.1.2. Cooling Power

The cooling device power consumption contributes to the maximum electricity consumption of the data center. Dynamic tuning of the cooling load based on the current workload may help reduce power consumption in data centers, which will have a direct impact on the PUE. The cooling power cannot be ignored, as it prevents service disruption caused by the heat generated by servers [49]. To analyze the power consumption of a cooling device, standard computer room air conditioning (CRAC) units are considered in this work. The power consumption of the chiller does not change much with regard to the outside air temperature or IT load [50]. The coefficient of performance (CoP) is the measure used to compute the efficiency of the cooling unit to determine its cooling load. The CoP is the ratio (d/w) of heat removed (for server load d) to the quantity of work (w) needed to remove the heat. A larger CoP indicates better efficiency, meaning less work is required to remove a greater amount of heat. The CoP of the CRAC unit is a changeable value that increases in proportion to the increase of the supply air temperature in the CRAC unit [51].

The total carbon footprint (TCF) generated at time t, including overhead power, is formulated as:

$$TCF\left(t\right)={\displaystyle \sum }_{d=1}^{tdc}(PU{E}_d\times {\displaystyle \sum }_{c=1}^{tc}(CF{R}_{d,c}\times P{S}_{d,c}))$$

(7)

where tdc, tc, M and N represent the numbers of data centers, clusters, machines, and requests, respectively. The overall energy consumption of all servers in a cluster (PS_c) within the interval [0, T], partitioned as a sequence of reservation intervals (ri) in the form of (t_ri, t_ri+1] (ri ϵ {0,1,…ri-1}), is formulated as:

$$P{S}_c={\displaystyle \sum }_{j=1}^M{\displaystyle \sum }_{\alpha =0}^{ri-1}{\displaystyle \sum }_{l=0}^k\times P_{j,l}\left(t_{\alpha +1}\right)\times (t_{\alpha +1}-t_{\alpha })$$

(8)

The CoP for the CRAC unit can be modeled as in [52]:

$$CoP=\left\{0.0068T_{sup}^2+0.0008T_{sup}+0.458 \right\}$$

(9)

where T_sup = (current_temperature − safe_temperature)

The PUE of the data center can be calculated as:

$$PU{E}_d=\left(\frac{Total facility Power}{IT power}\right)$$

(10)

$${\mathrm{Total} \mathrm{facility} \mathrm{Power}}_{\mathrm{d}}=\left({\displaystyle \sum }_{c=1}^{tc}P{S}_c+{\displaystyle \sum }_{c=1}^{tc}O{P}_c\right)$$

(11)

$$IT Powe{r}_d={\displaystyle \sum }_{c=1}^{tc}P{S}_c$$

(12)

The total overhead power (OP) of a cluster (c) is calculated as:

$$O{P}_c=\left(\frac{P{S}_c}{CoP}\right)$$

(13)

The objective function TCF(t) is subject several limitations. The total number of VMs allocated to a machine should not exceed the servers computing (U) and memory capacity (mem), as follows:

$${\displaystyle \sum }_{i=1}^N{R}_{j,i}{}^U\le P{M}_j^{cpu.max}$$

(14)

$${\displaystyle \sum }_{i=1}^N{R}_{j,i}^{mem}\le P{M}_j^{mem.max}$$

(15)

The relation R between VMs and PMs is many-to-one, meaning R ⊆ N × M if:

$$\forall \mathrm{i} ϵ N \&\forall \mathrm{j},\mathrm{k} ϵ M:\left(i,j\right)ϵR\wedge \left(i,k\right)ϵR\Rightarrow j=k$$

(16)

The total energy (eng) consumed is supposed to be within the limit of the available brown energy (B) at the data center, as follows:

$${\displaystyle \sum }_{i=1}^N{R}_{j,i}^{eng}\le Total available B$$

(17)

The total brown energy consumed is supposed to be within the limits of the cloud provider’s agreed upon grid electricity consumption (G):

$$Total available B\le Total assigned G$$

(18)

5. Evaluated Algorithms

5.1. RAMAlgorithm

The high-level design of the resource allocation management (RAM) algorithm executed in the management node is presented in Algorithm 1. The functionality of Algorithm 1 can be grouped into two sections. In Section 1, lines 2–4 perform VM-to-PM mapping using the placement algorithm. In Section 2, lines 5 and 6 perform resource deallocation for every interval.

Algorithm 1: High-level overview of the algorithm approach

Input: Hostlist, VM instancelist

Output: TargetVMQ

1 for interval do

2      ReqQ←Get VMs from VM instance list;

3      HostQ←Get Hosts from HostList;

4       TargetVMQ←Call placement algorithm (presented in Section 5.2, Section 5.3, Section 5.4, Section 5.5 and Section 5.6);

5      if interval >min-exe-time then

6          Completedlist←Get VMs with active time completion from TargetVMQ;

7         for completedlistdo

8            Deallocate resources associated with the VM;

9        Endfor

10   Endif

11 Endfor

12   Return Target VMQ.

5.2. Carbon- and Power-Efficient Optimal Frequency VM Placement (C-PEF)

The C-PEF algorithmic approach is detailed in Algorithm 2. The proposed strategy allocates the new VMs to feasible servers, ensuring:(i)carbon-efficient clusters based on the PUE and carbon footprint rate (CFR);(ii) the power-efficient optimal operating frequency of servers; and(iii) a minimum increase in overall power after allocation.

Algorithm 2: CPEF Carbon and Power-Efficient Optimal Frequency VM Placement

The aim of the C-PEF algorithm is to distribute the load within the cluster. Each server is set to its minimum utilization level. The utilization level is increased gradually when the VM allocation is not feasible at the current utilization level. The greedy selection of the destination hosts for the VMs among the feasible hosts is based on a minimum increase in overall power consumption at the current utilization level. The utilization of each node is reduced to an extent by distributing the load without performance compromise to avoid hotspots due to CPU turbulence. Each node is utilized at the required minimum utilization level as much as possible. Algorithm 2 receives the Clusterlist of all data centers, the Hostlist of each cluster, and the VM resource request through ReqQ. Lines 2 and 3 consolidate the entire cluster list into the Totclusterlist. The algorithm considers the carbon footprint rate (CFR) and power usage effectiveness (PUE) for cluster selection and sorts the Totclusterlist in ascending order based on PUE × CFR. The greedy search, considering power limited to the current utilization level, is performed in line 6 of Algorithm 2. The feasible host system with nominal operating frequency for VM placement is identified as the SelectedHost. The difference in dynamic power before and after VM placement, ΔP, is calculated in line 14 of Algorithm 2. The power consumption P2 is not constant throughout the execution of the VM, as it depends on the next incoming and outgoing tasks of the machine to which it is allocated. As the incoming task is not known in advance, the known details of outgoing tasks based on the remaining execution time and utilization level are used effectively to predict the dynamic power. This approach has an impact if there is a time gap between the first request submission and the next.

The destination host (Desthost) is identified based on the new VM(NVM) execution time and the next outgoing task’s remaining execution time in lines 21–33 in algorithm 2. Figure 2 diagrammatically elucidates lines 21–23 with an example. Assume M₁, M₂, M₃, M₄, and M₅ are machines in the SelectedHost. The execution time of the NVM is assumed to be 8 units. In each machine, P1 is the current dynamic power of the machine, P2 is the dynamic power after the NVM has been placed and R1 represents the remaining execution time interval of each task running in the machine at time t_o. The hosts are sorted based on ΔP. The SelectedHosts are {M₄,M₃,M₂,M₅,M₁}. The number of tasks with the minimum remaining execution time (R1) at time t_o in M₁ is 4(Task₂), for M₂ is 9(Task₂),for M₃ is 5(Task₁), forM₄ is 3(Task₁), and for M₅ is 2(Task₁). Here, the execution time of NVM–R1 (ΔR) forM₁ is 4, for M₂ is (−1), for M₃is3, for M₄is5, and for M₅is6. M₂ has a greater minimum remaining execution time than the NVM execution time, so the selectedhost = {M₄}, while all others are considered as “choosyhosts”. Pow1 represents the assumed dynamic power after the completion of the task with the minimum remaining execution time. Based on C-Exp-Pow, M₃ is chosen as the Desthost, irrespective of M₄, which has the minimum ΔP.

The time complexity of Algorithm 2 can be analyzed by considering n VM requests in ReqQ. The sort function inline 3 with c clusters takes O(c log(c)) times. Considering f frequency levels, line 8–17 and line 25–27 with m number of hosts take O (m) times. The sort function in lines 21 and 29 takes O (mlog(m)) times. The algorithm complexity is derived as O (n (clog(c) + cf (m + m log m + m+ m log m))).The final complexity is O(ncfmlog(m))).

5.3. Carbon-Aware First-Fit Optimal Frequency VM Placement (C-FFF)

The C-FFF algorithmic approach is detailed in Algorithm 3. The aim of the C-FFF algorithm is to distribute the load within the cluster. Each server is set to its minimum utilization level. The utilization level is increased gradually when the VM allocation is not feasible at the current utilization level. This approach differs from C-PEF in terms of host selection. There is no greedy selection performed for the minimum increase in overall power consumption, and the VM is placed in the first-fit host when feasibility is confirmed at the current utilization level. Algorithm 3 receives the Clusterlist of all data centers, the Hostlist of each cluster, and the VM resource request through ReqQ. Lines 2 and 3 consolidate the entire cluster list into the Totclusterlist. The algorithm considers the carbon footprint rate (CFR) and power usage effectiveness (PUE) for cluster selection and sorts the Totclusterlist in ascending order based on PUE×CFR. The C-FFF algorithm differs from C-PEF in terms of host selection. C-FFF does not use the greedy approach on the feasible host with minimum power, as with C-PEF; instead, it places the VM in the first feasible host that is limited to the current p-state. For n VM requests, m number of hosts, f frequency levels, and c number of clusters, the complexity of the algorithm is derived as O(nfmclog(c)).

5.4. Carbon- and Power-Efficient VM Placement (C-PE)

The cluster selection is the same as with C-FFF, meaning it is based on PUE and CFR. The standard power-efficient algorithm does consider the DVFS and remaining execution time for outgoing tasks for VM allocation [25]. In this work, the C-PE algorithm performs cluster selection similarly to C-PEF and C-FFF, but differs in its host selection policy. The aim of the C-PE algorithm is to find a feasible host for a VM, considering the maximum utilization level. The host selection is based on a minimum increase in overall power consumption (i.e., minimum ΔP). The Selected Hosts are sorted based on estimated ΔP (line 14of C-PEF). The destination host is selected as in Algorithm 2, with maximum utilization. The algorithm complexity with n VM requests, c clusters, and m nodes is derived as O(n(clog(c) + c(m + mlog(m) + m))). The final complexity is expressed as O(ncmlog(m)).

5.5. Carbon-Aware First-Fit Least-Empty VM Placement (C-FFLE)

This approach performs data center and cluster selections similarly to C-PEF, C-FFF, and C-PE, but differs in terms of its host selection policy. The C-FFLE algorithm considers the carbon footprint rate (CFR) and power usage effectiveness (PUE) for cluster selection and sorts the Totclusterlist in ascending order based on PUE×CFR. The host selection is based on the first-fit strategy, whereby the hosts are ordered based on the least available resources. This approach does not perform any greedy searching for minimum power heuristic methods; instead, it uses VM best-fit heuristic methods based on resource requirements for node selection.

Algorithm 3: C-FFF Carbon-Aware First-Fit Least-Empty VM Placement

5.6. Carbon-Aware First-Fit VM Placement (C-FF)

The cluster selection by the C-FF algorithm is similar to C-PEF,C-FFF,C-FFLE, and C-PE algorithms, but differs in terms of the host selection policy. The algorithm considers the carbon footprint rate (CFR) and power usage effectiveness (PUE) for data center selection and sorts the Totclusterlist in ascending order based on PUE×CFR. It uses first-fit heuristic methods for host selection.

6. Experimental Environment and Assumptions

Considering the expense and time incurred in the evaluation of large-scale experiments in real time, Matlab software is used to simulate the environment. Each reservation interval is assumed to have duration of 300 s. The input request is accepted at the beginning of each reservation cycle. A data center with heterogeneous systems with different power models capable of provisioning multiple VMs is considered. The virtual resource size is not known and the VM request has no limitations. The VM is assumed to be active throughout the execution time. All the tasks are considered to be CPU-intensive. The power consumption of a task is measured by its CPU utilization, as this is considered to consume a significant fraction of energy. All the machines are assumed to be in an off state when not in use. The VM’s resource requirements are assumed to be constant throughout the reservation interval. The data center’s safe operating temperature is considered to be 23 °C. The peak IT load (server only) power estimation for the data center is 52 kW for the physical machine specifications given in

Table 1. Physical machine characteristics [24].

Machines	Frequency(GHz)	No. of Cores	Power Model	Memory(GB)	Storage (GB)	Network Bandwidth (Mbps)
M1	2	2	1	16	2000	1000
M2	4	4	1	32	6000	1000
M3	4	8	2	32	7000	2000
M4	8	8	2	64	7000	4000
M5	16	8	2	128	9000	4000

[53]. The data center is assumed to have a floor space of approximately 500 square feet. The total electricity power requirement is calculated as 124 kW (including cooling and lighting load). The CPU power consumption for all servers should not exceed 17.3kW. The cooling load concerning CPU utilization is limited to 12.11 kW [54]. The data centers are assumed to be powered only by grid energy sources.

Physical Machine and VM Reservation Modeling

Table 1 shows model of physical machines with varying power models to simulate heterogeneity and configurations of heterogeneous systems taken from the SPEC power benchmark [53] used in the simulation.

Table 2. Power (in watts) model of physical machines (PMs) [24].

Power Model	Idle	10%	20%	30%	40%	50%	60%	70%	80%	90%	100%
1	60	63	66.8	71.3	76.8	83.2	90.7	100	111.5	125.4	140.7
2	41.6	46.7	52.3	57.9	65.4	73	80.7	89.5	99.6	105	113

presents the power consumption, with equal CPU utilization distribution ranging from 0% to 100%. The power calculation for the periods in between intervals is estimated based on Equation (4). For example, the power consumption with 13% CPU utilization for power model 1 is between 10% and 20%,while the resulting power is 64.14 W based on ((13%−10%)/(20%−10%) ×(66−63)) + 63 with reference to Table 2.

To evaluate the proposed algorithms, 4 small-scale data centers with 100 heterogeneous systems are used to model infrastructure-as-a-service(IaaS).The VM characteristics for elastic compute units (ECU) shown in

Table 3. Virtual machine request types [24].

Name	ECU	Core Speed(GHz)	Memory(MB)	Storage(GB)	Network Band Width(Mbps)	Probability
M1.small	1	1	1740	160	500	0.25-BT
M1.large	2	4	7680	850	500	0.25-BT/0.12-WR
M1.xlarge	4	8	15,360	1000	1000	0.08-WR
M2xlarge	2	6.5	17,510	1000	1000	0.12-WR
M22.xlarge	4	13	35,020	1000	1000	0.08-WR
C1.median	2	5	1740	500	500	0.1-BT

are used to model the virtual machine reservations. Each data center is assumed to have 2 clusters with varying values for carbon footprint rates. The carbon footprint rates of clusters and PUE values of data centers are considered based on [55,56], as presented in

Table 4. Datacenter features [24].

Data Center	Carbon Footprint Rate (CFR) in Tons/MWh	PUE
DC1	0.124, 0.147	1.56
DC2	0.350, 0.658	1.7
DC3	0.466, 0.782	1.9
DC4	0.678, 0.730	2.1

. The workload is generated based on the Lublin–Feitelson model [57]. By taking advantage of the arrival rate, gamma, and hyper-gamma Lublin parameters, the bag-of-tasks and web requests are generated, which have long and short holding times, respectively, as compared to the VM types given in Table 3 (shown in Figure 3 and Figure 4). Figure 3 depicts the variation in the numbers of requests in each reservation cycle. Figure 4 presents the total number of CPU utilization requests received from the VMs concerning different reservation cycles.

7. Results and Discussions

The workload data described above are used to evaluate the proposed VM placement algorithms C-PEF and C-FFF against C-PE, C-FF, and C-FFLE approaches. The C-FFLE algorithm is used to show the impact on power consumption when only resource usage is considered as a parameter in the heuristic approach. The C-FF is the first-fit placement algorithm, which is used for initial placement for all algorithms in this work. The other algorithms improve the placement strategy for power reduction as an extension of C-FF. In this work, along with initial placement, C-FF is used separately to model the worst possible power consumption. Naturally, the C-FFLE and C-FF algorithms will have worse performance than the power management algorithms. The C-PE algorithm is considered as a fair measure to evaluate power management approaches.

• Reduction in Overall Carbon Footprint

The reduction of grid energy consumption in datacenters is considered as a crucial metric for carbon footprint reduction. Equation (7) formulates the total carbon footprint emission of data centers.

• The ratio of VM acceptance (RVA)

The RVA is considered as a measure of service level agreement (SLA). The RVA is the ratio between the number of VM requests placed and the number of requests submitted.

7.1. Scenario-I: Energy-Efficient Mapping of VMs to PMs with StaticPUE

The VM placement algorithms are evaluated based on the reduction of the carbon footprint with static PUE, as shown in Table 4. Figure 5 and

Table 5. Power consumption for 100% ratio of VM acceptance (RVA).

Interval (300 s)	Active VMs	Power Consumption (kW)					Total Active PMs					Total CPU Utilization %
		C-PEF	C-PE	C-FFF	C-FFLE	C-FF	C-PEF	C-PE	C-FFF	C-FFLE	C-FF
1	64	1512.53	1436.84	1626.45	2070.33	1954.24	50	36	56	60	56	10.94
2	123	2596.98	2624.79	2879.36	3624.09	3185.06	89	74	98	106	91	20.90
3	181	3646.54	4106.17	4003.47	5360.29	4935.64	111	116	123	151	136	29.34
4	237	4997.41	5505.73	5272.04	6951.47	6278.71	141	152	153	196	172	38.46
5	288	6228.52	6717.20	6270.41	8246.13	7189.17	175	186	175	231	199	47.02
6	334	7056.55	7453.36	7207.61	9257.91	8179.25	198	208	203	258	227	54.36
7	381	8010.12	8428.66	8295.90	10295.59	9043.74	227	238	238	289	252	63.09
8	420	8770.99	9264.71	8951.71	11077.27	9856.81	253	262	258	312	276	69.76

present the number of active PMs for the same utilization level, with 100% RVA for all algorithms. In order to interpret the number of active PMs shown in Table 5, this has to be compared with the minimum and maximum utilization levels given in

Table 6. Processor utilization for 100% RVA.

Reservation Interval	Minimum CPU Utilization %					Number of Hosts with 100% CPU Utilization
	C-PEF	C-PE	C-FFF	C-FFLE	C-FF	C-PEF	C-PE	C-FFF	C-FFLE	C-FF
1	40.625	50	25	50	50	17	23	17	35	33
2	31.25	62.5	25	50	62.5	19	42	27	62	56
3	40.625	50	40.625	62.5	62.5	56	75	64	96	91
4	62.5	62.5	40.625	50	62.5	91	101	94	127	118
5	62.5	50	50	50	62.5	113	121	117	152	138
6	50	40.625	40.625	62.5	62.5	125	146	135	173	159
7	62.5	62.5	40.625	50	40.625	142	162	153	188	172
8	31.25	62.5	31.25	40.625	62.5	156	176	164	204	189

for each interval. In C-PE, for 10% utilization, the number of active PMs is 36 with a minimum utilization 50% and the number of PMs with 100% utilization is 23. These numbers are far greater than C-FFF and C-PEF, for which the number of active PMs is 17, with minimum utilization ranging between 25% and 40.6%. The C-PE placement strategy utilizes a lower number of PMs with the maximum utilization possible for the current workload, but the C-FFF and C-PEF algorithms utilize the maximum number of PMs with the minimum possible utilization level and a lower number of fully utilized PMs. The results show that distribution of the load among the servers using DVFS with C-FFF and C-PE algorithms limits the percentage of load received at each interval. This approach does not lead to optimal results with very low loads. The minimum load required for best result depends on the machine configuration and power model. According to our specifications, repeated execution with different VM requests shows that20% is the minimum load. For the C-PEF algorithm, the optimal load requirement is less than C-FFF, because in spite of DVFS, it uses the greedy approach, which limits load distribution. It can be noticed that the C-FF algorithm achieves a significant improvement over C-FFLE, displaying a trade-off between effective resource utilization and power consumption. The utilization results presented in Table 6 prove the above algorithm strategies. Figure 6a,b and Table 5 illustrates the power consumption for all algorithms at 100% RVA for the first 8 intervals, the power consumption of the C-PEF algorithm is 3.79% lower than for C-FFF, while the power consumption for C-FFF is 2.26% lower than C-PE, 21.75% lower than C-FFLE, and 12.08% lower than C-FF. Based on the cumulative carbon footprint depicted in Figure 6b, which is equivalent to the power given in Table 5, the C-PEF reduces the carbon footprint to 4.09%, while C-PE, C-FFLE, and C-FF reduce the carbon footprint to 3.35%, 38.8%, and 17.6%, respectively. Based on Figure 7a and

Table 7. Power and carbon footprint for different VM placement algorithms.

Placement Algorithm	Power (kW)	Carbon Footprint(Tons)	Number of VMs Placed
C-FFF	676,296.2775	48.72382575	1634
C-PE	691,256.2894	49.87335009	1611
C-PEF	665,341.0031	48.28496006	1623
C-FFLE	782,225.6419	59.79732901	1622
C-FF	695,623.92	50.4902789	1598

, the total power consumed by the servers using the C-PEF placement algorithm is reduced by 1.61%. The C-FFF algorithm reduces power consumption by2.16%, 13.54%, 2.77% when compared to C-PE, C-FFLE, and C-FF, respectively. Based on Figure 7b and Table 7, the C-PEF placement algorithm’s carbon emission is 1.64% less than C-FFF. The C-FFF placement algorithm consumes 2%, 15%, and 2.8% less power than C-PE, C-FFLE, and C-FF, respectively.

In Table 5, the carbon footprint values for C-FFF and C-PEF are about 2.30% and 3.18% lower than for C-PE, respectively, with an increased RVA of 1.2%. C-FFLE and C-FF algorithms have 19.89% and 1.23% greater carbon footprints than C-PE, respectively.

Table 8. RVA for all VM placement algorithms.

RVA% Under Different VM Requests
Algorithm	481	910	1276	1591	1861	2000
C-FFF	100	88.68132	81.5047	81.58391	81.30038	81.7909
C-PE	100	87.03297	80.17241	80.32684	80.06448	80.5903
C-PEF	100	88.35165	81.03448	81.01823	80.65556	81.14057
C-FFLE	100	87.25275	80.721	81.26964	80.60183	81.14057
C-FF	100	87.03297	80.01567	80.01257	79.36593	79.93997

depicts the substantial improvement in VM request acceptance for C-PEF and C-FFF algorithms compared with other heuristic approaches with different numbers of VM requests. The C-FFF and C-PEF algorithms have approximately 1% higher RVA percentage than other counterparts. The statistical analysis presented in

Table 9. Statistical analysis of different VM placement algorithms.

	C-FFF			C-PE		C-PEF		C-FFLE		C-FF
Metric	% CPU Utilization	Power (kW)	% CPU Utilization		Power (kW)	% CPU Utilization	Power (kW)	% CPU Utilization	Power (kW)	% CPU Utilization	Power (kW)
Min	0.1471	1626	0.1471		1437	0.1471	1513	0.1471	2070	0.1471	1954
Max	89.12	6.76 × 105	90.04		6.91 × 105 5	88.38	6.65 × 105	92.46	7.82 × 105	89.87	6.96 × 105
Mean	48.6	4.49 × 105	48.91		4.58 × 105	47.82	4.42 × 105	50.46	5.17 × 105	48.97	4.64 × 105

supports the fact that the C-FFLE algorithm, which is based on resource utilization, operates with maximum utilization compared to other approaches. The C-FFF, C-PE, and C-PEF algorithms have approximately 1% variation in maximum CPU utilization rates, with similar average utilization rates. With regard to power, the C-PEF placement algorithm’s power consumption is 1.64% less than C-FFF. C-FFF consumes 2%, 15%, and 2.8% less power than C-PE, C-FFLE, and C-FF, respectively.

The results in Figure 8a,b were obtained by varying the system load with respect to the number of requests in order to measure the power consumption of different algorithms for a single interval with a common initial state. This was done so as to rank the performance of the algorithms from lowest to highest in terms of CPU utilization. Figure 8a displays the power consumption values for all the algorithms, with CPU utilization rates ranging between 40% and 90%. It can be noticed that the C-PEF algorithm shows significant performance improvement between 40%and 85% utilization. Figure 8b presents the carbon footprint values for all of the algorithms for utilization rates above 85%. Above 90% utilization, C-PEF and C-FFF are in close proximity to each other.

Figure 9 and Figure 10 present the amounts of carbon and power consumed by different power-efficient VM placement algorithms. C-PEF and C-FFF algorithms consume more power initially at lower loads than C-PE, which distributes the loads among all the servers. C-FFLE and C-FF algorithms consume more power as they do not utilize power-efficient allocation. The non-parametric Mann–Whitney U test and Wilcoxon rank sum test are utilized to test whether there is a noteworthy difference in the results obtained. Based on the abovementioned non-parametric tests on two samples for C-PEF with C-PE, C-FFLE, and C-FF, the p-values obtained are less than 0.0001. Therefore, it can be concluded that DVFS-aware scheduling (C-PEF and C-FFF) makes a significant difference compared with standard power-aware scheduling(C-PE) and other heuristic approaches interms of energy consumption. The difference between the two DVFS-aware algorithms, C-PEF and C-FFF, is not substantial (p-value of 0.76 > 0.05).

7.2. Scenario-II: Energy-Efficient Mapping of VMs to PMs with DynamicPUE

The power usage effectiveness is the metric used to analyze the efficiency of a datacenter. This is the ratio between the total energy requirements of a data center (total facility power) and the power consumed by IT devices. In total, 60% of the energy consumption is due to cooling device power consumption, which has a direct impact on PUE. The proposed power-aware algorithms C-FFF and C-PEF, along with the standard C-PE algorithm, are considered in scenario II to analyze the impact of dynamic PUE on carbon footprint values, based on Equation (11).

Table 10. Power and carbon footprint with dynamic power usage effectiveness (PUE).

Placement Algorithm	Power(kW)	Carbon Footprint (Tons)	Number of VMs Placed
C-PEF	564,350.9	23.631227	1617
C-PE	628,328.4	24.191322	1636
C-FFF	582,335.1	24.35445	1641

presents the power consumption and carbon footprint values observed with dynamic PUE under the same workload used for the observed values in Table 5 for fair comparison. Dynamic PUE reduces the carbon footprint by approximately 50%, as shown in Table 10. The RVA percentages for C-FFF and C-PE displayed in

Table 11. RVA % with dynamic PUE.

RVA% with Different VM Requests
Algorithm	481	910	1276	1591	1861	2000
C-PEF	98.96	87.14	81.42	80.95	80.65	80.89
C-PE	100	88.4	81.97	82.08	81.67	82.09
C-FFF	99.37	88.69	82.66	83.58	82.3	82.84

show a slight dip at the beginning and then a significant increase of 1%.

Table 12. Statistical analysis with dynamic PUE.

	C-FFF		C-PE		C-PEF
Metric	%CPU Utilization	Power (kW)	%CPU Utilization	Power (kW)	%CPU Utilization	Power (kW)
Min	0.1471	1029	0.1471	644.5	0.1471	969
Max	88.94	5.823 × 105	89.67	6.283 × 105	86.69	5.644 × 105
Mean	48.33	3.854 × 105	48.48	4.156 × 105	46.25	3.74 × 105

shows the overall statistics for CPU utilization and power consumption related to dynamic PUE. The values in Table 10 confirm the impact of dynamic PUE. The power consumption of the C-PE algorithm is reduced by approximately 55% compared to static PUE. The power consumption for the mean CPU utilization presented in Table 9 and Table 12 reveals the impact of dynamic PUE on power reduction. C-FFF, C-PE, and C-PEF algorithms achieve approximately 14%, 9%, and 15% greater reductions than static PUE. The results support the approach of energy reduction by dynamically adjusting the cooling device load based on the active power consumption of the server for the current workload.

Let n, f, m, and c represent the number of VM requests, the fixed DVFS levels, the number of nodes, and number of clusters, respectively. The complexity of the C-PE algorithm is expressed by O(ncmlog(m)). Its complexity is dominated by mlog(m). The complexity of the proposed C-PEF algorithm is expressed by O(nfcmlog(m)). The C-PEF complexity is f times that of C-PE. The complexity of the C-FFF algorithm O(nfmclog(c)) is dominated by f (fixed frequency level) and clogc. As the number of nodes (m) in the data center increases, the complexity of the C-PE dominates the overhead caused by the constant f in C-FFF. The proposed C-FFF algorithm with complexity O(nfmclog(c)) performs load balancing, while maintaining a better tradeoff between utilization and power consumption than the standard C-PE algorithm with complexity O(ncmlog(m)).

8. Conclusions

Energy consumption and carbon footprint problems in data centers are handled using different VM placement algorithms with static and dynamic PUE. The data center energy efficiency metric PUE and carbon usage effectiveness are used as important measures for data center selection. The proposed C-FFF and C-PEF placement algorithms perform placement decisions by maintaining the optimal p-state of the servers. In C-PEF, host selection is based on the power-efficient optimal p-state of the servers. In C-FFF, the host selection is based on the optimal p-state of the servers. Both C-FFF and C-PE are compared with a standard power-efficient algorithm (C-PE), where the host selection is based on the highest power-efficient p-state of the servers. Different VM types with varying execution times and arrival rates are used to simulate the system load. The resulting outcomes for scenario I reveal that C-FFF can reduce the carbon footprint by a minimum of 2% more than C-PE, C-FFLE, and C-FF. The experimental results illustrate the importance of considering the DVFS of the servers, along with PUE and carbon release of clusters in data centers. The results for the algorithms in scenario II emphasize the impact of dynamic PUE on the carbon footprints. The C-FF algorithm shows significant improvement over C-FFLE in power reduction, displaying a trade-off between effective resource utilization and power consumption. Among the three power-aware algorithms, C-PEF and C-PE have additional computational overhead due to greedy search function. The results support the fact that C-FFF balances computational overhead and utilization, and stands in between C-PEF and C-PE with some degree of minimum resource request constraint. In conclusion, C-FFF is a power-efficient algorithm for VM placement with reduced computational overhead. The formulations presented in this work open new and challenging areas of further research relating to renewable energy sources.