Task-Level Aware Scheduling of Energy-Constrained Applications on Heterogeneous Multi-Core System

Abstract

Minimizing the schedule length of parallel applications, which run on a heterogeneous multi-core system and are subject to energy consumption constraints, has recently attracted much attention. The key point of this problem is the strategy to pre-allocate the energy consumption of unscheduled tasks. Previous articles used the minimum value, average value or a power consumption weight value as the pre-allocation energy consumption of tasks. However, they all ignored the different levels of tasks. The tasks in different task levels have different impact on the overall schedule length when they are allocated the same energy consumption. Considering the task levels, we designed a novel task energy consumption pre-allocation strategy that is conducive to minimizing the scheduling time and developed a novel task schedule algorithm based on it. After getting the preliminary scheduling results, we also proposed a task execution frequency re-adjustment mechanism that can re-adjust the execution frequency of tasks, to further reduce the overall schedule length. We carried out a considerable number of experiments with practical parallel application models. The results of the experiments show that our method can reach better performance compared with the existing algorithms.

1. Introduction

Computer systems nowadays must perform much better than ever before, ensuring the simultaneous running of many applications. By using heterogeneous multi-core processors and increasing the number of processor cores, it is possible to improve the performance while keeping energy consumption at the bay [1,2,3,4,5]. From small, embedded devices to large data centers, heterogeneous multi-core systems have been widely used. It is expected that in the near future, the number of heterogeneous processors and cores in these systems will increase dramatically [6,7,8]. On the other hand, although the performance of such systems has been greatly improved, the power consumption is also increasing. Huge energy consumption has caused various problems, such as economy, environment, technology and so on [9,10,11,12,13]. Therefore, energy consumption is one of the main design constraints for such heterogeneous multi-core systems. A well-known typical mechanism for reducing power consumption of computing systems is dynamic voltage and frequency scaling (DVFS), which is realized to achieve the balance between energy consumption and performance by reducing the power supply voltage and frequency of the processor at the same time, when the task is running [1,14,15,16,17,18,19]. Therefore, energy consumption constrained task scheduling on processors with adjustable voltage and frequency has attracted extensive research [20,21,22,23].

In the existing research, works related to the energy consumption of heterogeneous multi-core processors, some of the objectives are to minimize the energy consumption under certain performance requirements. Others are to minimize or maximize other indicators, such as performance and reliability, under certain energy consumption constraints [24,25,26,27,28,29,30,31]. Our studies fall into the second category. To be more precise, we study the following problem: minimizing the schedule length of energy consumption constrained parallel applications on heterogeneous multi-core systems. There have been some studies in recent years that are consistent with our goals. In order to determine the energy consumption constraint of each task, the common method in previous studies is to pre-allocate the energy consumption of the unscheduled tasks [20,21,22,23]. Therefore, these studies focus on how to allocate the overall energy consumption constraints to each task reasonably. Xiao et al. [20] came up with an algorithm called MSLECC (minimizing schedule length of energy consumption constrained). It takes the minimum energy consumption of a task as its pre-allocation energy consumption. The drawback of this method is that it is unfair to low priority tasks. High priority tasks take up too much energy consumption, so that low priority tasks can only be allocated to low-power processor cores and close to the lowest execution frequency due to the energy consumption constraint, which will lead to an increase in the overall schedule length. After that, some studies improved the energy pre-allocation method, so that the energy pre-allocation is fair to each task. One study pre-allocated the energy consumption according to the task execution time proportion [21], another study used the average energy consumption as the pre-allocation energy consumption of each task [22], and another study used a defined task energy consumption weight value as the pre-allocation energy consumption of tasks, whose algorithm is called ISAECC [23]. However, it is not the best practice to treat each task fairly and the task level is an important issue when pre-allocating energy consumption to the tasks. We pre-allocate more energy consumption to the tasks in the task levels with fewer tasks, because when their execution time becomes shorter, the overall scheduling length is more likely to decrease. In addition, previous articles ignored the negative impact of local optimal scheduling algorithm. Therefore, we develop a task execution frequency re-adjustment mechanism that also uses DVFS to further reduce the schedule length. To summarize, the main contributions of this article are as follows.

• We design a novel energy pre-allocation strategy considering task level and prove its feasibility.
• We develop a novel scheduling algorithm to minimize the schedule length under energy consumption constraints based on the new energy pre-allocation strategy.
• We introduce a frequency re-adjustment mechanism after task scheduling to reduce the negative impact of local optimization.
• We evaluate our algorithm based on real parallel applications. The experimental results consistently prove the superiority and competitiveness of our algorithm.

The structure of this article is as follows. Section 2 reviews some existing studies that are relevant to us. Section 3 gives some preliminaries related to the problem of minimizing the schedule length for energy consumption constrained parallel applications. In Section 4, we present our approach for this problem. In Section 5, we discuss and analyze the experimental results. Finally, we conclude the paper in Section 6.

2. Related Work

Energy saving design technology based on DVFS was first proposed by [1]. Nowadays, DVFS has been widely used in multi-core task scheduling problems related to energy consumption. Reference [2] studied the problem of minimizing the schedule length of independent sequential applications with energy consumption constraints. In Reference [29], the task scheduling problem with energy consumption constraints is considered as a combinatorial optimization problem. In Reference [31], the authors consider three constraints (energy consumption, deadline and reward). These studies are mainly focused on homogeneous systems, so they are different from our study.

In addition to the above research, many researchers have studied the task scheduling problem on heterogeneous multi-core systems. For example, reference [32] proposed an energy-saving workflow task scheduling algorithm based on DVFS. Huang et al. [33] proposed an enhanced energy-saving scheduling algorithm to minimize energy consumption under the condition of satisfying a certain performance level. Rusu et al. [31] added constraints and proposed an efficient algorithm for minimizing energy consumption under multiple constraints. The goal of these studies is generally contrary to ours. We study the minimization of the schedule length under energy consumption constraints, while those studies focus on minimizing energy consumption under other constraints.

There are also a lot of excellent studies that are closely related to our study. For example, a representative paper proposed the classical Heterogeneous Earliest Finish Time (HEFT) algorithm, which was developed to minimize the schedule length in heterogeneous multicore systems. The application model and energy consumption model they use are consistent with ours, but they do not consider the constraints of energy consumption. At present, the studies close to us should be the four articles mentioned in Section 1 [20,21,22,23]. They pre-allocate energy consumption to each task according to the minimum energy consumption of the task [20], the execution time ratio of the task [21], the overall average energy consumption [22], or the defined task energy consumption weight [23]. What is not considered in the above studies is that different task hierarchies have different impacts on the overall schedule length. Moreover, they ignored the negative impact of the local optimal characteristics of scheduling algorithm on the schedule length. Based on this, we propose a novel energy pre-allocation strategy considering task level, and, in order to reduce the negative impact of local optimal characteristics, we develop a scheduling task execution frequency re-adjustment mechanism. Finally, our method achieves better performance than previous studies.

3. Models and Preliminaries

In this section, we first introduce the application model (Section 3.1), then the energy consumption model (Section 3.2), and next we describe in detail the issues that need to be addressed (Section 3.3). Finally, we briefly introduce the current situation and reveal its limitations (Section 3.4).

Table 1. The main notations we use.

Notation	Description
$w_{i,k}$	Execution time of task $n_i$ on the processor core $u_k$ with the maximum frequency
$c_{i,j}$	Communication time from $n_i$ to $n_j$
$pred\left(n_i\right)$	The set of direct predecessor tasks of task $n_i$
$succ\left(n_i\right)$	The set of direct successor tasks of task $n_i$
$n_{entry}$	Entry task of an application
$n_{exit}$	Exit task of an application
$E\left(n_i,u_k,f_{k,h}\right)$	The energy consumption of the task $n_i$ on the processor core $u_k$ with the frequency $f_{k,h}$
$EST\left(n_i,u_k\right)$	The earliest start time of task $n_i$ running on processor core $u_k$
$EFT\left(n_i,u_k,f_{k,h}\right)$	The earliest finish time of task $n_i$ running on processor core $u_k$ with frequency $f_{k,h}$
$AST\left(n_i\right)$	The actual start time of task $n_i$
$AET\left(n_i\right)$	The actual execution time of task $n_i$
$AFT\left(n_i\right)$	The actual finish time of task $n_i$
$LFT\left(n_i\right)$	The latest finish time of task $n_i$
$L\left(n_i\right)$	The level of task $n_i$
$u_{pr\left(i\right)}$	The processor core allocated to task $n_i$
$f_{pr\left(i\right),hz\left(i\right)}$	The execution frequency allocated to task $n_i$ on processor core $u_{pr\left(i\right)}$
$E_{given}\left(n_i\right)$	The calculated energy consumption constraint of task $n_i$
$E_{pre}\left(n_i\right)$	The pre-allocated energy consumption of task $n_i$
$E_{given}\left(G\right)$	The given energy consumption constraint of application G
$E\left(G\right)$	The energy consumption of application G
$SL\left(G\right)$	The schedule length of application G

shows the main notations we use.

3.1. Application Model

As in previous studies [20,21,22,23,24,34], we also use directed acyclic graph (DAG) to represent parallel application models. As for the processor cores, we define $U=\left\{u_1,u_2,\dots ,u_k,\dots ,u_{\left|U\right|}\right\}$ to represent a collection of processor cores, where $\left|U\right|$ is defined as the number of processor cores. Note that for any set $X$, we use $\left|X\right|$ to denote its size. We define the DAG application model as $G=\left\{N,W,C\right\}$. $N=\left\{n_1,n_2,\dots ,n_i,\dots ,n_{\left|N\right|}\right\}$ represents the set of nodes in the graph, that is, the set of tasks in the application. Due to the heterogeneous nature of the processor, the execution time of $n_i\in N$ on different processor cores is different. $W$ refers to a matrix with size $\left|N\right|\times \left|U\right|$, where $w_{i,k}$ denotes the execution time for $n_i$ to run on $u_k$ with the maximum frequency. $C$ denotes the weight of edges between connected nodes in DAG, that is, the communication time between tasks. $c_{i,j}\in C$ represents the communication time from $n_i$ to $n_j$. If $c_{i,j}=0$, it means there is no communication from $n_i$ to $n_j$. We define $pred\left(n_i\right)$ and $succ\left(n_i\right)$ as the set of direct predecessor tasks and the set of direct successor tasks of task $n_i$. For example, $pred\left(n_2\right)=\left\{n_1\right\}$ and $succ\left(n_2\right)=\left\{n_8,n_9\right\}$. We define $n_{entry}$ and $n_{exit}$ as the entry task and the exit task of an application. In Figure 1, $n_{entry}$ and $n_{exit}$ are $n_1$ and $n_{10}$.

Figure 1 shows an example of a parallel application based on DAG with ten tasks. Each node in Figure 1 represents a task, and the values on the edges between connecting nodes represent the communication time between the two nodes if they are not assigned to the same processor core. For example, the value 18 on the edge between $n_1$ and $n_2$ indicates that the communication time between $n_1$ and $n_2$ is 18.

Assuming that there are three heterogeneous processor cores $\left\{u_1,u_2,u_3\right\}$ in the system,

Table 2. Execution time of tasks on different processors with the maximum frequency of the application in Figure 1.

	Task	${\mathit{n}}_1$	${\mathit{n}}_2$	${\mathit{n}}_3$	${\mathit{n}}_4$	${\mathit{n}}_5$	${\mathit{n}}_6$	${\mathit{n}}_7$	${\mathit{n}}_8$	${\mathit{n}}_9$	${\mathit{n}}_{10}$
Core
$u_1$		14	13	11	13	12	13	7	5	18	21
$u_2$		16	19	13	8	13	16	15	11	12	7
$u_3$		9	18	19	17	10	9	11	14	20	16

shows the execution time of the tasks in Figure 1 running on each processor core with the maximum frequency. For example, the first number 14 in Table 2 indicates that the execution time of $n_1$ running on $u_1$ with the maximum frequency is 14.

3.2. Energy Model

In DVFS technology, the relationship between supply voltage and operating frequency is almost linear. Therefore, DVFS will also adjust the supply voltage when adjusting the clock frequency. Similar to [20,21,22,23], we use frequency regulation to indicate simultaneous regulation of supply voltage and frequency. In this article, we use the same energy model as the references [20,21,22,23]. Therefore, the calculation formula of system power consumption with respect to frequency is as follows:

$$P\left(f\right)=P_s+h(P_{ind}+P_d)=P_s+h(P_{ind}+C_{ef}{f}^m).$$

(1)

In the above equation, $P_s$ denotes static power and can only be removed when the system is completely powered down. $P_{ind}$ is a constant that represents the frequency-independent dynamic power, that is, it corresponds to power independent of CPU processing speed. $P_d$ denotes frequency-dependent power, including the power primarily consumed by the CPU and any power that depends on the system processing frequency $f$. $h$ denotes the system state, specifically, $h=1$ means the system is active and the application is executing; $h=0$ means the system is in the sleep mode or powered down. $C_{ef}$ denotes the effective capacitance and $m$ denotes the dynamic power exponent and is no smaller than 2. $C_{ef}$ and $m$ are constants related to the processor system.

Our study is in the active state of the system ($h=1$), so dynamic power consumption is the main part of the whole energy consumption. Considering the unmanageability of static power consumption, this article, like references [20,21,22,23], does not consider static power consumption. Therefore, the calculation formula of system power consumption in this article becomes the following equation:

$$P\left(f\right)=P_{ind}+C_{ef}{f}^m.$$

(2)

Due to the heterogeneity of processors, each processor should have its own parameters. Assuming that the frequency range of the processor $u_k$ is from the lowest frequency $f_{min}$ to maximum frequency $f_{max}$ temporarily, we define the following sets of parameters:

• The set of $P_{ind}$: $\left\{P_{1,ind},P_{2,ind},\dots ,P_{\left|U\right|,ind}\right\}$;
• The set of $P_d$: $\left\{P_{1,d},P_{2,d},\dots ,P_{\left|U\right|,d}\right\}$;
• The set of $C_{ef}$:$\left\{C_{1,ef},C_{2,ef},\dots ,C_{\left|U\right|,ef}\right\}$;
• The set of $m$: $\left\{m_1,m_2,\dots ,m_{\left|U\right|}\right\}$;

The set of execution frequency:

$$\left\{\begin{array}{c}\left\{f_{1,min},f_{1,a},\dots ,f_{1,max}\right\},\\ \left\{f_{2,min},f_{2,a},\dots ,f_{2,max}\right\},\\ \dots ,\\ \left\{f_{\left|U\right|,min},f_{\left|U\right|,a},\dots ,f_{\left|U\right|,max}\right\}\end{array}\right\}.$$

The execution time of the task $n_i$ on the processor core $u_k$ with the frequency $f_{k,h}$ can be obtained by the following equation:

$$w_{i,k,h}=w_{i,k}\times \frac{f_{k,max}}{f_{k,h}}.$$

(3)

Then the energy consumption $E\left(n_i, u_k, f_{k,h}\right)$ of the task $n_i$ on the processor core $u_k$ with the frequency $f_{k,h}$ can be obtained by the following equation:

$$E\left(n_i, u_k, f_{k,h}\right)=\left(P_{k,ind}+C_{k,ef}\times {\left(f_{k,h}\right)}^{m_k}\right)\times w_{i,k}\times \frac{f_{k,max}}{f_{k,h}}.$$

(4)

Therefore, the energy consumption of application G will be

$$E\left(G\right)={{\displaystyle \sum }}_{i=1}^{\left|N\right|}E\left(n_i,u_{pr\left(i\right)},f_{pr\left(i\right),hz\left(i\right)}\right).$$

(5)

As a result of the $P_{ind}$, $E$ is not monotonic with $f$ and less $f$ does not always result less energy. Therefore, we can get the minimum value of energy-effective frequency by finding the minimum value of Equation (4). Similar to [20,21,22,23], we define the minimum value of energy-effective frequency as $f_{ee}$. After calculation, we can get that

$$f_{ee}=\sqrt[m]{\frac{P_{ind}}{\left(m-1\right)C_{ef}}}.$$

(6)

When the execution frequency is less than $f_{ee}$, it is meaningless to continue to reduce the frequency, because this will increase energy consumption. Therefore, the range of execution frequency variation is $\left[f_{low},f_{max}\right]$, where $f_{low}=\max \left(f_{min},f_{ee}\right)$. The new set of execution frequency becomes as follows:

$$\left\{\begin{array}{c}\left\{f_{1,low},f_{1,a},\dots ,f_{1,max}\right\},\\ \left\{f_{2,low},f_{2,a},\dots ,f_{2,max}\right\},\\ \dots ,\\ \left\{f_{\left|U\right|,mi},f_{\left|U\right|,a},\dots ,f_{\left|U\right|,max}\right\}\end{array}\right\}.$$

3.3. Preliminaries

3.3.1. Problem Description

The problem to be solved in this study is to assign a suitable frequency and processor core to each task, and minimize the schedule length of the application under the condition that the energy consumption of the application does not exceed the energy consumption constraint [35,36,37,38].

First, we define the earliest start time (EST) and the earliest finish time (EFT) of tasks. Given a task $n_i$ executed on processor $u_k$, its earliest start time (EST) is denoted as $EST\left(n_i,u_k\right)$, which is computed as

$$EST\left(n_i,u_k\right)=max\left(avail\left[k\right],\underset{n_j\in pred\left(n_i\right)}{\max }\left\{AFT\left(n_j\right)+c_{i,j}^{\prime }\right\}\right).$$

(7)

where $avail\left[k\right]$ is the earliest available time of processor core $u_k$, that is, all tasks executed on processor core $u_k$ have been completed, and processor core $u_k$ is ready to execute new tasks. $AFT\left(n_j\right)$ is the actual finish time of task $n_j$. $c_{i,j}^{\prime }$ is the actual communication time between task $n_i$ and $n_j$. If $n_i$ and $n_j$ are assigned to the same processor core, $c_{i,j}^{\prime }=0$; otherwise, $c_{i,j}^{\prime }=c_{i,j}$.

The earliest finish time (EFT) of task $n_i$ executed on processor $u_k$ with frequency $f_{k,h}$ is the earliest start time plus the execution time of task $n_i$, which is computed as

$$EFT(n_i,u_k,f_{k,h})=EST(n_i,u_k,f_{k,h})+w_{i,k}\times \frac{f_{k,\mathit{max}}}{f_{k,h}}.$$

(8)

We define$SL\left(G\right)$ as the schedule length of application, where

$$SL\left(G\right)=AFT\left(n_{exit}\right).$$

We define $E_{given}\left(G\right)$ as the given energy consumption constraint of application G. Therefore, the problem to solve can be expressed as minimizing $SL\left(G\right)$ while

$$E\left(G\right)={{\displaystyle \sum }}_{i=1}^{\left|N\right|}E\left(n_i,u_{pr\left(i\right)},f_{pr\left(i\right),hz\left(i\right)}\right)\le E_{\mathit{given}}\left(G\right).$$

(9)

where $u_{pr\left(i\right)}$ denotes the processor core assigned to the task $n_i$, and $f_{pr\left(i\right),hz\left(i\right)}$ denotes the execution frequency assigned to the task $n_i$.

3.3.2. Effective Range of Energy Consumption Constraint

Since the execution time of each task on each processor core is known, we can obtain the minimum and maximum energy consumption of $n_i$ represented by $E_{min}\left(n_i\right)$ and $E_{max}\left(n_i\right)$ respectively by traversing all processors. $E_{min}\left(n_i\right)$ and $E_{max}\left(n_i\right)$ perform task $n_i$ at minimum and maximum frequencies, respectively. The equations are as follows:

$$E_{min}\left(n_i\right)=\underset{u_k\in U}{\mathit{min}}E\left(n_i,u_k,f_{k,min}\right),$$

(10)

$$E_{max}\left(n_i\right)=\underset{u_k\in U}{\mathit{max}}E\left(n_i,u_k,f_{k,max}\right).$$

(11)

Therefore, the minimum and maximum energy consumption of application $G$ can be computed as follows:

$$E_{min}\left(G\right)={{\displaystyle \sum }}_{i=1}^{\left|N\right|}{E}_{min}\left(n_i\right),$$

(12)

$$E_{max}\left(G\right)={{\displaystyle \sum }}_{i=1}^{\left|N\right|}{E}_{max}\left(n_i\right).$$

(13)

It should be noted that the given energy consumption constraint has a reasonable range. If $E_{given}\left(G\right)<E_{min}\left(G\right)$, the energy consumption constraint can never be satisfied; if $E_{given}\left(G\right)>E_{max}\left(G\right)$, the energy constraint can always be met. Both of the above situations are unreasonable, so the reasonable range of the given energy consumption constraint is $E_{min}\left(G\right)\le E_{given}\left(G\right)\le E_{max}\left(G\right)$.

3.3.3. Task Priority Determination

Before scheduling, we need to determine the priority of tasks. Similar to [20,21,22,23], we use the upward rank value ($ran{k}_u$) as the criterion to determine the priority of tasks. $ran{k}_u$ is defined as follows:

$$ran{k}_u(n_i)=\frac{{{\displaystyle \sum }}_{k=1}^{\left|U\right|}{w}_{i,k}}{\left|U\right|}+\underset{n_i\in \mathit{succ}(n_i)}{\mathit{max}}\{c_{i,j}+ran{k}_u(n_j)\}.$$

(14)

The priority of tasks is sorted in descending order of $ran{k}_u$, that is, the higher $ran{k}_u$ of a task, the higher the priority of it.

Table 3. Upward rank values for tasks of the application in Figure 1.

Task	${\mathit{n}}_1$	${\mathit{n}}_2$	${\mathit{n}}_3$	${\mathit{n}}_4$	${\mathit{n}}_5$	${\mathit{n}}_6$	${\mathit{n}}_7$	${\mathit{n}}_8$	${\mathit{n}}_9$	${\mathit{n}}_{10}$
$ran{k}_u$	108	77	80	80	69	63.33	42.67	35.67	44.33	14.67

shows the upward rank values of all the tasks in Figure 1. Therefore, the task priority list of the application in Figure 1 will be $\left\{n_1,n_3,n_4,n_2,n_5,n_6,n_9,n_7,n_8,n_{10}\right\}$.

3.4. The ISAECC Method

In this subsection, we review the existing method that is closest to us and reveal its limitations.

3.4.1. Method Description

The ISAECC method is proposed in [23], and it consists of several major steps:

• It prioritizes tasks by using the upward rank value which is defined in Section 3.3.3;
• It uses a self-defined energy consumption weight value to give each unscheduled task a pre-allocated energy consumption;
• It calculates the energy consumption constraint of each task according to the given energy consumption constraint of the whole application and the pre-allocated energy consumption value of each task;
• According to the order of the task priority list, it assigns each task the processor core and frequency that can minimize its EST time by traversing each processor core and optional execution frequency.

For the sake of generality, we use $\left\{n_{o\left(1\right)},n_{o\left(2\right)},\dots ,n_{o\left(\left|N\right|\right)}\right\}$ to represent the task priority order. Assuming that the currently scheduled task is $n_{o\left(j\right)}$, then the scheduled task set is $\left\{n_{o\left(1\right)},n_{o\left(2\right)},\dots ,n_{o\left(j-1\right)}\right\}$ and the unscheduled task set is $\left\{n_{o\left(j+1\right)},n_{o\left(j+2\right)},\dots ,n_{o\left(\left|N\right|\right)}\right\}$. Therefore, when scheduling the task $n_{o\left(j\right)}$, the overall energy consumption of application G can be expressed as

$$E_{o\left(j\right)}\left(G\right)={{\displaystyle \sum }}_{x=1}^{j-1}E\left(n_{o\left(x\right)},u_{pr\left(o\left(x\right)\right)},f_{pr\left(o\left(x\right)\right),hz\left(o\left(x\right)\right)}\right)+E\left(n_{o\left(j\right)}\right)+{{\displaystyle \sum }}_{y=j+1}^{\left|N\right|}{E}_{pre}\left(n_{o\left(y\right)}\right),$$

(15)

where $E_{pre}\left(n_{o\left(y\right)}\right)$ denotes the pre-allocation energy consumption of task $n_{o\left(y\right)}$.

According to the energy consumption constraints shown in Equation (8), we can get

$$E_{o\left(j\right)}\left(G\right)\le E_{given}\left(G\right).$$

(16)

Therefore, we can get

$$E\left(n_{o\left(j\right)}\right)\le E_{given}\left(G\right)-{{\displaystyle \sum }}_{x=1}^{j-1}E\left(n_{o\left(x\right)},u_{pr\left(o\left(x\right)\right)},f_{pr\left(o\left(x\right)\right),hz\left(o\left(x\right)\right)}\right)-{{\displaystyle \sum }}_{y=j+1}^{\left|N\right|}{E}_{pre}\left(n_{o\left(y\right)}\right).$$

(17)

Let the energy consumption constraint of task $n_{o\left(j\right)}$ be

$$E_{given}\left(n_{o\left(j\right)}\right)=E_{given}\left(G\right) -{{\displaystyle \sum }}_{x=1}^{j-1}E\left(n_{o\left(x\right)},u_{pr\left(o\left(x\right)\right)},f_{pr\left(o\left(x\right)\right),hz\left(o\left(x\right)\right)}\right)-{{\displaystyle \sum }}_{y=j+1}^{\left|N\right|}{E}_{pre}\left(n_{o\left(y\right)}\right).$$

(18)

With Equation (17), we only need to consider the energy consumption constraint of each task which is shown as follows:

$$E\left(n_{o\left(j\right)}\right)\le min\left\{E_{given}\left(n_{o\left(j\right)}\right),E_{max}\left(n_{o\left(j\right)}\right)\right\}.$$

(19)

Therefore, the key problem is how to determine the pre-allocation energy consumption ($E_{pre}\left(n_{o\left(j\right)}\right)$) of each task. The central idea of the method ISAECC used is to pre-allocate the energy consumption for unscheduled tasks by a weight mechanism. First, they define the improvable energy of application G called $E_{ie}\left(G\right)$, which is computed as

$$E_{ie}\left(G\right)=E_{given}\left(G\right)-E_{min}\left(G\right).$$

(20)

Then, they define $E_{ave}\left(n_i\right)$ and $E_{ave}\left(G\right)$ as the energy consumption level of task $n_i$ and the energy consumption level of application G. $E_{ave}\left(n_i\right)$ and $E_{ave}\left(G\right)$ are computed as

$$E_{ave}\left(n_i\right)=\frac{E_{min}\left(n_i\right)+E_{max}\left(n_i\right)}{2},$$

(21)

$$E_{ave}\left(G\right)=\frac{E_{min}\left(G\right)+E_{max}\left(G\right)}{2}.$$

(22)

Next, they define $el\left(n_i\right)$ as the weight of energy consumption level of task $n_i$, which is computed as

$$el\left(n_i\right)=\frac{E_{ave}\left(n_i\right)}{E_{ave}\left(G\right)}.$$

After that, they calculated the pre-allocated energy consumption for task $n_i$ as follows:

$$E_{pre}\left(n_i\right)=min\left\{E_{wa}\left(n_i\right),E_{max}\left(n_i\right)\right\}$$

where $E_{wa}\left(n_i\right)$ is computed as

$$E_{wa}\left(n_i\right)=E_{ie}\left(G\right)\times el\left(n_i\right)+E_{min}\left(n_i\right).$$

After determining the pre-allocated energy consumption of each task, the task scheduling can be completed according to steps 3 and 4, described at the beginning of this section (Section 3.4.1).

3.4.2. Limitations of ISAECC

ISAECC solves the problem of increasing schedule length caused by unfair energy constraint allocation of low priority tasks by MSLECC in [20]. However, we find that it is not the best practice to treat each task fairly, because tasks in different levels have different impacts on the whole application. For example, in Figure 1, the task $n_1$ and task $n_{10}$ should be assigned more energy than other tasks, because their execution time can affect the overall schedule length of the application G directly. It is obvious that if the execution time of task $n_1$ or task $n_{10}$ is shortened or lengthened under other same conditions, the overall schedule length will shorten or lengthen the same amount accordingly. Other tasks like $n_2$ cannot be compared to $n_1$ and $n_{10}$. If the execution time of task $n_2$ is shortened or lengthened under other same conditions, the schedule length of application G may not change obviously, or even not change at all. Therefore, we should consider the levels of the tasks and the number of tasks in each level when pre-allocating the energy consumption of tasks. In addition, previous studies ignored the negative impact of the local optimal characteristics of scheduling algorithm on the schedule length. In our design, these problems have been greatly improved.

4. Our Solution

In this section, we introduce our new strategy in detail. First, we introduce a new task energy pre-allocation method considering task level (Section 4.1). Then, we give a new task scheduling algorithm to minimize the schedule length under the constraint of energy consumption (Section 4.2). Finally, we describe the task execution frequency re-adjustment mechanism we added after getting the preliminary scheduling results (Section 4.3).

4.1. The New Task Energy Pre-Allocation Method

The central idea of our pre-allocation method is to consider the levels of tasks. The impact of tasks in different hierarchies on the overall schedule length of an application is different. For example, in Figure 1, if we shorten the execution time of task $n_1$ by increasing its execution frequency, the schedule length of application G will certainly shorten the corresponding time; but if we shorten the execution time of task $n_2$, the schedule length of application G is unlikely to change much, because there are still many tasks in a similar position to it. Therefore, a more reasonable approach is to appropriately pre-allocate more energy consumption to task $n_1$ in the case of Figure 1. Based on this idea, we put forward a new energy pre-allocation method, based on the weight value of energy consumption and the levels of tasks.

4.1.1. Method Description

We define the level of tasks as follows:

$$\{\begin{array}{l}L\left(n_{entry}\right)=0\\ L\left(n_i\right)=\underset{n_x\in pred\left(n_i\right)}{\mathit{max}}\left\{L\left(n_x\right)+1\right\}\end{array}.$$

We define $N_l=\left\{n_i,n_j,\dots \right\}$ as the set of tasks contained in level l, where $L\left(n_i\right)=L\left(n_j\right)=L(\dots )=l$. The number of tasks in level l is $\left|N_l\right|$. We can have that the maximum of the level of tasks is $L\left(n_{exit}\right)$.

We define the improvable energy of application G ($E_{ie}\left(G\right)$) and the energy consumption level of task $n_i$ ($E_{ave}\left(n_i\right)$) the same as ISAECC. We define that the energy consumption level of the task level l is the sum of energy consumption level in level l, which is computed as

$$E_{ave}\left(N_l\right)={{\displaystyle \sum }}_{n_i\in N_l}{E}_{ave}\left(n_i\right).$$

(23)

We define $el\left(n_{i,l}\right)$ as the energy consumption weight of task $n_i$ in its level l, which is computed as

$$el\left(n_{i,l}\right)=\frac{E_{ave}\left(n_i\right)}{E_{ave}\left(N_l\right)}.$$

(24)

We define $E_{var}\left(N_l\right)$ as the variation energy consumption of $N_l$, which is computed as

$$E_{var}\left(N_l\right)=\frac{E_{ave}\left(N_l\right)}{K}, K=\{\begin{array}{c}\left|N_l\right|, \left|N_l\right|<\left|U\right|\\ \left|U\right|, \left|N_l\right|\ge \left|U\right|\end{array}.$$

(25)

Correspondingly, the variation energy consumption of application G is computed as

$$E_{var}\left(G\right)={\displaystyle \sum }_l{E}_{var}\left(N_l\right).$$

(26)

The energy consumption weight of $N_l$ in application G can be defined as

$$el\left(N_l\right)=\frac{E_{var}\left(N_l\right)}{E_{var}\left(G\right)}.$$

(27)

We define $E_{ie}\left(N_l\right)$ as the improvable energy of $N_l$, which is computed as

$$E_{ie}\left(N_l\right)=E_{ie}\left(G\right)\times el\left(N_l\right).$$

(28)

Therefore, we can get the new energy pre-allocation formula of $n_i$ as follows:

$$E_{pre}\left(n_i\right)=min\left\{E_{wa}\left(n_i\right),E_{max}\left(n_i\right)\right\},$$

(29)

where

$$E_{wa}\left(n_i\right)=E_{ie}\left(N_l\right)\times el\left(n_{i,l}\right)+E_{min}\left(n_i\right).$$

(30)

4.1.2. Feasibility of the Task Energy Pre-Allocation Mechanism

In order to prove the feasibility of our method, we need to prove the following theorem: Given an application G, if the unscheduled tasks are pre-allocated energy consumption according to our method, then each task $n_j$ can satisfy Equation (15).

We use mathematical induction to prove the above theorem. First, we need to prove task $n_{o\left(1\right)}$ can satisfy Equation (15), and the other tasks are all unscheduled. By Equations (14), (22)–(29), we can have

$$\begin{array}{cc}\hfill E_{o\left(1\right)}\left(G\right)& =E\left(n_{o\left(1\right)}\right)+{{\displaystyle \sum }}_{y=2}^{\left|N\right|}{E}_{pre}\left(n_{o\left(y\right)}\right)\hfill \\ & \le E\left(n_{o\left(1\right)}\right)+{{\displaystyle \sum }}_{y=2}^{\left|N\right|}{E}_{wa}\left(n_{o\left(y\right)}\right)\hfill \\ & = E\left(n_{o\left(1\right)}\right)+{{\displaystyle \sum }}_{y=1}^{\left|N\right|}{E}_{wa}\left(n_{o\left(y\right)}\right)-E_{wa}\left(n_{o\left(1\right)}\right)\hfill \\ & = E\left(n_{o\left(1\right)}\right)+E_{given}\left(G\right)-E_{wa}\left(n_{o\left(1\right)}\right).\hfill \end{array}$$

We know that

$$E_{wa}\left(n_{o\left(1\right)}\right)=E_{ie}\left(N_l\right)\times el\left(n_{i,l}\right)+E_{min}\left(n_i\right)\ge E_{min}\left(n_{o\left(1\right)}\right).$$

(31)

We can at least find a situation in which

$$E\left(n_{o\left(1\right)}\right)=E_{min}\left(n_{o\left(1\right)}\right).$$

In other words, at least when $E\left(n_{o\left(1\right)}\right)=E_{min}\left(n_{o\left(1\right)}\right)$, we can have

$$E_{o\left(1\right)}\left(G\right)\le E\left(n_{o\left(1\right)}\right)+E_{given}\left(G\right)-E_{wa}\left(n_{o\left(1\right)}\right)\le E_{given}\left(G\right).$$

From the above derivation, we prove that task $n_{o\left(1\right)}$ can satisfy Equation (15).

Then, we assume that task $n_{o\left(j\right)}$ can satisfy Equation (15). That is

$$\begin{array}{cc}\hfill E_{o\left(j\right)}\left(G\right)& ={{\displaystyle \sum }}_{x=1}^{j-1}E\left(n_{o\left(x\right)},u_{pr\left(o\left(x\right)\right)},f_{pr\left(o\left(x\right)\right),hz\left(o\left(x\right)\right)}\right)+E\left(n_{o\left(j\right)}\right)+{{\displaystyle \sum }}_{y=j+1}^{\left|N\right|}{E}_{pre}\left(n_{o\left(y\right)}\right)\hfill \\ & ={{\displaystyle \sum }}_{x=1}^{j}E\left(n_{o\left(x\right)},u_{pr\left(o\left(x\right)\right)},f_{pr\left(o\left(x\right)\right),hz\left(o\left(x\right)\right)}\right)+{{\displaystyle \sum }}_{y=j+1}^{\left|N\right|}{E}_{pre}\left(n_{o\left(y\right)}\right)\hfill \\ & \le E_{given}\left(G\right).\hfill \end{array}$$

The above formulation can be written as

$${{\displaystyle \sum }}_{x=1}^{j}E\left(n_{o\left(x\right)},u_{pr\left(o\left(x\right)\right)},f_{pr\left(o\left(x\right)\right),hz\left(o\left(x\right)\right)}\right)\le E_{given}\left(G\right)-{{\displaystyle \sum }}_{y=j+1}^{\left|N\right|}{E}_{pre}\left(n_{o\left(y\right)}\right).$$

(32)

Next, we prove task $n_{o\left(j+1\right)}$ can satisfy Equation (15). By Equation (30), we can have

$$\begin{array}{cc}\hfill E_{o\left(j+1\right)}\left(G\right)& ={{\displaystyle \sum }}_{x=1}^{j}E\left(n_{o\left(x\right)},u_{pr\left(o\left(x\right)\right)},f_{pr\left(o\left(x\right)\right),hz\left(o\left(x\right)\right)}\right)+E\left(n_{o\left(j+1\right)}\right)+{{\displaystyle \sum }}_{y=j+2}^{\left|N\right|}{E}_{pre}\left(n_{o\left(y\right)}\right)\hfill \\ & \le E_{given}\left(G\right)-{{\displaystyle \sum }}_{y=j+1}^{\left|N\right|}{E}_{pre}\left(n_{o\left(y\right)}\right)+E\left(n_{o\left(j+1\right)}\right)+{{\displaystyle \sum }}_{y=j+2}^{\left|N\right|}{E}_{pre}\left(n_{o\left(y\right)}\right)\hfill \\ & =E_{given}\left(G\right)+E\left(n_{o\left(j+1\right)}\right)-E_{pre}\left(n_{o\left(j+1\right)}\right).\hfill \end{array}$$

From Equations (28) and (30), we can have

$$E_{pre}\left(n_{o\left(j+1\right)}\right)=min\left\{E_{wa}\left(n_{o\left(j+1\right)}\right),E_{max}\left(n_{o\left(j+1\right)}\right)\right\}\ge E_{min}\left(n_{o\left(j+1\right)}\right).$$

Therefore, at least when $E\left(n_{o\left(j+1\right)}\right)=E_{min}\left(n_{o\left(j+1\right)}\right)$, we can have

$$E_{o\left(j+1\right)}\left(G\right)\le E_{given}\left(G\right)+E\left(n_{o\left(j+1\right)}\right)-E_{pre}\left(n_{o\left(j+1\right)}\right)\le E_{given}\left(G\right).$$

In summary, given an application G, if the unscheduled tasks are pre-allocated energy consumption according to our method, then each task $n_j$ can satisfy Equation (15). The feasibility of our method has been proved.

4.2. The Proposed Algorithm for Minimizing Schedule Length

In this section, we show our new task scheduling algorithm in Algorithm 1. In the algorithm, Line 1 is to prioritize tasks in the input application; Lines 2–10 calculate some required values for each task, each level and the application G; Lines 11 and 12 calculate the pre-allocation energy consumption of each task; Lines 13–26 are to select processor and frequency for each task; Lines 27 and 28 are to calculate the actual energy consumption E(G) and the final schedule length SL(G).

Algorithm 1. A new scheduling algorithm for minimizing schedule length.

Input: G = {N, W, C}, U, $E_{given}\left(G\right)$ Output: SL(G), E(G)  1:  Sort tasks in a list $tl$ by descending order of $ran{k}_u$;  2:  for ($\forall i, n_i\in N)$ do  3:    Compute $E_{min}\left(n_i\right)$ and $E_{max}\left(n_i\right)$; //By (10) (11)  4:    Compute $E_{ave}\left(n_i\right)$; //By (21)  5:  for ($\forall l, 1\le l\le L\left(n_{exit}\right))$ do  6:    Compute $E_{ave}\left(N_l\right)$; //By (23)  7:    Compute $E_{var}\left(N_l\right)$; //By (25)  8:  Compute $E_{min}\left(G\right)$ and $E_{max}\left(G\right)$; //By (12) (13)  9:  Compute $E_{ave}\left(G\right)$; //By (22)  10:  Compute $E_{var}\left(G\right)$; //By (26)  11:  for ($\forall i, n_i\in N)$ do  12:    Compute $E_{pre}\left(n_i\right)$; //By (29)  13:  while $\left(tl\ne \varnothing \right)$ do  14:    $n_i=tl$.out();  15:    $AFT\left(n_i\right)=\infty$;  16:    Compute $E_{given}\left(n_i\right)$; //By (18)  17:    for ($\forall k, u_k\in U)$ do  18:      for ($\forall f_{k,h}, f_{k,h}\in \left[f_{k,low},f_{k,max}\right])$ do  19:        Compute $E\left(n_i, u_k, f_{k,h}\right)$; //By (4)  20:        if ($E\left(n_i, u_k, f_{k,h}\right)> E_{given}\left(n_i\right)$) then  21:          continue;  22:        Compute $EFT\left(n_i, u_k, f_{k,h}\right)$; //By (8)  23:        if ($EFT\left(n_i, u_k, f_{k,h}\right)< AFT\left(n_i\right)$) then  24:          Let $u_{pr\left(i\right)}=u_k$ and $f_{pr\left(i\right),hz\left(i\right)}=f_{k,h}$;  25:          $E\left(n_i, u_{pr\left(i\right)}, f_{pr\left(i\right),hz\left(i\right)}\right)= E\left(n_i, u_k, f_{k,h}\right)$;  26:          $AFT\left(n_i\right)= EFT\left(n_i, u_{pr\left(i\right)}, f_{pr\left(i\right),hz\left(i\right)}\right)$;  27:  Compute actual energy consumption $E\left(G\right)$;  28:  Compute the schedule length $SL\left(G\right)$;  29:  return SL(G), E(G).

For each task, selecting the processor with the minimum EFT has complexity $O\left(\left|N\right|\times \left|U\right|\times \left|F\right|\right)$, where $\left|F\right|$ represents the maximum number of discrete frequencies from $f_{k,low}$ to $f_{k,max}$. Therefore, the complexity of Algorithm 1 is $O\left({\left|N\right|}^2\times \left|U\right|\times \left|F\right|\right)$ the same as ISAECC in [23].

4.3. The Task Execution Frequency Re-Adjustment Mechanism

Through the new task energy pre-allocation method in Section 4.1 and the new task scheduling algorithm in Section 4.2, we can get the preliminary scheduling results. Other methods generally end here such as those in [20,21,22,23], but they do not realize that the scheduling results can be optimized to further shorten the schedule length. The same as the scheduling algorithms in [20,21,22,23], the algorithm in Section 4.2 makes tasks finish as soon as possible when scheduling them, which is not entirely reasonable. Premature completion of some tasks cannot shorten the overall schedule length, but will take up more energy consumption. Therefore, we introduce the concept of the latest finish time of tasks [28,39,40], which is defined as follows:

$$\{\begin{array}{l}LFT\left(n_{exit}\right)=AFT\left(n_{exit}\right)\\ LFT\left(n_i\right)=min\left\{\underset{n_j\in succ\left(n_i\right)}{\mathit{min}}\left\{AST\left(n_j\right)-c_{i,j}^{\prime }\right\},AST\left(n_{dn\left(i\right)}\right)\right\}\end{array},$$

(33)

where $n_{dn\left(i\right)}$ represents the downward neighbor task of $n_i$, that is, $n_{dn\left(i\right)}$ is on the same processer core as $n_i$ and it is the first case after $n_i.$

Through Equation (32), we can replace AFT of tasks with LFT to delay the finish time of some tasks without increasing the schedule length. As the execution time of tasks is prolonged, their running frequency will be reduced accordingly, which can save some energy consumption. Therefore, the new execution time of task $n_i$ is changed from $AFT\left(n_i\right)-AST\left(n_i\right)$ to $LFT\left(n_i\right)-AST\left(n_i\right)$, and the new frequency of task $n_i$ can be changed as follows:

$$f_{pr\left(i\right),nhz\left(i\right)}=f_{pr\left(i\right),hz\left(i\right)}\times \frac{AFT\left(n_i\right)-AST\left(n_i\right)}{LFT\left(n_i\right)-AST\left(n_i\right)}.$$

The frequency range of task $n_i$ is $\left[f_{pr\left(i\right),low},f_{pr\left(i\right),max}\right]$, so $f_{pr\left(i\right),nhz\left(i\right)}$ should be

$$f_{pr\left(i\right),nhz\left(i\right)}=max\left\{f_{pr\left(i\right),hz\left(i\right)}\times \frac{AFT\left(n_i\right)-AST\left(n_i\right)}{LFT\left(n_i\right)-AST\left(n_i\right)},f_{pr\left(i\right),low}\right\}.$$

(34)

Therefore, the actual execution time (AET) of task $n_i$ will be

$$AET\left(n_i\right)=w_{i,pr\left(i\right)}\times \frac{f_{pr\left(i\right),max}}{f_{pr\left(i\right),nhz\left(i\right)}}.$$

(35)

Finally, the new $AST\left(n_i\right)$ should be updated as

$$ST\left(n_i\right)=LFT\left(n_i\right)-AET\left(n_i\right)=LFT\left(n_i\right)-w_{i,pr\left(i\right)}\times \frac{f_{pr\left(i\right),max}}{f_{pr\left(i\right),nhz\left(i\right)}}.$$

(36)

On the basis of above equations, the algorithm to save energy after preliminary scheduling is shown in Algorithm 2. In Line 2, we reorder the tasks in descending order of the actual finish time (AFT) of tasks according to the scheduling results of Algorithm 1. In Lines 4–10, $AFT\left(n_i\right)$, $AST\left(n_i\right)$ and $f_{pr\left(i\right),hz\left(i\right)}$ are updated. In Line 11, we compute the new $E\left(n_i\right)$ using the above updated values. In Lines 12 and 13, we compute the saved energy $E_{save}\left(G\right)$.

Algorithm 2. The method to save energy.

Input: G = {N, W, C}, U, $E_{given}\left(G\right)$ Output: $E_{save}\left(G\right)$  1:  Call Algorithm 1 to get the preliminary scheduling results;  2:  Sort the tasks in the list $al$ according to the descending $AFT\left(n_i\right)$ order of values;  3:  while $\left(al\ne \varnothing \right)$ do  4:    $n_i=tl$.out();  5:    Compute $LFT\left(n_i\right)$; //By (33)  6:    Compute the new frequency $f_{pr\left(i\right),nhz\left(i\right)}$; //By (34)  7:    Compute the new $AET\left(n_i\right)$; //By (35)  8:    Update $AFT\left(n_i\right)\leftarrow LFT\left(n_i\right)$;  9:    Update $AST\left(n_i\right)\leftarrow LFT\left(n_i\right)-AET\left(n_i\right)$; //By (36)  10:    Update $f_{pr\left(i\right),hz\left(i\right)}\leftarrow f_{pr\left(i\right),nhz\left(i\right)}$;  11:    Compute the new $E\left(n_i\right)$; //By (4)  12:  Compute the new energy consumption of G $E_{new}\left(G\right)$; //By (5)  13:  Compute the saved energy $E_{save}\left(G\right)=E\left(G\right)-E_{new}\left(G\right)$;  14:  return $E_{save}\left(G\right)$.

Generally speaking, after the task scheduling is completed, there will be remaining energy consumption that is not used up as follows:

$$E_{rem}\left(G\right)=E_{given}\left(G\right)-E\left(G\right).$$

(37)

Therefore, the energy consumption that can be reused is

$$E_{reu}\left(G\right)=E_{rem}\left(G\right)+E_{save}\left(G\right).$$

(38)

After calculating the energy consumption that can be reused, we need to re-allocate the energy consumption to the tasks that can directly affect the overall schedule length. Directly affecting the overall schedule length means that according to how much the task running time changes, the overall schedule length will also change. For example, task $n_1$ in Figure 1, the total schedule length will be shortened or extended as much as its execution time is shortened or extended. However, due to the diversity of task models, it is very difficult to find out which tasks can directly affect the overall schedule length. Therefore, we adopt a more direct way: If the application model level is strict (the tasks in level l only communicate with the tasks in their adjacent levels which are level l+1 and level l−1), we re-allocate the reused energy consumption ($E_{reu}\left(G\right)$) to the tasks that have not reached the highest execution frequency in the levels whose $\left|N\right|=1$; otherwise, we only re-allocate the reused energy consumption to $n_{entry}$ and $n_{exit}$. For ease of description, we define $N_{das}$ as the set of tasks that can directly affect the overall schedule length.

After the assignment object of $E_{reu}\left(G\right)$ is determined, we need to determine the allocation proportion. We define the maximum energy consumption of $n_i\in N_{das}$ on processer core $u_{pr\left(i\right)}$ is

$$E_{max}\left(n_i,u_{pr\left(i\right)}\right)=\left(P_{pr\left(i\right),ind}+C_{pr\left(i\right),ef}\times {\left(f_{pr\left(i\right),max}\right)}^{m_{pr\left(i\right)}}\right)\times w_{i,pr\left(i\right)}.$$

(39)

Therefore, the growable energy consumption of $n_i\in N_{das}$ on processer core $u_{pr\left(i\right)}$ will be

$$E_{gro}\left(n_i\right)=E_{max}\left(n_i,pr\left(i\right)\right)-E\left(n_i\right).$$

(40)

We take the values of $E_{gro}\left(n_i\right)$ as the allocation proportion of $E_{reu}\left(G\right)$. Therefore, the reused energy consumption of $n_i\in N_{das}$ will be

$$E_{reu}\left(n_i\right)=E_{reu}\left(G\right)\times \frac{E_{gro}\left(n_i\right)}{{{\displaystyle \sum }}_{i=1}^{\left|N\right|}{E}_{gro}\left(n_i\right)}.$$

(41)

Adding $E_{reu}\left(n_i\right)$ and $E\left(n_i\right)$ which is computed in Algorithm 2, we can get the energy consumption that $n_i\in N_{das}$ can use will be

$$E_{use}\left(n_i\right)=E_{reu}\left(n_i\right)+E\left(n_i\right).$$

(42)

According to $E_{use}\left(n_i\right)$, we can find the new frequency $f_{pr\left(i\right),nh}$ of $n_i\in N_{das}$ by traversing the execution frequency on processor core $u_{pr\left(i\right)}$. Then we can compute the shortened actual execution time of $n_i\in N_{das}$ as follows:

$$AE{T}_{shor}\left(n_i\right)=w_{i,pr\left(i\right)}\times f_{pr\left(i\right),max}\times \left(\frac{1}{f_{pr\left(i\right),h}}-\frac{1}{f_{pr\left(i\right),nh}}\right).$$

(43)

Therefore, the new length of the application G will be

$$S{L}_{new}\left(G\right)=SL\left(G\right)-{{\displaystyle \sum }}_{n_i\in N_{das}}AE{T}_{shor}\left(n_i\right).$$

(44)

Combined with the Algorithms 1 and 2, the task execution frequency re-adjustment mechanism is shown in Algorithm 3. Lines 1 and 2 call Algorithms 1 and 2 to get the preliminary scheduling results and $E_{save}\left(G\right)$. Line 3 compute the reused energy consumption $E_{reu}\left(G\right)$. Lines 6 and 7 calculate some required values for each task belonging to $N_{das}$. Lines 8–14 are to select the new frequency for each task belonging to $N_{das}$. Finally, we compute the new schedule length of the application G $S{L}_{new}\left(G\right)$ after the task execution frequency re-adjustment mechanism in Line 15. The value of $S{L}_{new}\left(G\right)$ is the final minimum schedule length we get.

In general, our method includes Algorithms 1–3. Algorithm 1 is a task energy pre-allocation strategy, and Algorithms 2 and 3 constitute the task execution frequency re-adjustment mechanism. For a given application, we operate in the order of Algorithms 1, Algorithm 2 and Algorithm 3, then we can get the scheduling result of minimizing the schedule length of the application.

Algorithm 3. The task execution frequency re-adjustment mechanism.

Input: G = {N, W, C}, U, $E_{given}\left(G\right)$ Output: $SL\left(G\right)$  1:  Call Algorithm 1 to get the preliminary scheduling results;  2:  Call Algorithm 2 to get $E_{save}\left(G\right)$;  3:  Compute $E_{reu}\left(G\right)$; //By (38)  4:  for ($\forall n_i$,$n_i\in N_{das}$) do  5:    $E\left(n_i\right)=0$;  6:    Compute $E_{gro}\left(n_i\right)$; //By (40)  7:    Compute $E_{use}\left(n_i\right)$; //By (42)  8:    for ($\forall f_{pr\left(i\right),h}, f_{pr\left(i\right),h}\in \left[f_{pr\left(i\right),low},f_{pr\left(i\right),max}\right])$ do  9:      Compute $E\left(n_i, u_{pr\left(i\right)}, f_{pr\left(i\right),h}\right)$; //By (4)  10:      if ($E\left(n_i, u_{pr\left(i\right)}, f_{pr\left(i\right),h}\right)> E_{use}\left(n_i\right)$) then  11:        continue;  12:      if ($E\left(n_i, u_{pr\left(i\right)}, f_{pr\left(i\right),h}\right)>E\left(n_i\right))$ then  13:        Let $E\left(n_i\right)= E\left(n_i, u_{pr\left(i\right)}, f_{pr\left(i\right),h}\right)$;  14:        Let $f_{pr\left(i\right),nh}= f_{pr\left(i\right),h}$;  15:    Compute $S{L}_{new}\left(G\right)$; //By (44)  16:    Update $SL\left(G\right)\leftarrow S{L}_{new}\left(G\right)$;  17:  return $SL\left(G\right)$.

5. Experiments

In this section, we use four algorithms, MSLECC [20], WALECC [21], EECC [22] and ISAECC [23], which are the same as the goal of this article to compare with our proposed method. The configuration of the experimental platform is AMD Ryzen 5 2500U CPU @ 2.00 GHz, 8 GB RAM (Santa Clara, CA, USA), 64-bit Windows 10 Home Edition. The whole set of codes is mainly implemented by C and scripts. The final schedule length SL(G) is the only evaluation standard of these algorithms.

The parameters of the processors and applications as follows: $10 \mathrm{ms}\le w_{i,k}\le 100 \mathrm{ms}$, $10 \mathrm{ms}\le c_{i,j}\le 100 \mathrm{ms}$, $0.03\le P_{k,ind}\le 0.07$, $0.8\le C_{k,ef}\le 1.2$, $2.5\le m_k\le 3.0$, and $f_{k,max}=1$ GHz. The execution frequency is discrete, and the precision is 0.01 GHz. The simulated heterogeneous platform for testing the problem of minimizing the schedule length uses four processor cores.

We chose two DAG models to evaluate our algorithm, which are two real-world applications (Fast Fourier transform and Gaussian elimination).

5.1. Fast Fourier Transform Application

We first consider the fast Fourier transform (FFT), Figure 2a shows an example of the FFT parallel application with $\rho =4$. The parameter $\rho$ can represent the size of application models. For FFT application, the number of tasks is $\left|N\right|=\left(2\times \rho -1\right)+\rho \times {\log }_2\rho$, and $\rho =2^x$ where $x$ is an integer. We can see that there are four exit tasks (task 12, 13, 14, 15) in the FFT graph in Figure 2a, and there will be $\rho$ exit tasks in the FFT graph with parameter $\rho$. In order to match the application model in Section 3.1, we add a dummy exit task whose execution time is 0. We also connect the dummy exit task to the last $\rho$ exit tasks, and we set their communication time is 0. For example, Figure 2b shows the changed FFT parallel application with $\rho =4$ which is added a dummy exit task.

Experiment 1. In order to observe the performance on different energy consumption constraints, this experiment is carried out to compare the final schedule length values of the FFT application for varying energy consumption constraints. We use the FFT application with $\rho =32$, that is, the number of tasks is 233 ($\left|N\right|=233$). We set the energy consumption constraints $E_{given}\left(G\right)=E_{min}\left(G\right)+\frac{M}{233}\left(E_{max}\left(G\right)-E_{min}\left(G\right)\right)$, where $1\le M\le 232$ and $M$ is an integer.

Table 4. Results of FFT applications with $\rho =32$ for varying $E_{given}\left(G\right)$.

${\mathit{E}}_{\mathit{g}\mathit{i}\mathit{v}\mathit{e}\mathit{n}}\left(\mathit{G}\right)$	$\mathit{S}\mathit{L}\left(\mathit{G}\right)$
	MSLECC	WALECC	EECC	ISAECC	Our method
2086	10,989	8023	7955	7814	6846
2714	5915	4491	4564	4417	3887
3342	4644	3302	3487	3390	3108
4048	4052	2918	2937	2966	2779
5226	3381	2546	2609	2484	2388
6168	2947	2253	2315	2273	2190

shows the details of the final schedule lengths of FFT application with $\rho =32$ for varying $E_{given}\left(G\right)$ by using all the algorithms, and a more intuitive feeling can be performed through Figure 3. It can be seen that our algorithm has the obvious advantage on the schedule length $SL\left(G\right)$ compared to other algorithms. From the experimental results, we can get that our method outperforms MSLECC by about 28.13%~36.35%, and it outperforms the newest method ISAECC by about 3.65%~10.48%. The results of WALECC and EECC are similar to that of ISAECC.

Further, we can find that the gaps between our method and other algorithms increase when $E_{given}\left(G\right)$ decreases. This is because all tasks can be assigned to a relatively large energy consumption constraint when the energy consumption constraint is large, so that the impact of task level is smaller. Moreover, at this time, most tasks belonging to $N_{das}$ have reached the maximum frequency, and cannot continue to increase the frequency to shorten the schedule length. In addition, as we expected, the larger $E_{given}\left(G\right)$ is, the shorter schedule length we can obtain.

Experiment 2. In order to observe the algorithm performance under different number of tasks, an experiment is carried out to compare the final schedule length values of the FFT application for varying number of tasks. The parameter $\rho$ is changed from 8 to 256. In order to get relatively obvious results, we set the energy consumption constraints to a relatively small value: $E_{given}\left(G\right)=E_{min}\left(G\right)+\frac{E_{max}\left(G\right)-E_{min}\left(G\right)}{8}$.

Table 5. Results of FFT applications for varying number of tasks.

$\mathit{\rho }$	$\mathit{S}\mathit{L}\left(\mathit{G}\right)$
	MSLECC	WALECC	EECC	ISAECC	Our Method
8	805	654	679	666	634
16	1619	1331	1358	1294	1227
32	3664	2960	2777	2814	2641
64	8433	6340	6314	6326	5922
128	18,954	13,892	13,975	14,131	13,168
256	43,524	31,982	32,814	32,354	29,987

shows the results of FFT applications for different number of tasks by using all the algorithms, and a more intuitive feeling can be performed through Figure 4. The results show that our method has better performance than other algorithms. Our method outperforms MSLECC by about 21.24%–31.10%, and outperforms ISAECC by about 4.80%–7.32%. From the results, we can find that the more tasks we have, the better our method performs. This is because that there will be more different hierarchies in FFT graphs as the number of tasks increases, so that our new energy pre-allocation strategy will become more advantageous.

5.2. Gaussian Elimination Application

Similarly, in Gaussian elimination (GE) application, we define $\rho$ as the size of the application, and the number of tasks can be calculated by $\left|N\right|=\frac{{\rho }^2+\rho -2}{2}$. Figure 5 shows the GE application model with $\rho =5$.

Experiment 3. This experiment compares the final schedule length values of GE application for varying energy consumption constraints. We use the GE application with $\rho =21$, that is, the number of tasks is 230 ($\left|N\right|=230$). We set the energy consumption constraints $E_{given}\left(G\right)=E_{min}\left(G\right)+\frac{M}{230}\left(E_{max}\left(G\right)-E_{min}\left(G\right)\right)$, where $1\le M\le 229$ and $M$ is an integer.

Table 6. Results of GE applications with $\rho =21$ for varying $E_{given}\left(G\right)$.

${\mathit{E}}_{\mathit{g}\mathit{i}\mathit{v}\mathit{e}\mathit{n}}\left(\mathit{G}\right)$	$\mathit{S}\mathit{L}\left(\mathit{G}\right)$
	MSLECC	WALECC	EECC	ISAECC	Our Method
2436	6826	6144	5912	5973	5692
2763	6155	5389	5603	5398	5128
3665	5519	4457	4521	4324	4119
6535	4721	3776	3502	3418	3090
7191	3864	3093	3214	3186	3043
8749	3397	3012	3129	3017	2898

shows the results of the final schedule lengths of GE application with $\rho =21$ for varying $E_{given}\left(G\right)$ by using all the algorithms, and a more intuitive feeling can be performed through Figure 6. We can see that our method still performs better than other algorithms, specifically, it outperforms MSLECC by about 14.69%–34.55%, and outperforms ISAECC by about 3.94%–9.60%, but the improvement is not obvious compared with FFT models. This is because that the level of GE application is not as strict as the FFT application, so that the effect of our method has diminished.

Experiment 4. This experiment compares the final schedule length values of GE application for varying number of tasks. We set that $E_{given}\left(G\right)=E_{min}\left(G\right)+\frac{E_{max}\left(G\right)-E_{min}\left(G\right)}{4}$ and change the number of tasks.

Table 7. Results of GE applications for varying number of tasks.

$\mathit{\rho }$	$\mathit{S}\mathit{L}\left(\mathit{G}\right)$
	MSLECC	WALECC	EECC	ISAECC	Our Method
8	1056	897	911	893	847
13	2185	1794	1805	1737	1599
21	3698	3269	3082	3214	3106
41	18,845	14,504	14,729	14,563	13,534
83	39,584	31,375	31,551	31,422	30,032
100	59,951	46,210	48,238	47,916	42,657

shows the results of the final schedule lengths of GE application for varying number of tasks by using all the algorithms, and a more intuitive feeling can be performed through Figure 7. Our method still has a better effect on the schedule length than other algorithms which performs better 16.00%–28.85% than MSLECC and 3.36%–10.98% than ISAECC.

5.3. Analysis and Summary of Experimental Results

We can obtain that our method has better performance compared with the other algorithms from the experimental results. However, when $E_{given}\left(G\right)$ becomes larger or the number of tasks is relatively small, the advantage of our method is not particularly obvious. The former is because that all tasks can be allocated to a relatively large energy consumption constraint when $E_{given}\left(G\right)$ is large, so that the impact of task level will be smaller. The latter is because that the experimental results are more accidental and cannot reflect the general law when the number of tasks is small. We can also find that our method performs better on FFT models than on GE models with similar $E_{given}\left(G\right)$ and the number of tasks. This is because that the task level of GE application is not as strict as the FFT application, so that the effect of our method has diminished. In summary, our method is more applicable when the energy constraints are stringent, the number of tasks is large, or the task level of the application is strict.

6. Conclusions

In this article, we propose a novel method to minimize the scheduling length of energy-constrained applications which run on a heterogeneous multi-core system. Our method mainly includes two parts: a novel task energy pre-allocation strategy and the schedule algorithm based on it; a re-adjustment mechanism of task execution frequency after preliminary scheduling. The core idea of our method is that the tasks in different hierarchies have different impacts on the whole application and the negative impact of local optimal scheduling should be reduced. Our method can be integrated into actual multi-core embedded systems, and it is particularly suitable for wearable devices, mobile robots and other products with high requirements for energy saving and performance. We carry out a considerable number of experiments with two practical parallel application models (FFT and GE). The results of experiments show that our method is generally superior to other existing algorithms. However, the experimental results also demonstrate the limitations of our method, which are that our method does not offer much of an advantage when the energy constraints are not stringent, the number of tasks is small or the task level of the application is not strict.

In the future, we will improve and extend our method, and some further studies will be done. The points that can be further studied are as follows:

• Consider other factors that affect the length of application scheduling, such as the way to determine the priority of tasks.
• Explore ways to improve the limitations of our method and make it more universal.
• Integrate our method into an actual embedded multi-core system and test its performance.
• Extend our method to study other indicators in multi-core task scheduling, such as reliability.