# Highly Adaptive Linear Actor-Critic for Lightweight Energy-Harvesting IoT Applications

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

- To provide better adaptability, we combined and evaluated the LAC algorithm with Adam (LAC-A) using smaller decay factors for transmission (TX) duty-cycle optimization in an application of sensor data TX in a point-to-point network.
- With the use of smaller decay rates in Adam, we can exclude the initialization bias correction terms to reduce the calculations, and we call the algorithm LAC-AB (LAC with Adam Biased).
- We defined the time of convergence quantitatively based on the mean and variance information for evaluating the speed of convergence of different approaches.
- Simulation results show that, for our application use case, smaller decay rates ${\beta}_{1}$ such as 0.2–0.4 are better for power-failure-sensitive applications and 0.5–0.7 for latency-sensitive applications with ${\beta}_{2}\in [0.1,0.3]$ in the LAC-AB algorithm.
- We show that LAC-AB with such decay rates helps achieve more reactivity and stability to drastic gradient changes, such as doubled workload, and enables fine-tuning the actions to the initial state more quickly.

## 2. Related Work

## 3. System Model

#### 3.1. Energy Harvesting and State-of-Charge Model

#### 3.2. Application Data and State-of-Buffer model

#### 3.3. Power Consumption and Transmission Model

## 4. Application Scenario

- -
- The control update interval (CUI) is ${T}^{cui}=30$ min.
- -
- For the PV cell model, we set the cell area A, conversion efficiency $\eta $, and tracking factor $TF$ as 2.5 ${\mathrm{cm}}^{2}$, $10\%$, and $96.3\%$, respectively [7]. This choice is consistent with an off-the-shelf solar cell that can harvest power from $\mu \mathrm{W}$ to $\mathrm{mW}$ per cm${}^{2}$, depending on the lighting condition [32]. Note that we use the real-life solar irradiance data provided by Oak Ridge National Laboratory [33].
- -
- The self-discharge of a supercapacitor whose capacitor size is $1\mathrm{F}$ is considered $20\%$ per day (detailed in Section 3.1). The harvest–use–store scheme is adopted [25,26] to provide high energy efficiency.
- -
- The wireless link quality is under the influence of path-loss and shadowing.
- -
- The workload follows the Poisson distribution. The average rate doubles after the first six months (where the algorithm is put through the test of fast adaptability/reactivity). More precisely, the system receives the average of 1.0 $\mathrm{pkt}/\mathrm{min}$ for the first six months, and it impulsively becomes twice (2.0 $\mathrm{pkt}/\mathrm{min}$) afterwards.

- EHD1: Non-processed real-life one-year data from 1 June 2018 to 31 May 2019
- EHD2: Real-life one-year data made by stacking 365 one day (1 December 2018) worth of solar irradiance data
- EHD3: Real-life one-year data made by stacking 365 one day (1 June 2018) worth of solar irradiance data

## 5. Reinforcement Learning

- Find the optimal policy ${\pi}^{*}$.
- Determine the estimate of the value function ${v}_{\pi}$ under a certain policy;

## 6. Algorithm: LAC-AB

- -
- Smaller values lead to more weight on recent changes, i.e., faster online adaptation.
- -
- The gradient variance can become too large and yet carries an important information for parameter updates that can be lost with larger values of ${\beta}_{1}$ and ${\beta}_{2}$.

Algorithm 1 LAC-AB: LAC Algorithm Using Adam with No Initialization Bias Correction |

Require:**$/*\phantom{\rule{4pt}{0ex}}$ Inputs$\phantom{\rule{4pt}{0ex}}*/$**- - State-of-Buffer ${\psi}_{t+1}^{SoB}$ and State-of-Charge ${\psi}_{t+1}^{SoC}$
**$/*\phantom{\rule{4pt}{0ex}}$ Hyper-parameters for Actor-Critic$\phantom{\rule{4pt}{0ex}}*/$**- - Learning rates $\beta $ and $\alpha $ for Actor and Critic, respectively
- - Discount factor $\gamma \in [0,1]$ for past reward ${R}_{t+1}$
- - Recency weight $\lambda \in [0,1]$ in the TD($\lambda $) algorithm
- - Exploration space $\sigma $ (standard deviation for the Gaussian policy)
- - Decay rates ${\beta}_{1}\in [0,1]$ and ${\beta}_{2}\in [0,1]$ for EWMA in Adam
- - $\u03f5$ to avoid division by infinitesimally small values in Adam
Ensure:- - Action ${a}_{t}\in [{a}^{min},{a}^{max}]$
- - Actor and Critic parameter ${\psi}_{t}$ and ${\theta}_{t}$
Initialize at time $t=0$:
- - Empty data buffer ${\psi}_{0}^{SoB}=0$ and fully-charged energy buffer ${\psi}_{0}^{SoC}=1$
- - ${\psi}_{0}$ and ${\theta}_{0}$ are random numbers ranging $[0,1]$
for each$t\in [0,\infty ]$do**$/*\phantom{\rule{4pt}{0ex}}$ Observe the current state$\phantom{\rule{4pt}{0ex}}*/$**
3: ${R}_{t+1}=(1-{\varphi}_{t+1}^{SoB})\xb7{\varphi}_{t+1}^{SoC}$ ▹ For minimizing SoB and maximizing SoC 4: ${V}_{t}={\theta}_{t}\xb7(1-{\varphi}_{t}^{SoB})\xb7{\varphi}_{t}^{SoC}$ ▹ Less SoB and more SoC are better states (better values) 5: $\widehat{{V}_{t+1}}={\theta}_{t}\xb7(1.0-{\varphi}_{t+1}^{SoB})\xb7{\varphi}_{t+1}^{SoC}$ $/*\phantom{\rule{4pt}{0ex}}$ TD-error for Actor-Critic$\phantom{\rule{4pt}{0ex}}*/$6: ${\delta}_{t+1}^{TD}=R(t+1)+\gamma \widehat{{V}_{t+1}}-{V}_{t}$▹ Advantage function: $A(s,a)=Q(s,a)-V\left(s\right)$ ($Q(s,a)$: state-action value function) $/*\phantom{\rule{4pt}{0ex}}$ Critic: TD($\lambda $) algorithm$\phantom{\rule{4pt}{0ex}}*/$7: ${z}_{t+1}=\gamma \lambda {z}_{t}+(1-{\varphi}_{t+1}^{SoB})\xb7{\varphi}_{t+1}^{SoC}$ ▹ Calculate the eligibility trace ${z}_{t+1}$ 8: ${\theta}_{t+1}={\theta}_{t}+\alpha {\delta}_{t+1}{z}_{t+1}$ ▹ Update the critic parameter $/*\phantom{\rule{4pt}{0ex}}$ Actor: Policy gradient theorem using Adam with no initialization bias corrections$\phantom{\rule{4pt}{0ex}}*/$9: ${g}_{t+1}={\delta}_{t+1}\xb7\frac{{a}_{t}-{\mu}_{t}}{{\sigma}^{2}}\xb7{\varphi}_{t}^{SoB}\xb7{\varphi}_{t}^{SoC}$ 10: ${m}_{t+1}={\beta}_{1}\xb7{m}_{t}+(1-{\beta}_{1})\xb7{g}_{t+1}$ ▹ Estimate the first-order moment 11: ${v}_{t+1}={\beta}_{2}\xb7{v}_{t}+(1-{\beta}_{2})\xb7{g}_{t+1}^{2}$ ▹ Estimate the second-order moment 12: ${\psi}_{t+1}={\psi}_{t}+{\beta}_{t+1}\xb7\frac{{m}_{t+1}}{\sqrt{{v}_{t+1}}+\u03f5}$ ▹ Update the actor parameter $/*\phantom{\rule{4pt}{0ex}}$ Next TX duty-cycle selection$\phantom{\rule{4pt}{0ex}}*/$13: ${\mu}_{t+1}={\psi}_{t+1}\xb7{\varphi}_{t+1}^{SoB}\xb7{\varphi}_{t+1}^{SoC}$ ▹ Less SoB, smaller action values; More SoC, higher action values 14: ${\mu}_{t+1}\leftarrow $ Clamp ${\mu}_{t+1}$ to $[{a}^{min},{a}^{max}]$ 15: ${a}_{t+1}\sim \mathcal{N}({\mu}_{t+1},\sigma )$ ▹ Gaussian policy for action generation 16: ${a}_{t+1}\leftarrow $ Clamp ${a}_{t+1}$ to $[{a}^{min},{a}^{max}]$ 17: Return ${a}_{t+1}$18: end for each |

## 7. Definition of Convergence

- All the mean values (e.g., actor parameter values ${\psi}_{t}$) taken over all the simulations at the same time points in a x-day sweeping window are all within $5\%$ error band of the average of all the mean values in the last x-day window under almost the same state (e.g., under the same workload scenario in our test study here).
- The variances of the mean values taken over all the simulations at the same time points are confirmed to be not different, i.e., the homogeneity of variance is tested and confirmed by means of Levene’s test [36], more precisely Brown–Forsythe [37] test, with the confidence interval of $y\%$ (note that we cannot say “the same” mathematically).

## 8. Simulation Results

#### 8.1. Divergence and Reactivity Problem

#### 8.2. Effectiveness and Convergence of LAC-AB

#### 8.3. Decay Rates Study for LAC-AB

## 9. Conclusions and Future Direction

## Author Contributions

## Funding

## Data Availability Statement

## Conflicts of Interest

## References

- Lallement, G.; Abouzeid, F.; Cochet, M.; Daveau, J.; Roche, P.; Autran, J. A 2.7 pJ/cycle 16 MHz SoC with 4.3 nW power-off ARM Cortex-M0+ core in 28 nm FD-SOI. In Proceedings of the ESSCIRC 2017—43rd IEEE European Solid State Circuits Conference, Leuven, Belgium, 11–14 September 2017; pp. 153–162. [Google Scholar] [CrossRef] [Green Version]
- Ait Aoudia, F.; Gautier, M.; Berder, O. RLMan: An Energy Manager Based on Reinforcement Learning for Energy Harvesting Wireless Sensor Networks. IEEE Trans. Green Commun. Netw.
**2018**, 2, 408–417. [Google Scholar] [CrossRef] [Green Version] - Ortiz, A.; Al-Shatri, H.; Li, X.; Weber, T.; Klein, A. Reinforcement learning for energy harvesting point-to-point communications. In Proceedings of the 2016 IEEE International Conference on Communications (ICC), Kuala Lumpur, Malaysia, 22–27 May 2016; pp. 1–6. [Google Scholar]
- Van Hasselt, H.; Wiering, M.A. Using continuous action spaces to solve discrete problems. In Proceedings of the 2009 International Joint Conference on Neural Networks, Atlanta, GA, USA, 14–19 June 2009; pp. 1149–1156. [Google Scholar] [CrossRef] [Green Version]
- Kingma, D.P.; Ba, J. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA, 7–9 May 2015. [Google Scholar]
- De Roose, J.; Xin, H.; Andraud, M.; Harpe, P.J.A.; Verhelst, M. Flexible and Self-Adaptive Sense-and-Compress for Sub-MicroWatt Always-on Sensory Recording. In Proceedings of the ESSCIRC 2018—IEEE 44th European Solid State Circuits Conference (ESSCIRC), Dresden, Germany, 3–6 September 2018; pp. 282–285. [Google Scholar] [CrossRef]
- Ju, Q.; Zhang, Y. Predictive Power Management for Internet of Battery-Less Things. IEEE Trans. Power Electron.
**2018**, 33, 299–312. [Google Scholar] [CrossRef] - Qiu, C.; Hu, Y.; Chen, Y.; Zeng, B. Lyapunov Optimization for Energy Harvesting Wireless Sensor Communications. IEEE Internet Things J.
**2018**, 5, 1947–1956. [Google Scholar] [CrossRef] - Hu, Y.; Qiu, C.; Chen, Y. Lyapunov-Optimized Two-Way Relay Networks With Stochastic Energy Harvesting. IEEE Trans. Wirel. Commun.
**2018**, 17, 6280–6292. [Google Scholar] [CrossRef] - Bhat, G.; Park, J.; Ogras, U.Y. Near-optimal energy allocation for self-powered wearable systems. In Proceedings of the 2017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), Irvine, CA, USA, 13-16 November 2017; pp. 368–375. [Google Scholar] [CrossRef]
- Bhat, G.; Bagewadi, K.; Lee, H.G.; Ogras, U.Y. REAP: Runtime Energy-Accuracy Optimization for Energy Harvesting IoT Devices. In Proceedings of the 2019 56th ACM/IEEE Design Automation Conference (DAC), Las Vegas, NV, USA, 2–6 June 2019; pp. 1–6. [Google Scholar]
- Vigorito, C.M.; Ganesan, D.; Barto, A.G. Adaptive Control of Duty Cycling in Energy-Harvesting Wireless Sensor Networks. In Proceedings of the 2007 4th Annual IEEE Communications Society Conference on Sensor, Mesh and Ad Hoc Communications and Networks, San Diego, CA, USA, 18–21 June 2007; pp. 21–30. [Google Scholar] [CrossRef] [Green Version]
- Debizet, Y.; Lallement, G.; Abouzeid, F.; Roche, P.; Autran, J. Q-Learning-based Adaptive Power Management for IoT System-on-Chips with Embedded Power States. In Proceedings of the 2018 IEEE International Symposium on Circuits and Systems (ISCAS), Florence, Italy, 27–30 May 2018; pp. 1–5. [Google Scholar] [CrossRef]
- Masadeh, A.; Wang, Z.; Kamal, A.E. An Actor-Critic Reinforcement Learning Approach for Energy Harvesting Communications Systems. In Proceedings of the 2019 28th International Conference on Computer Communication and Networks (ICCCN), Valencia, Spain, 29 July–1 August 2019; pp. 1–6. [Google Scholar] [CrossRef]
- Murad, A.; Kraemer, F.A.; Bach, K.; Taylor, G. Autonomous Management of Energy-Harvesting IoT Nodes Using Deep Reinforcement Learning. In Proceedings of the 2019 IEEE 13th International Conference on Self-Adaptive and Self-Organizing Systems (SASO), Umea, Sweden, 16–20 June 2019; pp. 43–51. [Google Scholar] [CrossRef] [Green Version]
- Qian, L.P.; Feng, A.; Feng, X.; Wu, Y. Deep RL-Based Time Scheduling and Power Allocation in EH Relay Communication Networks. In Proceedings of the ICC 2019—2019 IEEE International Conference on Communications (ICC), Shanghai, China, 20–24 May 2019; pp. 1–7. [Google Scholar] [CrossRef]
- Li, M.; Zhao, X.; Liang, H.; Hu, F. Deep Reinforcement Learning Optimal Transmission Policy for Communication Systems With Energy Harvesting and Adaptive MQAM. IEEE Trans. Veh. Technol.
**2019**, 68, 5782–5793. [Google Scholar] [CrossRef] - Sawaguchi, S.; Christmann, J.F.; Molnos, A.; Bernier, C.; Lesecq, S. Multi-Agent Actor-Critic Method for Joint Duty-Cycle and Transmission Power Control. In Proceedings of the 2020 Design, Automation Test in Europe Conference Exhibition (DATE), Grenoble, France, 9–13 March 2020; pp. 1015–1018. [Google Scholar]
- Li, D.; Xu, S.; Zhao, J. Partially Observable Double DQN Based IoT Scheduling for Energy Harvesting. In Proceedings of the 2019 IEEE International Conference on Communications Workshops (ICC Workshops), Shanghai, China, 20–24 May 2019; pp. 1–6. [Google Scholar] [CrossRef]
- Zhao, N.; Liang, Y.; Niyato, D.; Pei, Y.; Jiang, Y. Deep Reinforcement Learning for User Association and Resource Allocation in Heterogeneous Networks. In Proceedings of the 2018 IEEE Global Communications Conference (GLOBECOM), Abu Dhabi, United Arab Emirates, 9–13 December 2018; pp. 1–6. [Google Scholar] [CrossRef]
- Qiu, C.; Hu, Y.; Chen, Y.; Zeng, B. Deep Deterministic Policy Gradient (DDPG)-Based Energy Harvesting Wireless Communications. IEEE Internet Things J.
**2019**, 6, 8577–8588. [Google Scholar] [CrossRef] - Biswas, D.; Balagopal, V.; Shafik, R.; Al-Hashimi, B.M.; Merrett, G.V. Machine learning for run-time energy optimisation in many-core systems. In Proceedings of the Design, Automation Test in Europe Conference Exhibition (DATE), Lausanne, Switzerland, 27–31 March 2017; pp. 1588–1592. [Google Scholar] [CrossRef] [Green Version]
- Das, A.; Merrett, G.V.; Tribastone, M.; Al-Hashimi, B.M. Workload Change Point Detection for Runtime Thermal Management of Embedded Systems. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst.
**2016**, 35, 1358–1371. [Google Scholar] [CrossRef] - Shafik, R.A.; Yang, S.; Das, A.; Maeda-Nunez, L.A.; Merrett, G.V.; Al-Hashimi, B.M. Learning Transfer-Based Adaptive Energy Minimization in Embedded Systems. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst.
**2016**, 35, 877–890. [Google Scholar] [CrossRef] [Green Version] - Yuan, F.; Zhang, Q.T.; Jin, S.; Zhu, H. Optimal Harvest-Use-Store Strategy for Energy Harvesting Wireless Systems. IEEE Trans. Wirel. Commun.
**2015**, 14, 698–710. [Google Scholar] [CrossRef] - Christmann, J.F.; Beigne, E.; Condemine, C.; Vivet, P.; Willemin, J.; Leblond, N.; Piguet, C. Bringing Robustness and Power Efficiency to Autonomous Energy-Harvesting Microsystems. IEEE Des. Test Comput.
**2011**, 28, 84–94. [Google Scholar] [CrossRef] - Dekimpe, R.; Xu, P.; Schramme, M.; Flandre, D.; Bol, D. A Battery-Less BLE IoT Motion Detector Supplied by 2.45-GHz Wireless Power Transfer. In Proceedings of the International Symposium on Power and Timing Modeling, Optimization and Simulation (PATMOS) 2018, Platja d’Aro, Spain, 2–4 July 2018; pp. 68–75. [Google Scholar] [CrossRef]
- Sangare, F.; Xiao, Y.; Niyato, D.; Han, Z. Mobile Charging in Wireless-Powered Sensor Networks: Optimal Scheduling and Experimental Implementation. IEEE Trans. Veh. Technol.
**2017**, 66, 7400–7410. [Google Scholar] [CrossRef] - Texas Instruments. CC2500 Low-Cost Low-Power 2.4 GHz RF Transceiver. 2019. Available online: http://www.ti.com/lit/ds/swrs040c/swrs040c.pdf (accessed on 6 April 2021).
- Urard, P.; Romagnello, G.; Banciu, A.; Grasset, J.C.; Heinrich, V.; Boulemnakher, M.; Todeschni, F.; Damon, L.; Guizzetti, R.; Andre, L.; et al. A self-powered IPv6 bidirectional wireless sensor amp; actuator network for indoor conditions. In Proceedings of the 2015 Symposium on VLSI Circuits (VLSI Circuits), Kyoto, Japan, 17–19 June 2015; pp. C100–C101. [Google Scholar] [CrossRef]
- Goldsmith, A. Wireless Communications; Cambridge University Press: Cambridge, UK, 2005. [Google Scholar] [CrossRef] [Green Version]
- Panasonic Industry. Available online: https://www.panasonic-electric-works.com/cps/rde/xbcr/pew_eu_en/ca_amorton_solar_cells_en.pdf (accessed on 6 April 2021).
- Oak Ridge National Laboratory (RSR) Daily Plots and Raw Data Files. Available online: https://midcdmz.nrel.gov/apps/sitehome.pl?site=ORNL (accessed on 6 April 2021).
- Sutton, R.S.; Barto, A.G. Reinforcement Learning: An Introduction, 2nd ed.; The MIT Press: Cambridge, MA, USA, 2018. [Google Scholar]
- Devlin, S.; Yliniemi, L.; Kudenko, D.; Tumer, K. Potential-based Difference Rewards for Multiagent Reinforcement Learning. In Proceedings of the 2014 International Conference on Autonomous Agents and Multi-agent Systems (AAMAS’14), Paris, France, 5–9 May 2014; pp. 165–172. [Google Scholar]
- Gleser, L.J.; Perlman, M.D.; Press, S.J.; Sampson, A.R. Contributions to Probability and Statistics: Essays in Honor of Harold Hotelling; Springer: Berlin/Heidelberg, Germany, 1960; pp. 278–292. [Google Scholar]
- Brown, M.B.; Forsythe, A.B. Robust Tests for the Equality of Variances. J. Am. Stat. Assoc.
**1974**, 69, 364–367. [Google Scholar] [CrossRef] - Sheskin, D.J. Handbook of Parametric and Nonparametric Statistical Procedures, 5th ed.; CRC Press: Boca Raton, FL, USA, 2011. [Google Scholar]

**Figure 3.**Transitions of actor parameter $\psi $ using LAC-AB (${\beta}_{1},{\beta}_{2}=0.4$) for EHD1, EHD2, and EHD3.

**Figure 5.**Divergences of $\psi $ (

**left**) and TD-errors (

**right**) over one year with the state-of-the-art LAC method (red crosses indicate the power failure points).

**Figure 6.**Transitions of $\psi $ (

**left**) and TD-errors (

**right**) in case of the LAC algorithm using fixed learning rate with the SoB as a performance upper bound (red crosses represent when power failure(s) occurred).

**Figure 7.**Transition of actor parameter $\psi $ using Adam and EHD1 dataset: (

**left**) LAC-A (${\beta}_{1}=0.9$, ${\beta}_{2}=0.999$); (

**middle**) LAC-A (${\beta}_{1}={\beta}_{2}=0.4$); and (

**right**) LAC-AB (${\beta}_{1}={\beta}_{2}=0.4$).

**Figure 8.**Power failure and Convergence speed analysis of LAC-AB for different sets of $({\beta}_{1},{\beta}_{2})$ using EHD1 and EHD3 datasets, respectively: (

**left**) number of power failures; (

**middle**) ToF; and (

**right**) ToR.

**Figure 9.**Latency analysis of LAC-AB during the first sxi months for different sets of $({\beta}_{1},{\beta}_{2})$ using EHD1 dataset: (

**left**) mean; and (

**right**) standard deviation.

**Figure 10.**Latency analysis of LAC-AB during the last six months for different sets of $({\beta}_{1},{\beta}_{2})$ using EHD1 dataset: (

**left**) mean; and (

**right**) standard deviation.

Paper | Method | SoB | SoC | Harvester | Action | Neural? | LR |
---|---|---|---|---|---|---|---|

[6] | Online statistics & offline optimization | - | Finite | - | Sense & Compress setups | No | NA |

[7] | Prediction & online stepwise adjustment | - | Finite | Solar | Duty-cycle, TX power | No | Fixed |

[8] | Lyapunov optimization | - | Finite | Solar | TX modulation | No | NA |

[9] | Lyapunov optimization | - | Finite | Solar | TX power | No | NA |

[10] | Online KKT & prediction | - | Finite | Solar | Duty-cycle | No | Fixed |

[11] | Simplex algorithm with prediction and offline data | - | Finite | Solar | Active time (Duty-cycle), accuracy | No | NA |

[13] | Q-learning | - | Finite | - | Suspension mode selection | No | Fixed |

[14] | Actor-Critic | Infinite | Finite | Solar | TX power | 3 layers (3-10-5-1) | Fixed |

[12] | Linear-Quadratic Regulator | - | Finite | Solar | Duty-cycle | No | Fixed |

[15] | DRL | - | Finite | Solar | Duty-cycle | 3 layers (4-64-64-2) | Fixed |

[16] | Asynchronous Advantage Actor-Critic | Finite | Finite | Uniform distribution | Duty-cycle, TX power (source and relay) | 3 layers (5-300-200-3(1)) | Fixed |

[17] | Deep RL | Infinite | Finite | Uniform distribution | TX modulation (=TX power) | 3 layers (3-10-10-1) | Fixed or decaying |

[2] | Linear Actor-Critic | - | Finite | Solar, wind | Packet rate | No (Linear function) | Fixed |

[18] | Multi-Agent Actor-Critic | Finite | Finite | Solar | Duty-cycle, TX power | No (Linear function) | Fixed |

[19] | Double Deep Q-Network | - | Finite | Two-state Markov process | Uplink scheduling policy | Yes (No details) | Adam (less adaptive) |

[20] | Multi-Agent Double Deep Q-Network | - | - | - | Base station and channel selections | 4-layers (50-64-32-32-26(30)) | Adam (less adaptive) |

[21] | Deep Deterministic Policy Gradient | - | Finite | Solar | Energy budget | Actor: 3(4)-60-30-1, Critic: 3(4)-60-60-60-60-60-1 | Adam (less adaptive) |

This paper | Linear Actor-Critic | Finite | Finite | Solar | Duty-cycle | Linear function | Adam (highly adaptive) |

Algorithm | $\mathit{\alpha}$ | $\mathit{\beta}$ | $\mathit{\gamma}$ | $\mathit{\sigma}$ | $\mathit{\lambda}$ | $\mathit{\u03f5}$ |
---|---|---|---|---|---|---|

LAC | $0.1$ | $2.0\times {10}^{-6}$ | $0.9$ | $5.0\times {10}^{-4}$ | $0.9$ | $1.0\times {10}^{-6}$ |

LAC-A | $0.1$ | $3.0\times {10}^{-4}$ | $0.9$ | $5.0\times {10}^{-4}$ | $0.9$ | $1.0\times {10}^{-6}$ |

LAC-AB | $0.1$ | $3.0\times {10}^{-4}$ | $0.9$ | $5.0\times {10}^{-4}$ | $0.9$ | $1.0\times {10}^{-6}$ |

Algorithm | LAC-A | LAC-AB | ||
---|---|---|---|---|

${\beta}_{1}/{\beta}_{2}$ | $0.9/0.999$ | $0.4/0.4$ | $0.4/0.4$ | |

Latency (Mean/Std) | First 6 months | $3.40/10.54$ | $3.52/11.11$ | $3.52/11.14$ |

Last 6 months | $6.21/11.07$ | $6.06/10.76$ | $6.06/10.76$ | |

# of power failures/# of failed simulations | $28/28$ | $0/0$ | $0/0$ |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Sawaguchi, S.; Christmann, J.-F.; Lesecq, S.
Highly Adaptive Linear Actor-Critic for Lightweight Energy-Harvesting IoT Applications. *J. Low Power Electron. Appl.* **2021**, *11*, 17.
https://doi.org/10.3390/jlpea11020017

**AMA Style**

Sawaguchi S, Christmann J-F, Lesecq S.
Highly Adaptive Linear Actor-Critic for Lightweight Energy-Harvesting IoT Applications. *Journal of Low Power Electronics and Applications*. 2021; 11(2):17.
https://doi.org/10.3390/jlpea11020017

**Chicago/Turabian Style**

Sawaguchi, Sota, Jean-Frédéric Christmann, and Suzanne Lesecq.
2021. "Highly Adaptive Linear Actor-Critic for Lightweight Energy-Harvesting IoT Applications" *Journal of Low Power Electronics and Applications* 11, no. 2: 17.
https://doi.org/10.3390/jlpea11020017