# Deep Federated Adaptation: An Adaptative Residential Load Forecasting Approach with Federated Learning

^{*}

## Abstract

**:**

## 1. Introduction

_{2}emission [3]. Residential load forecasting can assist sectors in balancing the generation and consumption of electricity, which improves energy efficiency through the management and conservation of energy.

- We propose a novel federated transfer approach DFA for residential STLF, which adopts a federated architecture to address the problems of data availability and privacy, and leverages transfer learning to deal with the non-IID datasets for improving forecasting performance;
- DFA is investigated for STLF of residential houses and has shown remarkable advantages in forecasting performance over other baseline models. Especially, the federated architecture is superior to the centralized architecture in computation time;
- The framework of DFA is extended with alternative transfer learning methods and all of them achieve good performances on STLF.

## 2. Technical Background

#### 2.1. Federated Learning Concepts

#### 2.2. Transfer Learning Concepts and MK-MMD

## 3. The Proposed Method

#### 3.1. The Overview of the Proposed Approach

#### 3.2. Federated Learning Process

#### 3.3. User Adaptation with Multiple Kernel Variant of Maximum Mean Discrepancies

#### 3.4. Learning Process and Summary

Algorithm 1: DFA: Deep Federated Adaptation |

Require: Data from different houses $\{{\mathcal{D}}_{1},{\mathcal{D}}_{2},\cdots ,{\mathcal{D}}_{N}\}$ |

Ensure: Adaptative forecasting models ${f}_{u}$ |

1: Build an initial global model ${f}_{G}$ with public datasets using (7) |

2: repeat |

3: Distribute ${f}_{G}$ to all computing devices |

4: Use local data to train ${f}_{u}$ based on ${f}_{G}$ with (8) |

5: Devices upload ${f}_{u}$ to the model server |

6: Average models with (9) |

7: Update the global model ${f}_{G}={{f}_{G}}^{\prime}$ |

8: until $\left|{\Theta}_{G}-{{\Theta}_{G}}^{*}\right|<\epsilon $ |

9: Use (12) to optimize the convergent global model ${f}_{G}$, get adaptative models ${f}_{u}$ |

## 4. Experiments

#### 4.1. Datasets Description and Pre-Processing

#### 4.2. Implementation Information

- LSTM network: the model is an artificial recurrent neural network (RNN) architecture with feedback connections used in the field of deep learning.
- Double seasonal Holt–Winters (DSHW): DSHW is a kind of exponential smoothing method which can accommodate two seasonal patterns besides parts of trend and level.
- Transformer: it is a deep learning model that adopts the mechanism of self-attention, differentially weighting the significance of each part of the input data.
- Encoder–Decoder: the model encodes the input as embedding features which are decoded by the decoder, adopting a sequence-to-sequence architecture.

#### 4.3. Model Evaluation Indexes

#### 4.4. Experimental Forecasting Performance

#### 4.5. Performance of Federated and Centralized Architecture

#### 4.6. Ablation and Extensibility Experiments

## 5. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Deb, C.; Zhang, F.; Yang, J.; Lee, S.E.; Shah, K.W. A review on time series forecasting techniques for building energy consumption. Renew. Sustain. Energy Rev.
**2017**, 74, 902–924. [Google Scholar] [CrossRef] - Amasyali, K.; El-Gohary, N.M. A review of data-driven building energy consumption prediction studies. Renew. Sustain. Energy Rev.
**2018**, 81, 1192–1205. [Google Scholar] [CrossRef] - Zhao, H.-x.; Magoulès, F. A review on the prediction of building energy consumption. Renew. Sustain. Energy Rev.
**2012**, 16, 3586–3592. [Google Scholar] [CrossRef] - Wu, D.; Wang, B.; Precup, D.; Boulet, B. Multiple Kernel Learning-Based Transfer Regression for Electric Load Forecasting. IEEE Trans. Smart Grid
**2020**, 11, 1183–1192. [Google Scholar] [CrossRef] - Liu, D.; Chen, Q.; Mori, K. Time series forecasting method of building energy consumption using support vector regression. In Proceedings of the 2015 IEEE International Conference on Information and Automation, Lijiang, China, 8–10 August 2015; pp. 1628–1632. [Google Scholar]
- Pavićević, M.; Popović, T. Forecasting Day-Ahead Electricity Metrics with Artificial Neural Networks. Sensors
**2022**, 22, 1051. [Google Scholar] [CrossRef] - Ahmad, T.; Chen, H. Short and medium-term forecasting of cooling and heating load demand in building environment with data-mining based approaches. Energy Build.
**2018**, 166, 460–476. [Google Scholar] [CrossRef] - Liu, N.; Tang, Q.; Zhang, J.; Fan, W.; Liu, J. A hybrid forecasting model with parameter optimization for short-term load forecasting of micro-grids. Appl. Energy
**2014**, 129, 336–345. [Google Scholar] [CrossRef] - Fan, G.F.; Peng, L.L.; Hong, W.C.; Sun, F. Electric load forecasting by the SVR model with differential empirical mode decomposition and auto regression. Neurocomputing
**2016**, 173, 958–970. [Google Scholar] [CrossRef] - Zhou, H.; Zhang, S.; Peng, J.; Zhang, S.; Li, J.; Xiong, H.; Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI, virtually, 2–9 February 2021. [Google Scholar]
- Meng, Z.; Xu, X. A hybrid short-term load forecasting framework with an attention-based encoder–decoder network based on seasonal and trend adjustment. Energies
**2019**, 12, 4612. [Google Scholar] [CrossRef] [Green Version] - Savi, M.; Olivadese, F. Short-term energy consumption forecasting at the edge: A federated learning approach. IEEE Access
**2021**, 9, 95949–95969. [Google Scholar] [CrossRef] - Khwaja, A.S.; Anpalagan, A.; Naeem, M.; Venkatesh, B. Smart Meter Data Obfuscation Using Correlated Noise. IEEE Internet Things J.
**2020**, 7, 7250–7264. [Google Scholar] [CrossRef] - Yang, Q.; Liu, Y.; Chen, T.; Tong, Y. Federated machine learning: Concept and applications. ACM Trans. Intell. Syst. Technol. (TIST)
**2019**, 10, 1–19. [Google Scholar] [CrossRef] - Gholizadeh, N.; Musilek, P. Federated learning with hyperparameter-based clustering for electrical load forecasting. Internet Things
**2022**, 17, 100470. [Google Scholar] [CrossRef] - Moradzadeh, A.; Moayyed, H.; Mohammadi-Ivatloo, B.; Aguiar, A.P.; Anvari-Moghaddam, A. A Secure Federated Deep Learning-Based Approach for Heating Load Demand Forecasting in Building Environment. IEEE Access
**2022**, 10, 5037–5050. [Google Scholar] [CrossRef] - Yang, T.; Andrew, G.; Eichner, H.; Sun, H.; Li, W.; Kong, N.; Ramage, D.; Beaufays, F. Applied federated learning: Improving google keyboard query suggestions. arXiv
**2018**, arXiv:1812.02903. [Google Scholar] - Wu, X.; Liang, Z.; Wang, J. Fedmed: A federated learning framework for language modeling. Sensors
**2020**, 20, 4048. [Google Scholar] [CrossRef] [PubMed] - Chen, Y.; Qin, X.; Wang, J.; Yu, C.; Gao, W. Fedhealth: A federated transfer learning framework for wearable healthcare. IEEE Intell. Syst.
**2020**, 35, 83–93. [Google Scholar] [CrossRef] [Green Version] - Pokhrel, S.R.; Choi, J. Federated learning with blockchain for autonomous vehicles: Analysis and design challenges. IEEE Trans. Commun.
**2020**, 68, 4734–4746. [Google Scholar] [CrossRef] - Ye, D.; Yu, R.; Pan, M.; Han, Z. Federated learning in vehicular edge computing: A selective model aggregation approach. IEEE Access
**2020**, 8, 23920–23935. [Google Scholar] [CrossRef] - Abreha, H.G.; Hayajneh, M.; Serhani, M.A. Federated Learning in Edge Computing: A Systematic Survey. Sensors
**2022**, 22, 450. [Google Scholar] [CrossRef] - Tzeng, E.; Hoffman, J.; Zhang, N.; Saenko, K.; Darrell, T. Deep domain confusion: Maximizing for domain invariance. arXiv
**2014**, arXiv:1412.3474. [Google Scholar] - Wang, H.; Lei, Z.; Zhang, X.; Zhou, B.; Peng, J. A review of deep learning for renewable energy forecasting. Energy Convers. Manag.
**2019**, 198, 111799. [Google Scholar] [CrossRef] - Xu, X.; Meng, Z. A hybrid transfer learning model for short-term electric load forecasting. Electr. Eng.
**2020**, 102, 1371–1381. [Google Scholar] [CrossRef] - Cai, L.; Gu, J.; Jin, Z. Two-Layer Transfer-Learning-Based Architecture for Short-Term Load Forecasting. IEEE Trans. Ind. Inform.
**2020**, 16, 1722–1732. [Google Scholar] [CrossRef] - Alhmoud, L.; Abu Khurma, R.; Al-Zoubi, A.M.; Aljarah, I. A Real-Time Electrical Load Forecasting in Jordan Using an Enhanced Evolutionary Feedforward Neural Network. Sensors
**2021**, 21, 6240. [Google Scholar] [CrossRef] [PubMed] - Long, M.; Cao, Y.; Wang, J.; Jordan, M. Learning transferable features with deep adaptation networks. In Proceedings of the 32nd International Conference on Machine Learning, Lille, France, 7–9 July 2015; pp. 97–105. [Google Scholar]
- Gretton, A.; Borgwardt, K.M.; Rasch, M.J.; Schölkopf, B.; Smola, A. A kernel two-sample test. J. Mach. Learn. Res.
**2012**, 13, 723–773. [Google Scholar] - McMahan, B.; Moore, E.; Ramage, D.; Hampson, S.; y Arcas, B.A. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, USA, 20–22 April 2017; pp. 1273–1282. [Google Scholar]
- Agga, A.; Abbou, A.; Labbadi, M.; Houm, Y.E. Short-term self consumption PV plant power production forecasts based on hybrid CNN-LSTM, ConvLSTM models. Renew. Energy
**2021**, 177, 101–112. [Google Scholar] [CrossRef] - Smart Meter Energy Consumption Data in London Households. Available online: https://www.kaggle.com/jeanmidev/smart-meters-in-london (accessed on 10 January 2022).
- DarkSky Service. Available online: ttps://darksky.net (accessed on 10 January 2022).
- Sun, B.; Saenko, K. Deep coral: Correlation alignment for deep domain adaptation. In European Conference on Computer Vision; Springer: Berlin/Heidelberg, Germany, 2016; pp. 443–450. [Google Scholar]

**Figure 2.**Overview of the deep federated adaptation. The top box is the master server while the 3 bottom boxes denotes 3 houses. Each house contains one computing device connected to the master server for processing the data. The data collected by the smart meter is locked and cannot be transmitted to the master server.

**Figure 3.**The architecture of proposed network, from top to bottom, consists of CNN layers, BiLSTM layers and fully connected layers.

**Figure 6.**MAPE values of four baseline models and DFA with different numbers of houses connected to the federated system for 10 houses.

**Figure 7.**Correlation between accuracy and number of rounds for the federated and centralized architecture.

Name | Description |
---|---|

${N}_{nodes}$ | Number of computing nodes |

${N}_{round}$ | Number of iterations |

${S}_{data}$ | Size of the trained data |

${S}_{model}$ | Size of the model |

DataSets | DFA | Transformer | DSHW | Encoder–Decoder | LSTM |
---|---|---|---|---|---|

House 1 | 4.92% | 8.86% | 13.76% | 13.59% | 14.74% |

House 2 | 4.56% | 8.43% | 18.94% | 11.27% | 15.47% |

House 3 | 4.77% | 9.02% | 16.72% | 13.34% | 16.43% |

House 4 | 5.23% | 8.56% | 15.36% | 14.67% | 16.86% |

House 5 | 5.13% | 8.31% | 16.88% | 15.67% | 15.92% |

House 6 | 4.88% | 8.73% | 18.23% | 12.39% | 15.13% |

House 7 | 4.89% | 8.64% | 19.89% | 13.17% | 14.62% |

House 8 | 4.78% | 8.52% | 21.88% | 13.40% | 14.33% |

House 9 | 4.47% | 9.16% | 17.94% | 15.17% | 15.31% |

House 10 | 4.67% | 8.68% | 15.46% | 15.55% | 15.26% |

Average | 4.83% | 8.69% | 17.51% | 13.82% | 15.41% |

Model | Day 3 | Day 11 | Day 15–Day 21 |
---|---|---|---|

DFA | 6.43% | 6.17 % | 8.83 % |

DSHW | 16.85% | 17.23% | 17.61% |

Transformer | 8.77% | 9.22 % | 10.24% |

Encoder–Decoder | 13.64% | 14.56 % | 16.79% |

LSTM | 18.23% | 17.64 % | 21.54% |

Approach | Forecasting Performance | Computation Time (s) | ||||
---|---|---|---|---|---|---|

Number of Local Records | 5000 | 8000 | 15,000 | 5000 | 8000 | 15,000 |

Federated (${N}_{nodes}=1$) | 13.82% | 13.42% | 13.38% | 5.26 | 5.58 | 5.73 |

Federated (${N}_{nodes}=4$) | 12.83% | 12.33% | 11.89% | 5.38 | 5.73 | 5.84 |

Federated (${N}_{nodes}=7$) | 11.57% | 11.35% | 10.87% | 5.23 | 5.46 | 5.91 |

Federated (${N}_{nodes}=10$) | 10.24% | 10.13% | 9.83% | 5.47 | 5.62 | 5.88 |

Centralized (${N}_{nodes}=10$) | 12.13% | 12.24% | 12.07% | 8.32 | 11.74 | 25.63 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Shi, Y.; Xu, X.
Deep Federated Adaptation: An Adaptative Residential Load Forecasting Approach with Federated Learning. *Sensors* **2022**, *22*, 3264.
https://doi.org/10.3390/s22093264

**AMA Style**

Shi Y, Xu X.
Deep Federated Adaptation: An Adaptative Residential Load Forecasting Approach with Federated Learning. *Sensors*. 2022; 22(9):3264.
https://doi.org/10.3390/s22093264

**Chicago/Turabian Style**

Shi, Yuan, and Xianze Xu.
2022. "Deep Federated Adaptation: An Adaptative Residential Load Forecasting Approach with Federated Learning" *Sensors* 22, no. 9: 3264.
https://doi.org/10.3390/s22093264