A Privacy Robust Aggregation Method Based on Federated Learning in the IoT

Abstract

Federated learning has been widely applied because it enables a large number of IoT devices to conduct collaborative training while maintaining private data localization. However, the security risks and threats faced by federated learning in IoT applications are becoming increasingly prominent. Except for direct data leakage, there is also a need to face threats that attackers interpret gradients and infer private information. This paper proposes a Privacy Robust Aggregation Based on Federated Learning (PBA), which can be applied to multiple server scenarios. PBA filters outliers by using the approximate Euclidean distance calculated from binary sequences and the $3\sigma$ criterion. Then, this paper provides correctness analysis and computational complexity analysis on the aggregation process of PBA. Moreover, the performance of PBA is evaluated concerning ensuring privacy and robustness in this paper. The results indicate that PBA can resist Byzantine attacks and a state-of-the-art privacy inference, which means that PBA can ensure privacy and robustness.

1. Introduction

Internet of Things (IoT) applications can truly realize smart cities, which would require a massive number of IoT devices [1]. Owing to remarkable growth in IoT network size and data volume, these IoT devices can be made possible by centralized machine learning methods [2]. But as centralized machine learning techniques need to transfer the data to a centralized server for training, they are faced with the inherent issues of privacy leakage [3]. Moreover, centralized machine learning may not be practicable with large and dispersed data. Therefore, federated learning (FL) has been introduced to build intelligent and privacy-enhanced IoT systems [4,5,6].

FL is a distributed collaborative approach that coordinates the data training without sharing datasets [7], while FL itself has privacy concerns, it does not completely guarantee privacy [8,9,10]. Firstly, FL aims to train a model on multiple dispersed devices collaboratively by transferring local model updates, while only local update needs to be shared during the training process, there is also a high possibility of personal information leakage. This means that if sufficient measures are not taken, it will cause serious data privacy leakage risks. Secondly, the model inversion attack is a type of privacy attack that tries to infer the training data [11,12]. Deep Leakage from Gradients (DLG) [13] and Generative Adversarial Networks (GAN) [14] attacks are both types of model inversion attacks. DLG attack is a gradient-based attack method that can infer the input data of a model by accessing its gradient information. GAN attack is an attack method based on a generative adversarial network. It can generate data similar to the target model by training a generator network and then infer the input data of the model. Furthermore, due to the autonomy of workers in FL environments, any worker that participates in the process may be malicious and disrupt the security of FL [15]. These malicious workers can submit threatening local updates and malicious models, intentionally interfere with the convergence of the global model, and affect the efficiency of FL. Above all, FL is still affected by privacy inference [13,16,17] and Byzantine attacks [18,19]. And these security risks and threats faced by FL in IoT applications are also increasingly prominent [12,20,21]. It is essential to study privacy protection and Byzantine-robust FL methods in IoT scenarios.

The contributions of this paper are summarized as follows:

• We design an approximate Euclidean distance calculated by the sign matrix and data matrix, where the sign matrix and data matrix are composed of the sign and absolute value of the model update from each worker. It can decrease the computational complexity.
• We propose a privacy robust aggregation method based on FL (PBA) in a multi-server scenario. PBA is implemented based on approximate Euclidean distance and adopts scientific $3\sigma$ criterion instead of sorting to screen outliers. We also analyze the correctness and computational complexity of PBA.
• We conduct experiments and evaluate the performance of PBA concerning privacy security, Byzantine robustness, and time costs. The results indicate that PBA can ensure privacy and robustness with less time.

The rest of this paper is organized as follows: Section 2 summarizes the works related to ours while highlighting the differences; Section 3 introduces the threat model and some basic techniques; Section 4 designs a privacy robust aggregation rule based on approximate Euclidean distance and analyzes the correctness and computational complexity; Section 5 conducts experimental evaluation; Section 6 concludes the paper.

2. Related Work

Compared with traditional machine learning, FL already increases the level of privacy. It mitigates the transmission of sensitive data and prevents a third party from performing learning tasks on the unpermitted individual data [22]. However, the local updates/gradients uploaded by individuals, especially parts of personal data that are sensitive to specific features or values, may reveal sensitive information about users’ data. Moreover, bad data may be uploaded because of device malfunction or malicious behavior.

Many studies have been conducted to solve the issues mentioned above. Currently, studies on privacy-preserving FL use either secure multi-party computation (MPC) or differential privacy (DP). By combining the locally trained classifiers, Pathak et al. [23] suggested a DP-based global classifier. A framework was proposed that adjusted the objective function during the training process to achieve DP [24]. Other methods based on encryption include homomorphic encryption (HE) and secure aggregation protocol [25]. A privacy-preserving deep learning (PPDL) algorithm was proposed [26], in which a number of distributed participants work together to train a deep learning model using local data. They established a trade-off between efficiency and security for the number of clients taking part in the training process. For Byzantine resilience, Krum [27] computed distance based on Euclidean norm between gradients to filter outliers. Different from Krum [27], coordinate-median and trimmed-mean [28] performed element-wise aggregation rules. After that, Yang et al. [29,30] presented ByRDiE and BRIDGE based on trimmed-mean from centralized to decentralized systems. At every epoch for every regular client, BRIDGE [30] performs the exchange and updates its local model’s coordinates, whereas ByRDiE [29] executes multiple scalar exchanges and updates. Essentially combining Krum [27] and a variant of trimmed-mean [28], Bulyan [31] was proposed to reduce the leeway of Byzantine clients. To provide robustness against a poisoning attack that lowers global accuracy, a lightweight federated multi-task learning framework [32] was proposed. Anomaly detection [33], principal component analysis [34], and so on were applied to identify malicious clients. Instead of considering the relationship between privacy and robustness, these efforts concentrated on privacy protection or Byzantine resilience in FL.

To achieve privacy-preserving and Byzantine-resilience, So et al. [35] proposed a FL algorithm based on secure aggregation. Secret sharing (SS) is used, and any two parties need to negotiate a random number to ensure the security of the information. However, this may require a large number of rounds of interaction, leading to higher communication complexity. Guo et al. [36] proposed an Uniform Byzantine-resilient Aggregation Rule (UBAR) to defeat an arbitrary number of Byzantine nodes. The main idea is to use training samples to test the performance of parameters that are benign clients. This method can converge quickly, but the balance between privacy protection and Byzantine robustness needs to be struck. FoolsGold [37] calculated the cosine similarity between the historical gradients of the clients to mitigate sybils. However, this scheme directly analyzed and calculated the gradients in plain text, which resulted in privacy leakage.

3. Preliminary

In this section, we introduce some techniques involved in this paper. The symbols and notations used in the remainder of this paper are given in

Table 1. Notations.

Notation	Description
C	the number of workers
S	the number of servers
f	the number of malicious workers
T	the number of epochs
g	model update (gradient) 1
x	global model
$g\left[k\right]$	the k-th value of the gradient
r	random vector
$\mathcal{S}$	sign matrix
$\mathcal{D}$	data matrix
$\mu$	mean value
$\sigma$	standard deviation
$\eta$	learning rate

¹ “Global model” and “gradient” are used interchangeably.

.

3.1. Federated Learning

Data are the cornerstone of model training in the field of artificial intelligence. However, data is frequently present in the form of data islands. Data processing in a centralized way is the direct remedy for data islands, but data leakage may occur during the collection and processing stages [38]. To this end, FL is developed and received widespread attention.

FL seeks to train a global collectively on numerous datasets distributed on individual devices without explicitly transferring data samples [39,40]. In IoT networks, several IoT devices can act as workers who interact with a server to train neural networks. The server first initializes a global model. Each worker downloads the most recent model, computes its own model update, and then sends the computed update to the server. Subsequently, the server aggregates all local updates and updates the global model [5]. Local training, aggregation, update, and other steps mentioned above are repeated until the termination condition is met. The preset training epochs or global model accuracy can be considered as termination conditions.

Although the development of FL solves the problem of data islands and reduces the risk of data leakage, many threats and challenges still need to be addressed urgently. The most core problems include the weakness of communication efficiency [41,42,43], the defects of privacy security [44,45,46], the lack of incentive mechanisms [47,48] and so on.

3.2. Privacy Protection Technology

Privacy computing is a set of technologies that can achieve the goal of “usable and invisible” data. Homomorphic encryption (HE), secret sharing (SS), and differential privacy (DP) are presented in this section.

3.2.1. Homomorphic Encryption

HE [49] is a type of encryption that preserves the functionality and format of the encrypted data while allowing a third party to perform some compute operations on the encrypted data. It is usually defined as:

If an encryption scheme satisfies the following equation, then the scheme is called homomorphic with respect to the operation ★:

$$E\left(m_1\right)★E\left(m_2\right)=E(m_1★m_2),\forall m_1,m_2\in M$$

(1)

where E represents the encryption algorithm, and M represents the collection of all information [50].

HE can be used to withstand various security threats, such as membership inference attacks [51], DLG attack [13], chosen plain text attacks [52], etc. Therefore, it has good application prospects in the field of information security.

3.2.2. Secret Sharing

SS is a technique that divides a secret into multiple parts and distributes them to different participants. The secret can only be reconstructed when specific conditions are met [53,54].

Taking the Shamir [55] as an example, it is a classic $(t,n)$ threshold SS method. A secret owner and a group of participants are involved in this method. The owner splits a secret into n shards and transfers them to n participants. The secret can be recovered only by simultaneously obtaining t shards. The specific implementation process of Shamir [55] is as follows:

• Encryption
  Assuming there is a secret S, take any $t-1$ random numbers and construct a polynomial $f\left(x\right)=a_0+a_1·x+a_2·x^2+\dots +a_{(t-1)}·x^{(t-1)}$, where $a_0=S$. It should be noted that all operations are performed in a finite field. Then take any n numbers, substitute $x_1,x_2,\dots ,x_n$ into the polynomial to obtain $f\left(x_1\right),f\left(x_2\right),\dots ,f\left(x_n\right)4$. Send $(x_i,f\left(x_i\right))$ to the participants.
• Decryption
  Polynomial coefficient calculation using secret shards held by any t participants:

  $$\left\{\begin{array}{c}{a}_0+a_1·x_1+a_2·{\left(x_1\right)}^2+\cdots +a_{t-1}·{\left(x_1\right)}^{t-1}=y_1,\\ a_0+a_1·x_2+a_2·{\left(x_2\right)}^2+\cdots +a_{t-1}·{\left(x_2\right)}^{t-1}=y_2,\\ ⋮\\ a_0+a_1·x_3+a_2·{\left(x_t\right)}^2+\cdots +a_{t-1}·{\left(x_t\right)}^{t-1}=y_t.\end{array}\right.$$

  (2)
  After obtaining the coefficients, substitute them into the polynomial, take $x=0$, and you can obtain the secret $S=a_0$.

3.2.3. Differential Privacy

DP [56] protects privacy by merging client data with added noise. This noise makes it difficult for attackers to determine which data belongs to a specific client, thereby increasing the privacy of the data. Local differential privacy (LDP) and global differential privacy (GDP) are variants of DP [57], which are used to protect privacy.

For LDP [58], each worker’s data is added with noise. On the contrary, GDP [59] needs to send all workers’ data to the central server for processing. This makes GDP more suitable for situations where the entire dataset needs to be analyzed. Obviously, the difference between LDP and GDP is that LDP is a method to achieve DP on a single dataset, while GDP is a method to add noise on multiple datasets.

3.3. Outlier Detection Method

Outliers, which may be caused by human errors, mechanical failures, and changes in system behavior, or natural deviations in the environment, can be identified through outlier detection [60], a method that discovers significant deviations from normal or mean values in given data.

3.3.1. $3\sigma$ Criterion

In $3\sigma$ criterion [61], outliers are defined as a collection of measurements that deviate from the mean by more than three times the standard deviation. Under normal distribution, the chance of values appearing beyond $3\sigma$ from the mean is $p\left(\right|x-\mu |>3\sigma )\le 0.003$, which is an extremely unusual event. If the data is not normally distributed, $a·\sigma$ can be used to characterize it. Therefore, Curtis et al. [62] proposed the z-score method, which measures the distance of a certain raw score x from the mean in standard deviation units. That is,

$$z=\frac{x-\mu }{\sigma }$$

(3)

z-score typically considers data points more than three times the standard deviation from the mean as outliers. In other words, data points with z-score $>a$ are considered outliers, where a can be set according to specific circumstances.

3.3.2. DBSCAN

Density-based Spatial Clustering of Applications with Noise (DBSCAN) [63] is a density-based spatial clustering algorithm. This technique finds clusters of any shape in a noisy spatial database and divides areas with sufficient density into one class. The core idea of DBSCAN is to start from a certain core point and continuously expand to a region where the density can reach so as to obtain a maximized region containing core points and boundary points. Any two points in the region are connected in density. Moreover, some studies use DBSCAN to perform outlier filters. The disadvantage is that the application of DBSCAN results in a longer time required for model training.

4. Privacy Robust Aggregation Based on Federated Learning

In this section, we present the system structure and threat model. Moreover, we design an approximate Euclidean distance calculated by the sign matrix and data matrix and propose a Privacy Robust Aggregation Based on Federated Learning (PBA) aggregation rule.

4.1. System Structure and Threat Model

Figure 1 depicts our system model. There are C customers in the overall structure, of which f is Byzantine. The gradients sent by these Byzantine clients may interfere with the convergence and precision of the model. These clients interact with $S(S\ge 2)$ servers. These servers are required to carry out the identification of outliers and global model updates.

On the server side, the threat model is classed as honest-but-curious (semi-honest). This means that servers follow the established protocol without deviating but could try to extract private information from the data transmitted with them. So, if these servers obtain raw gradients, they might employ established methods to extract sensitive client data. Only a fraction of clients may be Byzantine. They may intentionally produce false data in an effort to undermine the training.

4.2. Algorithm Overview

The privacy-robust aggregation rule PBA proposed in this paper is implemented based on S servers, where $S\ge 2$. Suppose that there are n participants in the system, out of which f are Byzantine. The servers in the system are honest and curious (semi-honest) but do not collide with each other. This implies that they follow algorithmic processes, do not disclose each other’s information, but may attempt to access user privacy and restore private data. Similarly, n participants are also honest and curious. However, f malicious participants might upload random data, leading to the deviation of the global model from its standard training process. Figure 1 illustrates the overall framework of PBA, where $g_{i}j$ represents the gradient information sent by participant $C_i$ to the server $S_j$.

During the model initialization stage, S servers negotiate to create $\frac{S(S-1)}{2}$ random vectors. These vectors need to satisfy the following equation:

$$\sum _{i=1}^{\frac{S(S-1)}{2}}(S-i+1)·r_i=c$$

(4)

where $r_i$ represents the i-th vector generated by servers and c is a constant. Then, each client downloads the most recent global model and computes local updates. Before uploading the updates to the servers, the following operations need to be performed:

• Differential Privacy and Denoise
  To prevent the data leakage in servers, client $C_i$ creates private noise and adds it to the local gradient $g_i$, i.e., ${\tilde{g}}_i\leftarrow g_i+\mathcal{N}(0,{▵}^2·{\sigma }^2)$. But noises added to the gradients have an impact on model convergence and accuracy. Hence, It is necessary to denoise the gradients after adding noises. Kolmogorov–Smirnov (KS) distance is used as a measure of denoising [64]. $C_i$ applies it to denoise the gradient. That is, ${\widehat{g}}_i\leftarrow KS({\tilde{g}}_i,\mathcal{N})$, where $KS\left(\right)$ represents the KS measure.
  The above operations can not only preserve the characteristics of DP to reduce the risk of data leakage but also eliminate the impact of noise on the model convergence and accuracy [64].
• Gradient Encoding
  Each client needs to perform some mask operations to provide further protection. The specific masking process is shown below:

  $${\widehat{g}}_{ij}={\widehat{g}}_i+\sum _{p:p\le j}{r}_p+\sum _{p:p\ge j}{r}_p$$

  (5)

  where ${\widehat{g}}_{ij}$ represents the encoded gradient that $C_i$ sends to the server $S_j$.

Above all, $C_i$ generates the gradient ${\widehat{g}}_{ij}$ based on its own local update $g_i$, which undergoes DP, denoise, and encoding operations and sends to the server $S_j$. Then, when $S_j$ receives the gradient information, it needs to calculate the approximate Euclidean distance between two gradients, perform outlier detection through $3\sigma$ criterion, and combine gradients.

After receiving C encoded gradient information, $S_j$ calculates the approximate Euclidean distance. And the approximate distance between $C_i$ and $C_j$ is temporarily recorded as $AED(i,j)$. Then, $S_j$ detects outliers in the approximate distances based on $3\sigma$ criterion. In the application of PBA, outliers can be defined as values with deviations from the mean exceeding a times, where a is not necessarily three. Thirdly, $S_j$ calculates the sum of all gradients based on the filtration results, i.e.,

$$g_{S_j}=\sum _{g_i\in N_{S_i}}{g}_i+n·(\sum _{p:p\le j}{r}_p-\sum _{q:q>j}{r}_q)$$

(6)

The computation result $g_{S_j}$ is sent to other servers while $S_j$ receives corresponding results from them. $S_j$ combines gradients as follows:

$$g=\frac{1}{S}\sum g_{S_j}-(n-f)·c$$

(7)

Finally, each server completes this epoch by updating the global model. During the initialization phase, S servers negotiate and generate random vectors $r_i$. Then, The process of PBA is shown in Algorithm 1, where $f_{AED}(·)$ represents the function that calculates the approximate Euclidean distance.

Algorithm 1 PBA.

Input:   The global model in the t-th round, $x_t$; Output:   The gradient aggregated in the t-th round, g;   1: for client $C_i,i=1,2,\dots ,C$: do   2:    randomly sample from the local dataset for local training $g_i=▽f(x_t,{\xi }_i)$;   3:    add private noise ${\tilde{g}}_i\leftarrow g_i+\mathcal{N}(0,{▵}^2{\sigma }^2)$;   4:    denoise ${\widehat{g}}_i\leftarrow KS({\tilde{g}}_i,\mathcal{N})$;   5:    encode ${\widehat{g}}_{ij}={\widehat{g}}_i+{\sum }_{p:p\le j}{r}_p+{\sum }_{p:p>j}{r}_p$;   6:    send ${\widehat{g}}_{ij}$ to the server $S_j$;   7: end for   8: for server $S_j,j=1,2,\dots ,S$: do   9:    calculate approximate Euclidean distance $AED(i,j)=f_{AED}({\widehat{g}}_{pj},{\widehat{g}}_{qj})$; 10:    filter out outliers based on the $3\sigma$ criterion, denoted as $N_{S_i}$; 11:    calculate the sum of gradients $g_{S_i}={\sum }_{g_i\in N_{S_j}}+n·({\sum }_{p:p\le i}{r}_p+{\sum }_{p:p>i}{r}_p)$; 12:    send the $g_{S_j}$ to other servers and receive gradients from them; 13:    compute the average $g=\frac{1}{S}·\sum g_{S_j}-(n-f)·c$; 14: end for 15: return g

4.3. Approximate Euclidean Distance

Due to the characteristics of data dispersion and multi-device collaborative training, Training Time Attack (TTA), including Model Poisoning Attack and Data Poisoning Attack, is a significant threat to the security of FL system [65]. Studies explored robust aggregation techniques, such as Krum [27] and Median [28], to handle devices that send corrupted updates to the server. Wang et al. [66] proposed that the cosine value between any two gradients is only related to their direction, not magnitude. Therefore, they replace the gradient with its sign rather than specific values, represented as a binary sequence, to highlight the direction differences between any two gradients. However, this method ignores the impact of gradient sizes on model convergence. In distance-based robust aggregation methods, Euclidean distance is the most typical distance measure. It is computed as follows:

$$distance(i,j)=\sqrt{\sum {(g_i\left[k\right]-g_j\left[k\right])}^2}$$

(8)

where $g_i\left[k\right]$ and $g_j\left[k\right]$ denote the k-th bit of local training gradients $g_i$ and $g_j$, respectively. However, distance-based robust aggregation methods could incur significant computational overhead with numerous clients in the system [9,67].

A method which transfers only gradient signs and computes cosine similarity through the binary sequence is proposed, which greatly reduces the communication overhead of FL system [66]. Based on this method, an approximate Euclidean distance based on binary sequences is designed. Let $g_i$ and $g_j$ denote the gradients of $C_i$ and $C_j$, respectively, and g denote their average gradient. The approximate distance between them is computed as follows:

• Servers form sign matrices and data matrices by taking the sign and absolute value of $g_i$ and $g_j$, respectively. The sign matrices are denoted as ${\mathcal{S}}_i$ and ${\mathcal{S}}_j$, while the data matrices are denoted as ${\mathcal{D}}_i$ and ${\mathcal{D}}_j$.
• An XOR operation between the two sign matrices is performed, i.e., $sign(i,j)={\mathcal{S}}_i⨁{\mathcal{S}}_i$. At the same time, servers need to subtract the two data matrices element-wise, take the absolute value, and compare it to g. Specifically,

  $$data(i,j)=\left\{\begin{array}{cc}data(i,j)\left[k\right]=0,& data(i,j)\left[k\right]<g\left[k\right];\\ data(i,j)\left[k\right]=1,& data(i,j)\left[k\right]\ge g\left[k\right];\end{array}\right.$$

  (9)

  where $data(i,j)\left[k\right]=|{\mathcal{D}}_i\left[k\right]-{\mathcal{D}}_j\left[k\right]|$.
• $sign(i,j)$ and $data(i,j)$ are combined, resulting in four different binary sequences: 00, 01, 10, and 11, which are represented as 0, 1, 2, and 3 in decimal form to reflect the degree of maliciousness of the gradients under different scenarios. Specifically,

  $$AED(i,j)=\left\{\begin{array}{cc}0,& (sign,data)=00\\ 1,& (sign,data)=01\\ 2,& (sign,data)=10\\ 3,& (sign,data)=11\end{array}\right.$$

  (10)

The four binary sequences mentioned above represent four scenarios: same-sign with a small value difference, same-sign with a large value difference, opposite-sign with a small value difference, and opposite-sign with a large value difference.

4.4. Theoretical Analysis

4.4.1. Correctness Analysis

PBA first performs DP for local updates and then reduces the noise according to the added noise in DP. In addition, due to the post-processing property of DP, the execution of denoising does not affect the differential property and can still maintain the model accuracy losslessly [64,66]. Therefore, privacy security can still be guaranteed. We provide the correctness analysis and proof for aggregation based on gradient encoding.

Firstly, the correctness of distance computed on encoded gradients is analyzed. For $S_i$, the gradient received from clients can be represented as ${\widehat{g}}_{xi}={\widehat{g}}_x+{\sum }_{p:p\le i}{r}_p+{\sum }_{p:p>i}{r}_p,x\in \left[n\right]$. Therefore, the distance of the gradients between any two clients $C_x$ and $C_y$ can be expressed as

$$\begin{array}{cc}\hfill distance(i,j)& ={({\widehat{g}}_{xi}-{\widehat{g}}_{yi})}^2\hfill \\ \hfill & ={[({\widehat{g}}_x+\sum _{p:p\le i}{r}_p+\sum _{p:p>i}{r}_p)-({\widehat{g}}_y+\sum _{p:p\le i}{r}_p+\sum _{p:p>i}{r}_p)]}^2\hfill \\ \hfill & ={({\widehat{g}}_x-{\widehat{g}}_y)}^2\hfill \end{array}$$

(11)

It can be seen that the result computed on the server $S_i$ is consistent with the result whose gradients are not encoded.

Then, a correctness proof of aggregation is provided. Suppose among n clients, f malicious clients are selected. The gradient sum calculated by server $S_i$ is represented as $g_{S_j}={\sum }_{g_i\in N_{S_i}}{g}_i+n·({\sum }_{p:p\le j}{r}_p-{\sum }_{q:q>j}{r}_q)$. Then,

$$\begin{array}{cc}\hfill \sum g_{S_i}& =\sum [\sum _{g_i\in N_{S_i}}{g}_i+n·(\sum _{p:p\le j}{r}_p-\sum _{q:q>j}{r}_q)]\hfill \\ \hfill & =S·\sum _{g_i\in N}{g}_i+(n-f)·[S·r_1+\cdots +(S-i+1)·r_i+\cdots +r_S]\hfill \\ \hfill & =S·[\sum _{g_i\in N}{g}_i+(n-f)·c]\hfill \end{array}$$

(12)

Thus, the gradient for the model update can be computed as follows:

$$g=\frac{1}{S}·\sum g_{S_j}-(n-f)·c=\sum _{g_i\in N}{g}_i$$

(13)

where N refers to the set of potential benign gradients.

4.4.2. Complexity Analysis

PBA is based on approximate Euclidean distance, which can reduce computational overheads. Hence, we analyze the computational complexity of approximate Euclidean distance and compare it with Euclidean distance.

Proposition 1.

Let n represent the dimension of a model update, k represent the bits of each dimension. Suppose that the time required to perform a one-bit XOR is unit time, the time required for addition and subtraction is t times the unit time, and the time required for multiplication and division is t times the unit time. Then, the computational complexity of approximate Euclidean distance is

$$O=(2·k+1)·n$$

(14)

Proof. 

For approximate Euclidean distance, it needs to perform an XOR operation between two sign matrices, which requires n times unit time. Moreover, subtraction is performed between data matrices, and the required time is $n·k$ times unit time. Finally, a comparison operation spends $n·k$ times unit time. Hence, the time spent computing an approximate Euclidean distance is $O=(2·k+1)·n$. □

For Euclidean distance, the calculation method is shown in Formula (8), including subtraction, multiplication, addition, and square root operations. From this, it can be concluded that the computational complexity of Euclidean distance is $O=2·(t+2)·n·k$. Then, after comparison, the computational complexity of approximate Euclidean distance is about $\frac{1}{(t+1)·k}$ times that of Euclidean distance.

5. Performance Evaluation

5.1. Experiment Setup

All trials are conducted on a computer equipped with an AMD Ryzen 7 5800H, Radeon Graphics 3.20 GHz, and NVIDIA GeForce GTX 1650. And PyTorch [68] is used for training in Python. In addition, a task for object recognition is developed that involves n clients working together to train a convolutional neural network (CNN) with two convolutional layers and two fully connected layers.

Assuming there are S samples and L categories in the dataset, two methods are studied for dividing the dataset among the clients:

• Independent Identically Distribution (IID)
  In this method, data is shuffled and split into $\frac{S}{C}$ samples per client.
• Non-Independent Identically Distribution (Non-IID)
  In this method, the data set is sorted and divided into C parts of size $\frac{S}{C}$, assigning one part to each client. Non-IID data will be used to study the privacy security of PBA in order to reflect the leakage of specific sensitive information from a client.
  In the experiment, the MNIST [69] image dataset is used to train a CNN model, which is a handwritten digit (0–9) dataset consisting of 60,000 training samples and 10,000 test samples, with each sample being a 28 × 28 grayscale image

Moreover, in order to evaluate the Byzantine robustness, several Byzantine attacks are considered in the experiment:

• Gaussian Attack (GA)
  This is an aimless poisoning attack aimed at reducing the accuracy of the model. Specifically, malicious clients can randomly sample from the Gaussian distribution and upload it as local update parameters, namely:

  $$\widehat{g}\left[d\right]=Gaussian(\mu ,{\sigma }^2)$$

  (15)
  Among them, $\mu$ represents the mathematical expectation of a Gaussian distribution, and ${\sigma }^2$ represents its variance.
• Label Flipping Attack (LFA)
  This is a targeted poisoning attack. Malicious clients flip the labels of local data to generate error gradients [66], especially flipping the labels of each sample from x to $L-1-x,x\in (0,\dots ,L-1)$.

Furthermore, some baselines are compared with PBA, such as Krum [27], trimmed-mean [28] and UBAR [36]. And following performance metrics will be considered:

• Global Accuracy: Test the accuracy of the trained model on the validation set;
• Loss Value: Test the cross-entropy of the training model on the validation machine;
• Reconstructed Images: Image reconstruction under DLG attack.

A total of $C=20$ clients are set up, with f Byzantine clients. We compare the global accuracy and loss of the model under different numbers of malicious clients. When evaluating the security of PBA, the DLG attack will be conducted to see if user-sensitive information in images can be reconstructed.

5.2. Performance Analysis

Through a series of experiments, a comparison and analysis of the security and robustness of PBA are conducted. The model performance of PBA is evaluated concerning ensuring security and robustness.

5.2.1. Security Evaluation

To evaluate the security of PBA, the DLG attack is implemented, and LeNet5 is used as the generator and model training network. Moreover, we take the MNIST dataset sample “7” as an example to observe whether the reconstructed image can be recognized. In this part, we simulate malicious attackers intercepting the model updates of the parties participating in the training and attempting to recover sensitive information. We take the FL method without any protection as the baseline and compare the reconstructed images of these two methods. Each attack result displays the reconstruction results of every ten rounds within 500 rounds.

Firstly, the DLG attack is conducted on the baseline, which is not perturbed. Although there is still some noise in the data sample in the 10th round, as shown in Figure 2a, it could already be recognized by the human eye. Hence, it can be indicated that information leakage occurs in this method. Subsequently, we conduct the DLG attack on PBA. When the gradient is perturbed in PBA, the DLG attack cannot reconstruct the user’s sensitive image, as shown in Figure 2b.

Moreover, the DLG attack is conducted on different degrees of perturbation, and the attack results are all unable to reconstruct the image. That is, the attack failed. The experimental results show that PBA is effective in resisting DLG attacks and can achieve the goal of protecting user privacy.

5.2.2. Robustness Evaluation

In the evaluation of robustness, we take the FL method without any protection as a baseline method to compare with PBA, Krum [27], trimmed-mean [28], and UBAR [36]. We, respectively, set Byzantine clients accounting for 0%, 10%, and 30% of the total, comparing, and analyzing global accuracy changes under Gaussian attacks and label flipping attacks.

To begin with, the performance without malicious clients and attacks is tested and compared with the presence of Byzantine clients. The results as shown in Figure 3. Compared with the baseline, the accuracy of PBA is close to other methods, about 88%. In the same way, the loss of PBA, about 0.4, is lower or similar to that of other methods. Moreover, the convergence speed of PBA is almost the same as the baseline.

After that, the performance of PBA is compared with that of Krum [27], trimmed-mean [28], and UBAR [36] with the different number of Byzantine clients. Faced with the same attack, the degree to which model performance is affected by Byzantine clients is related to the number of malicious clients in the system.

We take GA to simulate a fraction of clients uploading bad data. From Figure 4, the model accuracy of all methods with $f=6$ Byzantine clients in the system is lower than that with $f=2$. When there are few malicious clients, the global accuracy of PBA is better than Krum [27] and UBAR [36], almost equal to the accuracy without Byzantine clients. But when malicious clients account for 30%, the performance of PBA and trimmed-mean [28] decreases significantly. On the contrary, the performance of UBAR [36] and Krum [27] can maintain more stability. Moreover, we conduct LFA to evaluate the model performance. As shown in Figure 5, although LFA is more severe than GA, the model performance of all methods is almost not affected with few Byzantine clients in the system, the accuracy of about 85%. However, if there are 60% malicious clients in the system, the performance of methods, except UBAR [36], is reduced remarkably. Krum [27] and trimmed-mean [28] can not achieve convergence. This means Byzantine clients reach the goal of disrupting the model. Although PBA and UBAR [36] can achieve convergence, their accuracy is influenced significantly. Hence, PBA is unsuitable for the system with many Byzantine clients.

Based on the above experiments, the performance of PBA is close to the baseline without Byzantine clients and attacks. However, if there are many malicious clients in the system and the attacks are strong, the performance of PBA is greatly compromised. Although PBA did not always perform well in experiments, it could reduce the time cost compared to other methods.

5.2.3. Cost Evaluation

Wang et al. [66] proposed a method called Brief that calculates cosine similarity using binary operations and then implements B2A to calculate the Euclidean distance further. Similarly, PBA uses binary sequences to calculate the Euclidean distance, somewhat reducing the time cost. Hence, we measure the time spent by different methods in an epoch. And the presentation is the average time required for five epochs. The results are shown in

Table 2. Running time of different methods.

Method	Time
Krum [27]	10.640 s
trimmed-mean [28]	17.588 s
UBAR [36]	23.030 s
Brief [66]	56.659 s
PBA	9.391 s

.

Because of B2A and clustering used in Brief [66], it needs a long time to finish a training epoch, about 57 s. Moreover, to achieve high accuracy, UBAR [36] reuses training samples to test the performance, which takes some time. The time spent by other distance-based methods is not significantly different. For PBA, it is based on approximate Euclidean distance and $3\sigma$ criterion. PBA, computing approximate Euclidean distance by binary sequences, has lower complexity than other distance-based methods. Compared with the clustering method in Brief [66], the $3\sigma$ criterion with lower complexity is scientific.

6. Conclusions

In order to solve the issue that previous federated learning methods cannot quickly execute outlier detection, this paper proposes a method to estimate the Euclidean distance using binary sequences that reflects the differences between gradients. Based on this approach, PBA, a $3\sigma$ criterion-based federated learning security robust method, is proposed. Moreover, this paper analyzes the correctness and complexity of PBA and evaluates the performance of PBA on open datasets. The results show that its privacy security can be guaranteed and Byzantine robustness can be achieved in some conditions. PBA offers good convergence performance and somewhat lowers the computational cost as compared to other approaches. However, faced with severe attacks or many Byzantine clients in the system, PBA can maintain stable performance.

We may try to deploy this method to multi-server scenarios that require a shorter time to train the FL model. In addition, in the future, we hope to further study how to better integrate FL with the IoT, which can protect privacy, achieve Byzantine resilience, and be more lightweight.