Privacy-Preserving Outsourced Artificial Neural Network Training for Secure Image Classification

Abstract

Artificial neural network (ANN) is powerful in the artificial intelligence field and has been successfully applied to interpret complex image data in the real world. Since the majority of images are commonly known as private with the information intended to be used by the owner, such as handwritten characters and face, the private constraints form a major obstacle in developing high-precision image classifiers which require access to a large amount of image data belonging to multiple users. State-of-the-art privacy-preserving ANN schemes often use full homomorphic encryption which result in a substantial overhead of computation and data traffic for the data owners, and are restricted to approximation models by low-degree polynomials which lead to a large accuracy loss of the trained model compared to the original ANN model in the plain domain. Consequently, it is still a huge challenge to train an ANN model in the encrypted-domain. To mitigate this problem, we propose a privacy-preserving ANN system for secure constructing image classifiers, named IPPNN, where the server is able to train an ANN-based classifier on the combined image data of all data owners without being able to observe any images using primitives, such as randomization and functional encryption. Our system achieves faster training time and supports lossless training. Moreover, IPPNN removes the need for multiple communications among data owners and servers. We analyze the security of the protocol and perform experiments on a large scale image recognition task. The results show that the IPPNN is feasible to use in practice while achieving high accuracy.

1. Introduction

Artificial Neural Network (ANN) learning is well-suited to tackle problems in which the training samples correspond to complex image data, such as inputs from video cameras and mobile phones [1]. In recent work, ANN has proven successful in text and image classification and is observed to achieve high accuracy in these tasks. Consequently, image classifier based on ANN has been widely used in many real-life fields [2,3,4]. Conceptually, ANN-based solutions rely on elaborate algorithms, but even more on large-scale training data. As the volume of data owned by a single user is insufficient to train reliable classifiers, aggregation of data is a solution to address the local limitations.

However, images are private mediums with the message intended to be used only by the owner. Due to the sensitive nature of the information content, there might be confidentiality and security constraints against sharing and using image data. These constraints form formidable obstacles in many image applications, including image classification [5], image synthesis [6], image retrieval [7,8], image sharing [9], and protection of location privacy in camera data of auto-driving vehicles [10]. As a consequence, the construction of privacy-preserving image classifiers is worth researching.

Existing approaches to train machine learning models in privacy-preserving settings mainly rely on secure multi-party computation [11], fully or partially homomorphic encryption [12,13,14,15] and secret sharing [16,17]. Most of these approaches have several limitations: (i) First, in traditional privacy-preserving machine learning schemes, the non-linear functions are hard to support using common cryptographic techniques, such as homomorphic encryption and boolean circuit. Hence, prior work proposes to approximate the non-linear function using polynomial or piecewise linear function [12,15,16,17,18]. It can be shown that approximation using a high-degree polynomial is able to provide a higher accuracy. While—to the best of our knowledge—for efficiency reasons, the degree of the approximation polynomial is limited to 5 or less, which results in a large accuracy loss of the trained model. (ii) Secondly, the majority of prior work requires multi-round communications between data users and servers [19,20,21,22,23,24,25,26], making a substantial overhead of computation and transmission cost. Furthermore, in practice, most of the users are unlikely to participate in the whole training process online. Consequently, the classifier needs to be learned only on encrypted image data without the help of users. Moreover, non-interactive setting is easy to extend to dynamical participation or dropping during the training phase.

The first known privacy-preserving scheme for neural network with non-interactive and lossless features simultaneously presented by [11], named NPMML. This method uses hybrid cryptographic techniques, including Paillier cryptosystem, RSA, and random mask to preserve data. Since the implementation of the NPMML utilizes multiple cryptosystems, the two servers need to interact several rounds for updating the parameters to accommodate the transformation of different masked techniques.

Towards this, we propose a solution that enables multiple users to share their private data to achieve a high-precision image classifier without violating privacy laws and communicating with the servers frequently. Specifically, we present a non-interactive and privacy-preserving image classifier based on an ANN model, named IPPNN. This scheme (i) substantially reduces the amount of communication required to train ANN model, which requires only one transmission for users; and (ii) does not require any approximation for non-linear functions and can support lossless training. To achieve these benefits, IPPNN orchestrates mask matrix and functional encryption for inner-product (FEIP) [27].

To summarize, our contributions are:

• Non-Interactive. We propose IPPNN, a non-interactive and privacy-preserving image classifier based on ANN over distributed image data, which only requires communication between the users and the servers as a one-way interaction. Compared to interactive privacy-preserving ANN systems, our approach introduces less computation and communication overheads for the users.
• High-Precision.IPPNN enables the building of high-precision image classification as it does not require the use of polynomials or linear piecewise function approximations to handle out the non-linear functions. IPPNN supports lossless classifier construction which does not alter any of the computation of the original ANN training algorithm.
• Efficiency. We have implemented and evaluated the performance of IPPNN. For two popular MNIST datasets with a large amount of image data and high dimensionality, the experiments validate that IPPNN is several orders of magnitude faster than the state-of-the-art privacy-preserving ANN schemes without compromising the accuracy of classifiers.

The rest of paper is organized as follows. We review the related work in Section 2. In Section 3, we introduce ANN model and cryptographic primitive (functional encryption for inner-product). We overview the proposed privacy-preserving framework of IPPNN in the first part of Section 4, and the core idea and construction details of IPPNN is described in the latter part of Section 4. The security analysis and evaluation are presented in Section 5. The conclusion of our paper are presented in Section 6.

2. Related Work

Image processing is a well established field and has been researched extensively [4,28]. As the artificial intelligence (AI) and big data develop by leaps and bounds, the image-processing technology, such as image recognition [29], image classification [30], face recognition system [31], smart agriculture [3], lung cancer classification and prediction [2], surface defect detection system [32], are also renewing, people lead a more convenient and have a higher quality of life.

However, the application of AI techniques can cause privacy leakage and security risks since solutions based on AI are reliant on a large number of training samples commonly belonging to different users. To make effective and safe use of data distributed in different locations while satisfying privacy requirements, some approaches have incorporated privacy-preserving techniques into image processing [5,6,7,8,9,14,33]. For example, Fagbohungbe et al. [5] proposed a secure intelligent computing framework for image classification based on deep learning and edge computing. Yang et al. [6] presented a lightweight secure GAN framework for image synthesis based on secret sharing. Shen et al. [7] proposed a privacy-preserving medical image retrieval system based on blockchain.

Multiple privacy-preserving machine learning approaches have been presented for classification tasks. Prior work [19,20,21,22,23,24,25,26] proposed schemes with an interactive fashion which requires users to communicate with servers multiple rounds. The online computation resources constraints form an obstacle in developing based-AI classification systems. Although non-interactive approaches have been proposed in the last few years, most of fall into fully homomorphic encryption (FHE) [12,15] or secret sharing (SS) [16,17] algorithms. Protocols based-FHE are hardly to be applied in the real world due to a very considerable computation costs. Methods based on SS still have higher requirements for the network since the two servers need to communicate frequently. Our proposed privacy-preserving method for constructing image classifiers realizes non-interactive between users and servers. Moreover, the protocol is feasible and efficient to use in practice using functional encryption cryptographic scheme.

Due to the limitation of FHE and SS cryptographic primitive, some privacy computation schemes are forced to use polynomials or piecewise linear functions to appropriate original non-linear functions [15,16,17,18,26,34]. For support vector machines and logistic regression, the obtained model precision is acceptable. However, for neural network or deep learning, such approximation methods will cause a loss of precision, which is not comparable to the original model and can not be applied in production and life. Our method removes the need for the approximation functions. That is, the private training protocol does not alter the original ANN model and result in a loss of accuracy.

The closest approach to our IPPNN is the NPMML method [11], which is a non-interactive privacy-preserving multi-party neural network framework with a dual server architecture. The protocol is complicated for the reason that NPMML employs multiple cryptographic technologies to preserve data. The benefit of non-interactive and lossless features (quite similar to ours) comes with a substantial overhead of computation and communication costs. The detailed comparison of the IPPNN and the NPMML are covered in Section 5.6 and Section 5.7.

3. Preliminaries

3.1. Artificial Neural Network

Artificial neural network [35] provides a general method for learning non-linear functions and constructing prediction model from training samples. The most common architecture of ANN is composed of the input layer, multiple hidden layers, and the output layer. ANN models have proven successful in many practical problems, such as learning to recognize handwritten characters. Figure 1 shows an ANN model for image recognition.

Notation. Here, we suppose that the input data $\mathit{x}=(x_1;\dots ;x_a)\in {\mathbb{R}}^{a\times 1}$ has a different features and each sample belongs to one of the c classes denoted by a label $\mathit{y}=(y_1,\dots ,y_c)$, the hidden layer has b neurons.

We represent the hidden layer weight matrix as $W^{\left(h\right)}\in {\mathbb{R}}^{b\times a}$. Each row of $W^{\left(h\right)}$ is a list of real numbers, which identifies a vector ${\mathit{w}}_j^{(h)}=(w_{j1}^{(h)},\dots ,w_{ja}^{(h)})\in {\mathbb{R}}^{1\times a},j=1,\dots ,b$. Similarly, we denote the matrix $W^{\left(o\right)}\in {\mathbb{R}}^{c\times b}$ as an output layer weight matrix. Each row of $W^{\left(o\right)}$ identifies a vector ${\mathit{w}}_i^{\left(o\right)}=(w_{i1}^{\left(o\right)},\dots ,w_{ib}^{\left(o\right)})\in {\mathbb{R}}^{1\times b},i=1,\dots ,c$.

We call the last layer as the output layer

$$\begin{array}{c}\hfill \mathit{o}=(o_1;\dots ;o_c)=g\left(W^{\left(o\right)}\mathit{h}\right)\in {\mathbb{R}}^{c\times 1},\end{array}$$

where $\mathit{h}=(h_1;\dots ;h_b)=f\left(W^{\left(h\right)}\mathit{x}\right)\in {\mathbb{R}}^{b\times 1}$ consists of all outputs of hidden layer. In specific, $f\left({\mathit{w}}^{\left(h\right)}\mathit{x}\right)\in \mathbb{R}$ is a sigmoid function $f\left(z\right)=\frac{1}{1+e^{-z}}$, where ${\mathit{w}}^{\left(h\right)}$ is a hidden layer weight vector. The function g in ANN can be a softmax function: $g\left(z_k\right)=e^{z_k}/{\sum }_i{e}^{z_i}$. Note that in our privacy-preserving image classifier construction scheme based on ANN any activation function can substitute sigmoid function f or softmax function g.

Gradient descent algorithm. Considering cross entropy as the loss function,

$$\begin{array}{c}\hfill E\left(w\right)=-{\displaystyle \sum _{i=1}^c}{y}_{i}ln\left(o_i\right),\end{array}$$

(1)

where $y_i$ is the class label, $o_i$ is the output of prediction function, w denotes the model parameter.

Gradient descent search aims to determine a weight $w^{*}$ that minimizes $E\left(w\right)$ by starting with an arbitrary initial weight. Then, the weight is altered in the direction that produces the steepest descent. The training rule for gradient descent is

$$\begin{array}{c}\hfill w_{t+1}\leftarrow w_t-\eta \nabla E\left(w_t\right)\end{array}$$

(2)

where $\eta$ is the learning rate, and $\nabla E\left(w_t\right)$ is the gradient computed at the tth iteration. This process continues until the global minimum is reached.

Derivation of the gradient descent rule. By selecting a mini-batch sample set S randomly from the whole database $\mathcal{DB}=\{({\mathit{x}}^{\left(1\right)},{\mathit{y}}^{\left(1\right)}),\dots ,({\mathit{x}}^{\left(m\right)},{\mathit{y}}^{\left(m\right)})\}$, the gradient with respect to the output layer weight $w_{i,j}^{\left(o\right)}$ in a 3-layer-perception becomes

$$\begin{array}{c}\hfill \frac{\partial E_S}{\partial w_{i,j}^{\left(o\right)}}=\frac{1}{\left|S\right|}\sum _{l\in S}(o_i^{\left(l\right)}-y_i^{\left(l\right)})h_j^{\left(l\right)},\end{array}$$

(3)

where $i=1,\dots ,c,j=1,\dots ,b.$

Similarly, we update the hidden layer weight $w_{j,k}^{\left(h\right)}$ via the formula as follows

$$\begin{array}{c}\hfill \frac{\partial E_S}{\partial w_{j,k}^{\left(h\right)}}=\frac{1}{\left|S\right|}\sum _{l\in S}-h_j^{\left(l\right)}(1-h_j^{\left(l\right)}){\displaystyle \sum _{i=1}^c}[(y_i^{\left(l\right)}-o_i^{\left(l\right)})w_{i,j}^{\left(o\right)}]x_k^{\left(l\right)},\end{array}$$

(4)

where $j=1,\dots ,b,k=1,\dots ,a.$

3.2. Functional Encryption for Inner-Product (FEIP)

Functional encryption (FE) is a new paradigm of the public key that allows one party $P_1$ to learn a function of ciphertext encrypted by another party $P_2$. In this paper, we employ a functional encryption for inner product (FEIP) scheme [27], which allows $P_1$ to compute the inner product between vectors $\mathit{x}$ and $\mathit{y}$, where $\mathit{x}$ is encrypted by $P_2$ and $\mathit{y}$ is a plaintext vector given by $P_1$. A FEIP scheme consists of five algorithms. More specific details of FEIP scheme refers to [27].

• Setup: The setup algorithm takes a security parameter as input and generates a mathematical group.
• Master key generation: The key generation algorithm creates a public key $p{k}^{FEIP}$ together with a master secret key $ms{k}^{FEIP}$.
• Functional key derivation: The functional key derivation algorithm takes the master secret $ms{k}^{FEIP}$ and the vector $\mathit{y}$ as input to generate a functionally derived key $d{k}_y$.
• Encryption: The encryption algorithm applies to message vector $\mathit{x}$ and obtains a ciphertext $c{t}_x$ by using the public key $p{k}^{FEIP}$.
• Decryption: Given the ciphertext $c{t}_x$ of a message $\mathit{x}$, the holder of the key $d{k}_y$ is able to compute the value of $〈\mathit{x},\mathit{y}〉$ using the decryption algorithm.

Consider the ANN training scenarios. The computation of inner product of weight vector $\mathit{w}$ and training sample $\mathit{x}$ is a formidable step during the ANN training in the encrypted domain. A user wants to keep his sample $\mathit{x}$ private but lets a server be able to compute the inner product $〈\mathit{w},\mathit{x}〉$. During the Setup phase, a trusted third-party authority ($\mathcal{TPA}$) provides the public key to the user. Then, the user encrypts $\mathit{x}$ and transfers the ciphertext to the server. In order to acquire $〈\mathit{w},\mathit{x}〉$ for the server, the $\mathcal{TPA}$ generates a functionally derived key that depends on the weight $\mathit{w}$. The server decrypts the ciphertext using the functionally derived key and obtains the result $〈\mathit{w},\mathit{x}〉$. In this way, the server is able to perform the inner product operation while maintaining the privacy of the user.

4. The Framework and Construction for Privacy-Preserving Image Classifier

We first introduce the framework of our proposed scheme, IPPNN, which is shown in Figure 2. The goal is to build a high-precision image classifier using neural network model without revealing training samples belonging to the users. Additionally, IPPNN enables cooperative learning without a need for any peer-to-peer communication among users and servers.

4.1. System Model

IPPNN has four types of entities: a group of users, a neural network training server (${\mathcal{S}}_{\mathcal{NN}}$), an auxiliary server (${\mathcal{S}}_{\mathcal{AU}}$) which does not collude with ${\mathcal{S}}_{\mathcal{NN}}$, and a trusted third-party authority ($\mathcal{TPA}$) to enable functional encryption.

• Users: Each user owns a dataset which contains multiple training samples and is willing to contribute his samples in a privacy-preserving way. In our system, the responsibility of a user is only to provide his data in well-designed encrypted version to the servers ${\mathcal{S}}_{\mathcal{AU}}$ and ${\mathcal{S}}_{\mathcal{NN}}$.
• ${\mathcal{S}}_{\mathcal{NN}}$: A neural network training server to collaboratively build a global neural network model for image classification without model parameters information leakage. The server ${\mathcal{S}}_{\mathcal{NN}}$ orchestrates the private training process with the help of the server ${\mathcal{S}}_{\mathcal{AU}}$. Finally, ${\mathcal{S}}_{\mathcal{NN}}$ is the owner of the image classifier.
• ${\mathcal{S}}_{\mathcal{AU}}$: An auxiliary server to help the trainer server ${\mathcal{S}}_{\mathcal{NN}}$ to complete a privacy-preserving neural network training. It does not possess any model updates parameters or optimized model weights.
• $\mathcal{TPA}$: A trusted third-party authority $\mathcal{TPA}$ is responsible for building the underlying cryptosystem, delivering the public key to each user and providing private key service to the server ${\mathcal{S}}_{\mathcal{AU}}$.

4.2. Security Requirement

The main goal of IPPNN is to build an image classifier by training a neural network model: (i) without being able to observe any information belonging to the users; and (ii) without leaking the model parameters trained by the server ${\mathcal{S}}_{\mathcal{NN}}$.

• Honest-but-curious ${\mathcal{S}}_{\mathcal{NN}}$: We assume that the server ${\mathcal{S}}_{\mathcal{NN}}$ correctly follows the IPPNN protocol, but may try to learn private information belonging to users from the processed information provided by ${\mathcal{S}}_{\mathcal{AU}}$.
• Honest-but-curious ${\mathcal{S}}_{\mathcal{AU}}$: We assume that the server ${\mathcal{S}}_{\mathcal{AU}}$ correctly follows the IPPNN protocol, but may try to learn private training samples from the encrypted information provided by data owners, and try to learn the trained model parameters.
• Trusted $\mathcal{TPA}$: We assume that the $\mathcal{TPA}$ is an independent entity trusted by the users and the servers ${\mathcal{S}}_{\mathcal{NN}}$ and ${\mathcal{S}}_{\mathcal{AU}}$.

We now present our privacy-preserving scheme for achieving an image classifier. We first introduce the idea of IPPNN model. Then, the data encryption algorithm and a detailed secure training process are presented.

4.3. Core Idea of the IPPNN

The main challenge in privacy-preserving machine learning is to compute the non-linear functions without at the expense of user privacy, which is hard to support using the existing cryptographic technologies, including homomorphic encryption, Boolean circuit, and secret sharing. Hence, prior work proposes to replace the non-linear functions by approximation polynomials or piece-wise linear functions. These substitutions can be computed using secure computation techniques efficiently.

Our IPPNN removes the need for the use of polynomial approximation by using FEIP scheme, and removes the multi-round communications between the users and servers by using mask matrix. Consequently, the private training protocol does not alter any of the computation of the original ANN training algorithm and, therefore, result in the same output, i.e., there is no accuracy loss in the secure version. In addition, our scheme ensures that any user can go offline after uploading the encrypted local data to the server.

In the IPPNN scheme, a user needs to encrypt his training data in two ways: (i) by the random matrix; and (ii) by the public key generated by the $\mathcal{TPA}$.

Specifically, in the first encryption way, the encrypted data are deliberately defined a tuple $(A\mathit{x},A^{-1})$, where A is a random matrix chosen by the user. The pair $(A\mathit{x},A^{-1})$ is split into two parts and sent to ${\mathcal{S}}_{\mathcal{AU}}$ and ${\mathcal{S}}_{\mathcal{NN}}$, respectively. By exploiting the mask matrix A the server ${\mathcal{S}}_{\mathcal{AU}}$ will obtain the inner-product $\mathit{w}·\mathit{x}$. Combining the linear transformation $\mathit{w}·\mathit{x}$ and non-linear activation function $f(·)$ together, the output of neuron $f(\mathit{w}·\mathit{x})$ is sent to ${\mathcal{S}}_{\mathcal{NN}}$ by ${\mathcal{S}}_{\mathcal{AU}}$. The operations for obtaining the gradient $\partial E_S/\partial w_{i,j}^{\left(o\right)}$ can be executed by the server ${\mathcal{S}}_{\mathcal{NN}}$ independently because the remainder computations only depend on model weights $w$ and the provided label.

In the second encryption way, we observe that for obtaining the gradient $\partial E_S/\partial w_{j,k}^{\left(h\right)}$ the feature dimension secure aggregation are required. From the computation process of the gradient $\partial E_S/\partial w_{i,j}^{\left(o\right)}$, we can obtain the coefficient $u_k^{\left(l\right)}$ of the attribute $x_k^{\left(l\right)}$ in the update rule (4), where $u_k^{\left(l\right)}$ is defined by (7). In order to compute the gradient $\partial E_S/\partial w_{j,k}^{\left(h\right)}$, the user encrypt his samples using the FEIP cryptosystem with the public key. With the functional-derived key depending on the vector consisting of the coefficients $u_k^{\left(l\right)}$, the server ${\mathcal{S}}_{\mathcal{NN}}$ is able to decrypt the ciphertext and acquire the gradient $\partial E_S/\partial w_{j,k}^{\left(h\right)}$.

4.4. Description of Proposed Scheme

We construct a scheme for the 3-layer perception which includes three phases: system initialization, user data encryption, and privacy-preserving training. In the system initialization phase, $\mathcal{TPA}$ generates a master secret key and public key based on a given security parameter; in the data encryption phase, the user sends his samples blinded by a random mask matrix A to the ${\mathcal{S}}_{\mathcal{AU}}$ and the inverse of the matrix $A^{-1}$ to the ${\mathcal{S}}_{\mathcal{NN}}$, respectively, and transfer his encrypted sample attributes by the public key to the server ${\mathcal{S}}_{\mathcal{NN}}$; in the privacy-preserving training phase, the servers ${\mathcal{S}}_{\mathcal{AU}}$ and ${\mathcal{S}}_{\mathcal{NN}}$ coordinate the neural network training. More specific details of each stage are described as follows.

4.4.1. System Initialization

The $\mathcal{TPA}$ takes a security parameter $\lambda$ as input and generates a group $(G,p,g)$ and a master secret key $ms{k}^{FEIP}=\mathit{s}=(s_1,...,s_l)\leftarrow {\mathbb{Z}}_p^l$, where p is the order of group G generated by $g\in G$, p is a $\lambda$-bit prime number and must be chosen, such that $p-1$ has at least one large prime factor [36], $l=\left|S\right|$ is the size of a mini-batch. The master key generation algorithm creates a public key $p{k}^{FEIP}={(h_i=g^{s_i})}_{i\in \left[l\right]}$ corresponding to $\mathit{s}$. Then, $\mathcal{TPA}$ distributes the public keys $p{k}^{FEIP}$ to the users, and provides private key service to the server ${\mathcal{S}}_{\mathcal{NN}}$ during the training phase.

4.4.2. Data Encryption

To preserve the data privacy, the user needs to encrypt each sample of his local dataset D. In the following, according to the main idea mentioned in Section 4.3 we present a customized method for protecting users data, adapting to create privacy-preserving ANN model by the ${\mathcal{S}}_{\mathcal{NN}}$ with the cooperation of the ${\mathcal{S}}_{\mathcal{AU}}$.

Assume that dataset D includes m samples. We first choose a mask matrix A to generate a two-part ciphertext $A^{-1}$ and $A\mathit{x}$ where the two parts are utilized by the ${\mathcal{S}}_{\mathcal{NN}}$ and ${\mathcal{S}}_{\mathcal{AU}}$, respectively. Specifically, the user selects a random non-singular matrix $A\in Z^{a\times a}$, then computes its inverse matrix $A^{-1}\in Z^{a\times a}$ and the blinded data $\tilde{\mathit{x}}=A\mathit{x}$ using A, and then finally sends the encrypted version $\tilde{\mathit{x}}$ to the ${\mathcal{S}}_{\mathcal{AU}}$, the matrix $A^{-1}$ and labels $\{{\mathit{y}}^{\left(1\right)},\dots ,{\mathit{y}}^{\left(m\right)}\}$ to the ${\mathcal{S}}_{\mathcal{NN}}$. Then, we divide the dataset D into $⌈m/l⌉$ groups. For each group, we use the public key $p{k}^{FEIP}$ to encrypt the vector formed by the ith $(i=1,2,\dots ,a)$ feature of all of the l samples. The obtained ciphertexts are sent to the server ${\mathcal{S}}_{\mathcal{NN}}$.

The user data encryption approach is specified in Algorithm 1. The user sends $A{\mathit{x}}^{\left(i\right)}$ to the server ${\mathcal{S}}_{\mathcal{AU}}$. Meanwhile, the user transfers $A^{-1}$ with labels $\mathit{y}$ to the the ${\mathcal{S}}_{\mathcal{NN}}$. For IPPNN model, we require the user to share labels to the ${\mathcal{S}}_{\mathcal{NN}}$ in plaintext. Note that the single label vector ${\mathit{y}}^{\left(i\right)}$ is without violating privacy laws since ${\mathit{y}}^{\left(i\right)}$ does not contain any substantial information.

Algorithm 1 User data encryption and uploading.

Input: user dataset $D=\{({\mathit{x}}^{\left(1\right)},{\mathit{y}}^{\left(1\right)}),\dots ,({\mathit{x}}^{\left(m\right)},{\mathit{y}}^{\left(m\right)})\}$, batch size l, public key $p{k}^{FEIP}={(h_i=g^{s_i})}_{i\in \left[l\right]}$. Output: the blinded sample ${\tilde{\mathit{x}}}^{\left(i\right)}$, $i=1,2,\dots ,m$, the encrypted vector for sample features.   1: Choose a random non-singular matrix $A\in {\mathbb{Z}}^{a\times a}$ and compute its inverse matrix $A^{-1}\in {\mathbb{Z}}^{a\times a}$;   2: for$i=1,2,\dots ,m$do   3:     Compute ${\tilde{\mathit{x}}}^{\left(i\right)}=A{\mathit{x}}^{\left(i\right)}$ using the mask matrix A.   4: end for   5: Divide D into $⌈m/l⌉$ groups. Each group contains l samples.   6: Add $l-\left(m\mathrm{mod}d\right)$ random samples uniformly picked from D to the last group if m is not divisible by l.   7: for$i=1,2,\dots ,⌈m/l⌉$do   8:     for $j=1,2,\dots ,a$ do   9:         $r\leftarrow {\mathbb{Z}}_p$ 10:         Encrypt data sample features as $$C{t}_j^{\left(i\right)}=En{c}_{p{k}^{FEIP}}(x_j^{\left(\right(i-1)l+1)},\dots ,x_j^{\left(il\right)})$$ $$=(g^r,h_1^r{g}^{x_j^{\left(\right(i-1)l+1)}},\dots ,h_l^r{g}^{x_j^{\left(il\right)}})$$ 11:     end for 12: end for 13: Send the encrypted data ${\tilde{x}}^{\left(i\right)}$ to the ${\mathcal{S}}_{\mathcal{AU}}$. 14: Send the matrix $A^{-1}$, ciphertexts $C{t}_j^{\left(i\right)}$ and labels $\{{\mathit{y}}^{\left(1\right)},\dots ,{\mathit{y}}^{\left(m\right)}\}$ to the ${\mathcal{S}}_{\mathcal{NN}}$.

4.4.3. Privacy-Preserving Training

We now show the procedure of lossless ANN training for image classification by FEIP scheme and mask matrix support without interacting with users. The server ${\mathcal{S}}_{\mathcal{NN}}$ initiates the protocol with a uniform $W^{\left(h\right)}$ and $W^{\left(o\right)}$ and sets the gradient update step $\lambda$. Through executing the IPPNN protocol, the server ${\mathcal{S}}_{\mathcal{NN}}$ can update its weight matrices iteratively on the encrypted data with the help of the server ${\mathcal{S}}_{\mathcal{AU}}$, and finally obtain a well-trained model. We describe training process of per iteration in the IPPNN protocol below.

The Secure Computation of$\mathit{\partial }{\mathit{E}}_{\mathit{S}}/\partial {\mathit{w}}_{\mathit{i},\mathit{j}}^{\left(\mathit{o}\right)}$. The update rule (3) with a random sample $\mathit{x}$ can be described as

$$\begin{array}{cc}\hfill \frac{\partial E_{\left\{x\right\}}}{\partial w_{i,j}^{\left(o\right)}}& =\left(g\left({\sum }_{k=1}^b{w}_{ik}^{\left(o\right)}f\left({\mathit{w}}_k^{\left(h\right)}·\mathit{x}\right)\right)-y_i\right)f({\mathit{w}}_j^{\left(h\right)}·\mathit{x})\hfill \end{array}$$

(5)

where $i=1,\cdots ,c$, $j=1,\cdots ,b$.

From (5), we find that the update rule requires ${\mathcal{S}}_{\mathcal{NN}}$ to compute the gradients of the loss function (2) which involve inner-product ${\mathit{w}}_k^{\left(h\right)}·\mathit{x}$, sigmoid function $f(·)$, and softmax function $g(·)$ and cannot be completed using only homomorphic additions and multiplications. We first deal with the formidable obstacle $f({\mathit{w}}_k^{\left(h\right)}·\mathit{x})$. Recall the server ${\mathcal{S}}_{\mathcal{NN}}$ owns the model parameters $w_{i,k}^{\left(o\right)}$, with the inner product $f({\mathit{w}}_k^{\left(h\right)}·\mathit{x})$ and provided label $y_i$, the ${\mathcal{S}}_{\mathcal{NN}}$ can obtain the output of hidden layer neurons. Then the computation of $(o_i-y_i)h_j$ takes place in plaintext. We provide details of how $f({\mathit{w}}_k^{\left(h\right)}·\mathit{x})$ are computed by mask matrix structure.

1.

The ${\mathcal{S}}_{\mathcal{NN}}$ first computes

$$\begin{array}{cc}\hfill {\tilde{W}}^{(h)}& =({\tilde{\mathit{w}}}_1^{(h)},\dots ,{\tilde{\mathit{w}}}_b^{(h)})\hfill \\ & =W^{(h)}{A}^{-1}\hfill \\ & =({\mathit{w}}_1^{\left(h\right)}{A}^{-1},\dots ,{\mathit{w}}_b^{\left(h\right)}{A}^{-1}).\hfill \end{array}$$

After that, the ${\mathcal{S}}_{\mathcal{NN}}$ sends ${\tilde{W}}^{\left(h\right)}$ to the ${\mathcal{S}}_{\mathcal{AU}}$.

2.

Then the ${\mathcal{S}}_{\mathcal{AU}}$ takes ${\tilde{W}}^{\left(h\right)}$ and $\tilde{\mathit{x}}$ as input, and return the inner-products ${\tilde{\mathit{w}}}_1^{\left(h\right)}·\tilde{\mathit{x}},\dots ,{\tilde{\mathit{w}}}_b^{\left(h\right)}·\tilde{\mathit{x}}$, that is ${\mathit{w}}_1^{\left(h\right)}·\mathit{x},\dots ,{\mathit{w}}_b^{\left(h\right)}·\mathit{x}$. This is because that

$${\tilde{\mathit{w}}}_j^{\left(h\right)}·\tilde{\mathit{x}}={\mathit{w}}_j^{\left(h\right)}{A}^{-1}A\mathit{x}={\mathit{w}}_j^{\left(h\right)}·\mathit{x}.$$

After that, an execution process of activation function takes place. This results in the ${\mathcal{S}}_{\mathcal{AU}}$ having all outputs of hidden layer neurons

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \mathit{h}=(h_1,\dots ,h_b)=(f({\mathit{w}}_1^{(h)}·\mathit{x}),\dots ,f({\mathit{w}}_b^{(h)}·\mathit{x}))\end{array}\end{array}$$

(6)

Then, the ${\mathcal{S}}_{\mathcal{AU}}$ sends $\mathit{h}$ to the ${\mathcal{S}}_{\mathcal{NN}}$.

Once the server ${\mathcal{S}}_{\mathcal{NN}}$ receives the outputs of hidden layer neurons, it prepares $w_{ik}^{\left(o\right)}$ and $y_i$ to perform the computation of $(g({\sum }_{k=1}^b{w}_{ik}^{(o)}{h}_k)-y_i)h_j$ and obtains the gradient $\partial E_{\left\{x\right\}}/\partial w_{i,j}^{\left(o\right)}$. The gradient $\partial E_S/\partial w_{i,j}^{\left(o\right)}$ in a mini-batch setting can be computed easily from a group of values $\partial E_{\left\{x^{\left(l\right)}\right\}}/\partial w_{i,j}^{\left(o\right)},l=1,...,\left|S\right|.$

The secure computation of ${\displaystyle \partial }{\mathit{E}}_{\mathit{S}}{\displaystyle /}{\displaystyle \partial }{\mathit{w}}_{\mathit{j},\mathit{k}}^{\left(\mathit{h}\right)}$. The server ${\mathcal{S}}_{\mathcal{AU}}$ follows the FEIP in the process of obtaining $\partial E_S/\partial w_{i,j}^{\left(o\right)}$. As a result, the ${\mathcal{S}}_{\mathcal{AU}}$ has the outputs of hidden layer neurons $h=(h_1,h_2,\dots ,h_b)$. The server ${\mathcal{S}}_{\mathcal{NN}}$ can exploit the results of $\partial E_S/\partial w_{i,j}^{\left(o\right)}$ and encrypted feature $x_k$ of data sample to obtain the gradient $\partial E_S/\partial w_{j,k}^{\left(h\right)}$.

Take the kth feature vector $(x_k^{\left(1\right)},\dots ,x_k^{\left(l\right)})$ of the first group samples as an example, with the help of FEIP, the server ${\mathcal{S}}_{\mathcal{NN}}$ is able to acquire the gradient

$$\frac{\partial E_S}{\partial w_{j,k}^{\left(h\right)}}=\frac{1}{\left|S\right|}\sum _{l\in S}{u}_k^{\left(l\right)}{x}_k^{\left(l\right)}$$

where

$$\begin{array}{c}\hfill u_k^{(l)}={\sum }_{i=1}^c[(y_i^{(l)}-g({\sum }_{n=1}^b{w}_{in}^{(o)}{h}_n^{(l)}))w_{i,j}^{(o)}]h_j^{(l)}(h_j^{(l)}-1),\end{array}$$

(7)

$j=1,\dots ,b,k=1,\dots ,a$.

Note that the terms $h_n^{\left(l\right)}(n=1,2,\dots ,b)$ have already obtained from the computation of $\partial E_S/\partial w_{i,j}^{\left(o\right)}$. With the provided $w_{ik}^{\left(o\right)}$ and labels $y_i^{\left(l\right)}$, the ${\mathcal{S}}_{\mathcal{NN}}$ is able to acquire the value $u_k^{\left(l\right)}$. The ${\mathcal{S}}_{\mathcal{NN}}$ takes the ciphertexts of vector $(x_k^{(1)},\dots ,x_k^{(l)})$, the public key $p{k}^{FEIP}$ and functional key for the vector ${\mathit{u}}_k=(u_k^{(1)},\dots ,u_k^{(l)})$ as input, and returns the inner-product ${\sum }_{l\in S}{u}_k^{\left(l\right)}{x}_k^{\left(l\right)}$. We provide the following steps towards this purpose.

1.

The $\mathcal{TPA}$ takes the master private key $ms{k}^{FEIP}=\mathit{s}$ and the plaintext vector ${\mathit{u}}_k=(u_k^{(1)},\dots ,u_k^{(l)})$ sent by the ${\mathcal{S}}_{\mathcal{NN}}$ as input, and generates a functionally derived key $d{k}_{{\mathit{u}}_k}=\langle {\mathit{u}}_k,\mathit{s}\rangle$ as output.

2.

The ${\mathcal{S}}_{\mathcal{NN}}$ takes the ciphertexts

$$C{t}_k^{\left(1\right)}=En{c}_{p{k}^{FEIP}}(x_k^{\left(1\right)},\dots ,x_k^{\left(l\right)})$$

$$=(g^r,h_1^r{g}^{x_k^{\left(1\right)}},\dots ,h_l^r{g}^{x_k^{\left(l\right)}})$$

received from the user, the public key $p{k}^{FEIP}$ and functionally derived key $d{k}_{{\mathit{u}}_k}$ as input, and return the inner-product ${\sum }_{l\in S}{u}_k^{\left(l\right)}{x}_k^{\left(l\right)}$.

In specific,

$$g^{\sum _{l\in S}{u}_k^{\left(l\right)}{x}_k^{\left(l\right)}}=\prod _{i\in \left[l\right]}{\left(h_i^r{g}^{x_k^{\left(i\right)}}\right)}^{u_k^{\left(i\right)}}/{\left(g^r\right)}^{d{k}_{u_k}}$$

In order to recover the final inner-product value ${\sum }_{l\in S}{u}_k^{\left(l\right)}{x}_k^{\left(l\right)}$ a discrete logarithm computation is implemented.

Correctness. Given the public key $p{k}^{FEIP}={(h_i=g^{s_i})}_{i\in \left[l\right]}$, we have

$$\begin{array}{cc}& \prod _{i\in \left[l\right]}{\left(h_i^r{g}^{x_k^{\left(i\right)}}\right)}^{u_k^{\left(i\right)}}/{\left(g^r\right)}^{d{k}_{{\mathit{u}}_k}}\hfill \\ \hfill =& \prod _{i\in \left[l\right]}{(g^{s_{i}r+x_k^{\left(i\right)}})}^{u_k^{\left(i\right)}}/g^{r({\sum }_{i\in \left[l\right]}{u}_k^{\left(i\right)}{s}_i)}\hfill \\ \hfill =& g^{{\displaystyle \sum _{i\in \left[l\right]}}{u}_k^{\left(i\right)}{s}_{i}r+{\sum }_{i\in \left[l\right]}{u}_k^{\left(i\right)}{x}_k^{\left(i\right)}-r({\sum }_{i\in \left[l\right]}{u}_k^{\left(i\right)}{s}_i)}\hfill \\ \hfill =& g^{{\displaystyle \sum _{i\in \left[l\right]}}{u}_k^{\left(i\right)}{x}_k^{\left(i\right)}}\hfill \end{array}$$

5. Analysis and Evaluation

In this section, we first analyze the security of IPPNN under the threat model described in Section 4.2. Recall that the $\mathcal{TPA}$ is trusted to generate the public key and master secret key, enabling the FEIP feasible, the servers ${\mathcal{S}}_{\mathcal{AU}}$ and ${\mathcal{S}}_{\mathcal{NN}}$ are non-colluding and could be honest-but-curious, the external adversary is malicious. We aim to verify if IPPNN can train an ANN model without revealing any information of users, meanwhile, hiding parameters trained by the server ${\mathcal{S}}_{\mathcal{NN}}$ from other entities.

Our IPPNN scheme should protect the data privacy against Class-I (honest-but-curious ${\mathcal{S}}_{\mathcal{AU}}$) and Class-II (honest-but-curious ${\mathcal{S}}_{\mathcal{NN}}$), and model privacy against Class-I.

5.1. Data Privacy

The primary privacy constraint is that the servers ${\mathcal{S}}_{\mathcal{AU}}$ and ${\mathcal{S}}_{\mathcal{NN}}$ should not be able to observe the user data and similarly, the external adversary should not be able to observe data belonging to users.

5.1.1. Class-I

In Algorithm 1, the user utilizes the random matrix A to mask his samples ${\mathit{x}}^{\left(i\right)}$, and then sends $A{\mathit{x}}^{\left(i\right)}$ to the ${\mathcal{S}}_{\mathcal{AU}}$. In this way, the ${\mathcal{S}}_{\mathcal{AU}}$ never gets to know any sample included in the training set since A is randomly selected by the user. In the privacy-preserving training process, although the ${\mathcal{S}}_{\mathcal{AU}}$ can obtain the inner-product ${\mathit{w}}_1^{\left(h\right)}·\mathit{x},\dots ,{\mathit{w}}_b^{\left(h\right)}·\mathit{x}$, but it cannot infer the user data without knowing the weight vectors ${\mathit{w}}_1^{\left(h\right)},\dots ,{\mathit{w}}_b^{\left(h\right)}$.

5.1.2. Class-II

In the computation of gradient $\partial E_S/\partial w_{i,j}^{\left(o\right)}$ of output layer, the ${\mathcal{S}}_{\mathcal{NN}}$ receives all outputs of hidden layer neurons $\mathit{h}$. Due to the non-linear property of sigmoid function f, the ${\mathcal{S}}_{\mathcal{NN}}$ can not solve the system $\{f({\mathit{w}}_1^{(h)}·\mathit{x}),\dots ,f({\mathit{w}}_b^{(h)}·\mathit{x})\}$. In the computation of gradient $\partial E_S/\partial w_{j,k}^{\left(h\right)}$ of hidden layer, the ${\mathcal{S}}_{\mathcal{NN}}$ receives the ciphertext $C{t}_j^{\left(i\right)},i=1,2,\dots ,⌈m/l⌉,j=1,2,\dots ,a$ (executed by step 10 of Algorithm 1). Under the DDH assumption, the encrypted scheme is against chosen-plaintext attacks.

5.1.3. External Adversary

In our threat model, we do not assume there exists an external adversary who can monitor network and try to infer users’ private samples and model parameters. If so, the solution is straightforward. The ${\mathcal{S}}_{\mathcal{NN}}$ can generate a pair of a public key and a secret key by any public key cryptosystem. The user encrypts the matrix $A^{-1}$ by the public key, and the ${\mathcal{S}}_{\mathcal{NN}}$ uses his secret key to decrypt the mask matrix. Even if the external adversary can obtain the information of $Ax$, he can not deduce the data $x$.

5.2. Model Privacy

5.2.1. Class-I

The secondary privacy constraint is to protect the model privacy trained by the ${\mathcal{S}}_{\mathcal{NN}}$. We should make sure that the privacy of $W^{\left(h\right)}$ and $W^{\left(o\right)}$ should be preserved against the ${\mathcal{S}}_{\mathcal{AU}}$. Similar to the analysis of data privacy in Section 5.1.1, finding the solutions of system $\{{\mathit{w}}_1^{\left(h\right)}·\mathit{x},\dots ,{\mathit{w}}_b^{\left(h\right)}·\mathit{x}\}$ is impossible. The reason is that the ${\mathcal{S}}_{\mathcal{AU}}$ can establish a linear system with b equations and $(b+1)a$ unknowns only.

5.2.2. External Adversary

The external adversary can eavesdrop network channel to obtain the information of ${\tilde{W}}^{\left(h\right)}$. He can not deduce the model parameters $W^{\left(h\right)}$ since he has no information of $A^{-1}$.

From the above analysis, our scheme satisfies the security requirements defined in Section 4.2.

5.3. The Goal of Experiments

To evaluate the performance of our proposed scheme, we compare IPPNN with the Baseline and NPMML [11] algorithms.

(i) Precision measurement: To measure the model precision of our method and the influence of using Taylor approximation in neural network model, an experiment with IPPNN and Baseline (a centralized version, non-privacy-preserving setting) and Baseline_app (with Taylor polynomial approximation) is designed to compare the classification accuracy.

(ii) Efficiency measurement: For the purpose of evaluating he efficiency of our method, we compare the execution time of IPPNN and NPMML training different image datasets. We use the NPMML proposed in [11] as the comparative object because it is the closest state-of-the-art approach to ours. In NPMML, the neural network model is built using the combination of Paillier cryptosystem and a CCA-2 secure public-key encryption scheme.

5.4. Experimental Setup

We conduct all experiments on a service platform (ECS) with ecs.n4.xlarge machines with 4-Core vCPU and 8GB RAM running on Linux system. We implement the neural network training on two image datasets digits-MNIST [37] and Fashion-MNIST [38]. Both of them contain 60,000 training samples (28 × 28) and 10,000 test samples. Each image is transformed into a vector with dimension 784 as the input of the neural network.

For all experiments, we construct a fully connected multi-layer perception with one hidden layer (with sigmoid function) and a output layer (with softmax function). For IPPNN, we use the function encryption proposed in [27] as the basic cryptographic technology. For NPMML, we use Python-RSA [39] and Python-Paillier [40] to implement the underlying encryption primitives.

The parameter setting is: learning rate $\eta =0.01$, scaling factor $={10}^6$ (for the purpose of extending the original encryption function to real numbers), epoch $=20$.

5.5. Evaluation of Accuracy

Our IPPNN removes the need for approximating neural network model by using Taylor series. As shown in Figure 3, the classification accuracy of IPPNN is comparable to that of the Baseline in the same environment. The difference in classification accuracy between the two methods is less than 0.5%, which is caused by extending the encryption function for inner product to real numbers by scaling factor. These results suggest that the IPPNN approach supports lossless image classifiers construction.

For the same network architecture with the same dataset, the accuracy obtained by the approximation model is much lower than the IPPNN. We observe that the errors in identifying the image classification by approximation models Baseline_app are not negligible. It is also noted that ANN model with a higher-degree polynomial can not cause precision improvement, which is not feasible to be used directly with training protocols using polynomial approximation.

5.6. Comparison of Execution Time

For the purpose of evaluating the efficiency of privacy-preserving image classifiers based on neural network algorithms, we compare the execution time of IPPNN and NPMML training different datasets and different network architecture under different size of security parameters, respectively. The training time of the two executing algorithms IPPNN and NPMML, respectively, is shown in

Table 1. Comparison of time cost for training one mini-batch (60 samples).

Network Architecture	Datasets	Bits of Security Parameter	Training Time (s)
			IPPNN	NPMML
784 ↦ 256 ↦ 10	digits-MNIST	1024	1315	31,054
784 ↦ 128 ↦ 10	digits-MNIST	1024	532	13,583
784 ↦ 256 ↦ 10	Fashion-MNIST	1024	1355	31,154
784 ↦ 128 ↦ 10	Fashion-MNIST	1024	552	13,267
784 ↦ 256 ↦ 10	digits-MNIST	2048	2102	52,590
784 ↦ 128 ↦ 10	digits-MNIST	2048	903	24,587
784 ↦ 256 ↦ 10	Fashion-MNIST	2048	2238	52,990
784 ↦ 128 ↦ 10	Fashion-MNIST	2048	931	23,183

Training time: execution time for training one mini-batch (60 samples). Network architecture: No. neurons in input layer ↦ No. neurons in hidden layer ↦ No. neurons in output layer.

.

These experiments illustrate that the running time of NPMML is roughly 25 times longer than that of IPPNN. To summarize the results: (i) the training time will more than double if the number of neurons in hidden layer become a double while other settings remain fixed. (ii) the training time will slightly less than double if the bits of security parameter become a double while other settings remain fixed. The above experimental results manifest that the IPPNN is more suitable for training large-volume image data sets with higher bits of security parameter and more neurons.

5.7. Comparison of Feature

For the purpose of measuring the properties of state-of-the-art privacy-preserving neural network algorithms, we compare the properties of current work in the literature in terms of lossless training, non-interactivity between users and cloud servers and techniques employed by the algorithms. From the

Table 2. Feature comparison of several superior schemes.

	Ours	NPMML [11]	FedV [19]	MK-FHE [12]	Chain-PPFL [20]	SecureML [16]	Ma et al. [13]
Non-interactive	✔	✔	✘	✘	✘	✔	✘
Lossless	✔	✔	✔	✘	✔	✘	✔
Technique	FE	Paillier&RSA	FE	FHE	FL	SS	FL

FE: function encryption. Paillier: Paillier cryptosystem. RSA: RSA cryptosystem. FHE: fully homomorphic encryption. SS: secret sharing. FL: federal learning. The symbol ✔ denotes the scheme has the feature of the corresponding row, while the symbol ✘ has not.

, we can find most of schemes can not have non-interactive and lossless features simultaneously.

One of the main novelties of the IPPNN method is to remove the need for multiple communications between the users and the servers. In other words, the IPPNN is a non-interactive scheme which only requires communication between the users and the servers as a one-way transmission. The key point is that the encrypted data are specific-designed through two folds: (i) by the public key of function encryption and (ii) by the random mask matrix. The server ${\mathcal{S}}_{\mathcal{AU}}$ will obtain the information of inner $\mathit{w}·\mathit{x}$, by exploiting the blinded data $x$ and mask matrix A. After computing the outputs of all neurons and sending them to the server ${\mathcal{S}}_{\mathcal{NN}}$, the tasks of the ${\mathcal{S}}_{\mathcal{AU}}$ are completed. The rest of the work can be completed by the server ${\mathcal{S}}_{\mathcal{NN}}$ alone by using the encrypted data by the public key. To summarize, the two servers are able to utilize the encrypted data to train the image classifier by cooperation without the help of the users. Once the users upload the encrypted data to the servers, they can be offline.

Although NPMML method has both of the characters with non-interactive and lossless, its efficiency is lower than ours. A few factors contribute to the IPPNN outperforms the NPMML. (i) Cryptographic primitive employed—reducing computation overhead. The running time gap between NPMML and IPPNN lies in the cryptographic techniques employed by these two algorithms. In NPMML, the neural network model is built using the combination of Paillier cryptosystem and a CCA-2 secure public-key encryption scheme, such as RSA, while the IPPNN employs the only one cryptographic technique, i.e., function encryption for inner product. Consequently, the NPMML has more complex implementation than ours. (ii) Well-designed encryption method—reducing communication overhead. At each iteration, the NPMML needs two-way transmission between the servers, while the IPPNN only needs one time to transfer the information.

6. Conclusions

We developed a non-interactive protocol for lossless training an ANN-based image classifier over images belonging to multiple users while preserving the privacy constraints. Meanwhile, we presented a theoretic analysis of the security of the protocol. We also experimented with an implementation of the protocol in the real machines on two large scale image datasets and demonstrate that our protocol is able to achieve high-precision performance in a feasible amount of execution time. The future direction of this work includes applying our method to image classifiers based on ANN over vertically partitioned data. We also plan to extend our protocol to use appropriate dimensionality reduction techniques to further increase the speed.