Investigation of Memristor-Based Neural Networks on Pattern Recognition ^†

Abstract

Mobile phones, laptops, computers, digital watches, and digital calculators are some of the most used products in our daily life. In the background, to make these gadgets work as per our desire, there are many simple components necessary for electronics to function, such as resistors, capacitors, and inductors, which are three basic circuit elements. The Memristor is one such component. This paper provides simulation results of the memristor circuit and its V-I characteristics at different functions as an input signal. A well-trained ANN is able to recognize images with higher precision. To enhance the properties such as accuracy, precision, and efficiency in recognition, memristor characteristics are introduced to the neural network, however, older devices experience some non-linearity issues, causing conductance-tuning problems. At the same time, to be used in some advanceable applications, ANN requires a huge amount of vector-matrix multiplication based on in-depth network expansion. An ionic floating gate (IFG) device with the characteristics of a memristive device can solve these problems. This work proposes a fully connected ANN using the IFG model, and the simulation results of the IFG model are given as synapses in deep learning. We use algorithms such as the gradient-descent model, forward and backward propagation for network building, and weight setting in neural networks to enhance their ability to recognize images. A well-trained network is formed by tuning those memristive devices to an optimized state. The synaptic memory obtained from the IFG device will be used in other deep neural networks to increase recognition accuracy. To be an activation function in the neural network, sigmoid functions were used but later replaced by the ReLu function to avoid vanishing gradients. This paper shows how images were recognized by their front, top, and side views.

1. Introduction

A memristor is a simple passive two-terminal structure first coined by Professor Leon Chua in 1971. It is the concatenation of a “Memory resistor”, which is a fourth fundamental circuit element besides resistor, capacitor, and inductor; this is the missing pair link that states the relationship between charge and flux over four basic circuit variables [1]. Depending on the input signal applied, there are two types of memristors: current-controlled and voltage-controlled [2]. The basic memristor is given in Figure 1.

Using Equations (1) and (2), we can find the value of memristance or memductance. The properties of the memristor made it useful for non-volatile memory and storage technology applications as it can theoretically develop multiple states. It can also operate at a very low voltage level. For the operation of a neural network as a synapse, these properties of memristor could be useful. Equations (1) and (2) below show the relationship between the flux and charge concerning voltage and current [1].

$$\mathrm{M}\left(\mathrm{q}\right)=\frac{\mathrm{d}\mathsf{\Psi }}{\mathrm{dq}}$$

(1)

$$\mathrm{M}\left(\mathrm{q}\left(\mathrm{t}\right)\right)=\frac{\mathrm{d}\mathsf{\Psi }}{\mathrm{dt}}.\frac{dt}{dq}=\frac{V\left(t\right)}{I\left(t\right)}$$

(2)

ANNs are interconnected groups of nodes that are composed of artificial neurons. An artificial neural network is a non-linear and self-adapting computational model that retains the biological concept of neural networks as human/animal brains [3]. An ANN has three layers, an input layer, a hidden layer, and the output layer. Every node of the input layer connects through links with every node of the next preceding layer, called a fully connected network [4]. It receives inputs, combines them, performs required computations based on a predefined activation function, and delivers the output. The devices with ANN using two terminals suffer from non-linearity and asymmetric conductance tuning problems. Due to this, their scalability was also affected. To solve these issues, an IFG (Ionic Floating Gate) model memristor-based device will be developed, resulting in a memristor-based neural network (MNN) from an ANN with memristor features. To implement this, a cadence Verilog A model will be used along with deep learning concepts. A memristor device has a simple structure with two terminals, its conductance can be modified by simple positive and negative pulses while representing synaptic weight [5]. These main functions of the memristor made it suitable for realizing the synaptic weight in an artificial neural network. By using a memristor device output as a synapse, a memristive neural network is expanded to be a multiple layer network with modified memristor-based backpropagation and gradient descent learning rules [6].

2. Memristor-Based Neural Networks

For the design of the MNN building, the network and weight loading both take a crucial role. Thereafter, the IFG model will act as a conductor. For the pattern recognition to be verified by the MNN, a dataset called MNIST is used at input neurons. Additionally, we use resistors to build a fully connected MNN. The optimization of weights is done by software [7]. The following Equation (3) is given for the output voltage.

$$\mathrm{Vout} =\mathrm{Rlimit}.{\displaystyle \sum }_{i=1}^{\Psi }VRi.\frac{1}{\mathrm{Rij}}$$

(3)

Here, Rij is the reciprocal of the conductance(G) which are obtained by Cadence Virtuoso, these are further used as weights Wij. To train and test the neural network, we need a massive dataset. Fortunately, the MNIST (modified national institute of standards and technology) database exists, which contains 60,000 training images and 10,000 testing images. In this dataset, picture pixels will be normalized by greyscale numbers divided from 0 to 255 values [8].

2.1. Memristor-Based IFG Model

This Ionic Floating Gate (IFG) memristive device has three terminals and is a combination of a redox transistor and a non-volatile CBM to make it a non-volatile synaptic memory [9]. A redox transistor was developed in memristor techniques to find a way to clear limitations such as low writing efficiency, vanishing gradients, and limited accuracy. It contains three layers. The first layer is PEDOT: which is made of polystyrene sulfonate i.e., PSS film followed by the Nafion layer. The last layer is the PEI (poly(ethylenimine)) layer, which is partially reduced by PEDOT: PSS film [10]. Connected to this is a conductive bridge memory that is designed with a layer of Ag in between Pt electrodes. The IFG model can be used as a memory storage device to memorize the last operation. Memory operates in two modes: read operation and write operation. Figure 2 and Figure 3 shows the internal view of the IFG memory [11].

By taking the above two models as a reference including some modifications in them, a compact IFG model characterized in Verilog-A code is developed, and importing this Verilog-A code file to cadence generates the IFG model [12]. The cadence model is shown in Figure 4. Here, we change the selector position to the electrolyte and middle of the gate terminal. These are suitable for the design of memristor-based synaptic circuits. The memristor IFG model conductance values are taken as synaptic arrays. For the recognition of patterns/images, we use a three-layer artificial network that is fully connected [13]. All weight connections hold their fixed conductance between the layers of the neural network, which is stored as IFG’s memristance “G” and mapped into the source-drain conductance of the synaptic array [14]. Here, the change in conductance is proportional to the flux, its gate voltage is greater than the threshold voltage, and tunes the gate to the source voltage [15]. The schematic circuit of IFG is shown in Figure 4, which is implemented in cadence.

The voltage applied at the gate node must be greater than the threshold voltage (V_th). The voltage level should be ±0.95 V which is given to the gate and source node.

2.2. Weight Setting through Gradient Descent Model and Backpropagation

Specific learning algorithms are useful to hasten the training of neural networks. Here, we adopt the gradient descent model and backpropagation along with memristor-based IFG circuit. It takes less than a few seconds to run the ANN [16]. A text-based “Spectra” is generated for the connections in the ANN circuit with the help of python. The input for this python code is the database from cadence simulation (Synapse) [17]. First, a normal-weighted synapse is given from output of the IFG model to the neural networks after getting the optimized values from the neural networks. Again, these optimized values should be applied to IFG and observe whether our activation function is optimized. Here, the weights are optimized by using gradient descent and backpropagation models. The differences are backpropagated to the neural networks. At epoch 0.68, the accuracy increases by 0.8% as compared to the traditional method. The circuit-level design and implementation of gradient and backpropagation learning architectures help to get optimized results. These results are compared with the original one. The below given Figure 5 shows the simple operation of ANN.

To achieve image pattern recognition, the optimized values are again given as weighted synapses in the neural networks along with the images as input. Here, ReLu and Adams activation functions are used. Wij are weighted neurons, and hij is the summation of input neurons and weights. The summation is fed to the activation function, which is shown in Figure 6.

3. Results and Analysis

To check the basic memristor characteristics observe Figure 7, the sine function is given as input harmonic with voltage (V) of 1 V amplitude and 1 Hz frequency (V(t) = V₀sin(wt) for w = 2w₀ angular frequency where the V-I characteristics shows with the applied negative voltage the current (I) increases, and the current (I) drops with the positive voltage value, causing a hysteresis loop to emerge. We can also see the linear relationship between the charge and flux.

In the result given Figure 8, for a positive pulse, the conductance values decrease for positive input voltage and increase with the negative input voltage i.e., similar to the characteristics of a memristor. These are the simulation results of the IFG model in cadence, for which the voltage value is ±0.95 V, and the threshold voltage is 0.4 V. The input voltage is tuned in between the gate and source, the threshold voltage (V_th) is applied to dc source i.e., connected in between drain and source.

Figure 9 below shows the post-simulation values for the optimized values from the IFG model.

After applying the optimized values to the IFG device, the same operation is repeated as it was in previous case but with increased accuracy by 0.8%. Here, the optimized parameters are X = 0.150 V, and Y = −0.612 V.

Here,

Table 1. Parametric values of the current, voltage, flux and charge for hysteresis curve with reference to Figure 1.

Measurement #	Input Voltage (V)	Input Current (I)	Flux (Ψ)	Charge (q)
1.	0.018 mV	0.022 mA	3.3 × 10−9 (Wb)	−2.8 × 10−9 C
2.	0.037 mV	0.044 mA	1.3 × 10−4 (Wb)	−1.1 × 10−8 C
3.	0.056 mV	0.066 mA	1.5 × 10−4 (Wb)	−2.5 × 10−8 C
4.	0.075 mV	0.088 mA	1.8 × 10−4 (Wb)	−4.5 × 10−8 C

mention some values of basic memristor showing how the current varies with a change in input voltage and

Table 2. Parametric value analysis when the input pulse response = (+0.95 V).

Measurement #	Input Voltage (V)	Siemens (ns)
		Gmax = 1 × 107(X)	Gmax = 1 × 107(Y)
1.	0.95 V	0	1.0 × 10−7
2.	0.60 V	2.0 × 10−8	1.0 × 10−7
3.	0.31 V	3.0 × 10−7	1.0 × 10−7
4.	0.19 V	4.02 × 10−6	9.9 × 10−8
5.	0	6.02 × 10−6	9.9 × 10−8

and

Table 3. Parametric value analysis when the input pulse response = (−0.95 V).

Measurement #	Input Voltage (V)	Siemens (ns)
		Gmax = 1 × 10−7(X)	Gmax=1 × 10−7(Y)
1.	0	0	1.0 × 10−7
2.	−0.19 V	2.0 × 10−8	1.0 × 10−7
3.	−0.31 V	3.0 × 10−7	1.0 × 10−7
4.	−0.60 V	4.0 × 10−6	1.0 × 10−7
5.	−0.95 V	6.0 × 10−6	1.0 × 10−7

mention values of IFG using Cadence Virtuoso, how conductance varies with a change in input voltage. The various parameters stated here for basic memristor are current (I), voltage (V), flux(Ψ), and charge (q). For the IFG model, the Gmax (conductance) values are given as a relationship between voltage and siemens.

By observing

Table 4. Comparison between analytical values and optimized values.

Model #	Activation Function	Accuracy
Traditional model	Sigmoid function	93.8%
Proposed model	ReLu function and Adams function	94.6%

, the traditional model is compared with proposed model, in traditional model sigmoid function is used and in proposed model ReLu and Adams functions are used as activation function in neural network with optimized parameters causes the accuracy of recognition to increased up to 94.6%. So, with the proposed IFG model, the accuracy increases.

4. Conclusions

Artificial neural networks have a vital role in unsupervised deep learning models to implement applications based on pattern recognition and also in many areas of our daily lives. This paper provides us with the knowledge of developing an IFG device using cadence with the characteristics of a memristor-based circuit. To demonstrate the capability of an IFG device in pattern recognition, the values are optimized using the gradient descent model and the resulting optimized parameters are compared with the existing parameters. The testing ability of neural network increased by 0.8% when using the IFG model. Hence, the accuracy of the original network is 93.8%, which increases to 94.6%. Here, ReLu and Adams activation helps in faster optimization, and by this result, we can conclude that, by using IFG-based memristor characteristics in neural networks, one can increase image recognition accuracy.