Multimodal Mood Consistency and Mood Dependency Neural Network Circuit Based on Memristors

Abstract

The factors that affect learning efficiency in different environments have been considered in many studies, but multimode mood-consistent learning has not been considered specifically. In this paper, a neural network circuit based on memristors to determine multimode mood consistency and mood dependency was constructed. The circuit is composed of a voltage control module, an emotion module, and a synaptic neuron module. Through the voltage control module and emotion module, learning materials with different properties are input into the synaptic neurons. The learning efficiency of different learning materials under different emotions was analyzed in detail. A dual-channel mood-consistent learning was realized, and the mood dependency was further considered. The influence of different channels on the learning was studied to provide ideas for the future development of intelligent brain-like neural networks.

1. Introduction

The study [1] found that the generation of emotions requires the participation of multiple parts of the brain, and their influence on emotions is also different. Emotions affect people’s learning and decision-making, which are conducive to the survival and adaptation of individuals. In recent years, the research on artificial neural networks has become a hot spot, and brain-like intelligence has received much attention. Software intelligence is developing very rapidly. However, software intelligence presents some defects, such as the power consumption and the footprint of computational resources. That being said, the memristor, proposed by Professor Cai and manufactured by HP Lab [2], has the characteristics of a small size and a low power consumption, as well as the advantages of nonlinearity and non-volatility. The realization of brain-like intelligence based on memristors can greatly promote the development of artificial neural networks for associative memory [3,4,5,6,7,8,9,10,11,12] and emotional intelligence [13,14,15,16,17,18,19]. With the rapid development of artificial intelligence, more and more people are paying attention to the research on brain-like emotional intelligence [20,21,22,23]. The relationship between learning and emotion is being studied to further promote the efficiency of learning.

The study of learning, memory, and emotion is the key to brain-like intelligence. Artificial neural networks based on memristors have been explored by many researchers [4,5,7]. A neural network based on memristors and dual-mode switching has been proposed to achieve mood consistency [19]. A two-dimensional emotional model based on memristors was proposed by [21], but they did not study learning. In [22], an emotional neural network model was proposed to analyze the relationship between emotion and memory through a memristor circuit. A memristor circuit based on a skin sensor was proposed to upgrade emotion generation [23]. A learning function was realized based on the combination of software and hardware using artificial synapses and neurons, but memory and emotion were ignored in [24]. A circuit to distinguish five basic emotions was proposed, but learning was not considered in [25]. The research on learning, memory, and emotion in brain-like intelligence has made continuous progress, as presented in the above work. However, the research on the relationship between emotion and learning in multimodal learning remains limited.

Learning, memory, and emotion are closely related and complement each other. The influence of emotion on memory can be divided into two types: mood-congruent memory (MCM) and mood-dependent memory (MDM) [26,27]. When learning or memory is consistent with the emotional state, it is called MCM. For example, when an individual is in a happy mood, the efficiency of learning positive content becomes higher, and the efficiency of learning negative content becomes lower. Conversely, when an individual is in a sad mood, negative content is better remembered than the positive content. MDM refers to the matching degree of the mood during both the learning and recalling stages. In other words, when an individual is in the same mood in the learning and the recognition or recalling stages, the amount of content that can be recalled will increase. When the mood is different in the recalling stage and the learning stage, the amount of content that can be recalled will be less. In this paper, a neural network circuit based on memristors is proposed to discuss the relationship between learning and emotion as regards multimodal mood consistency and mood dependency. Moreover, a single-channel visual and auditory mode and a dual-channel audio visual mode were considered, and the learning efficiency in different environments is discussed.

Compared with the above work [19,20,21,22,23,24,25,26,27], the work in this paper has the following advantages. First, learning is combined with emotion, and the consistency of mood is discussed. The paper is based on memristors to analyze learning and memory in different environments. The good features of memristors are conducive to circuit integration and information processing. A more practical learning process is compared with [24]. Secondly, the combination of the single-channel visual and auditory mode and the dual-channel visual and auditory mode was considered to analyze the mood consistency. The learning efficiency of the different modes of learning is more comprehensive than that in [19,20,21,22]. Finally, the mood dependency in the forgetting stage was also considered. The mood consistency and mood dependency are the most-important aspects to discover the relationship between learning and emotion, which is more practical than the methodology in [19,20,24,25].

The rest of the paper is organized as follows. In Section 2, the memristor model and parameter settings are introduced. In Section 3, the relationship between mood consistency and mood dependency is introduced, and the single-channel and dual-channel modes are discussed. In Section 4, the specific analysis and parameter settings of each module in the circuit are described. In Section 5, the simulation results of the circuit are analyzed in detail. In Section 6, some conclusions and prospects for future work are drawn.

2. Memristor Model

In recent years, many memristor models have been proposed [28,29,30]. In this paper, the threshold memristor model of AIST (AgInSbTe), which is the most suitable for constructing synaptic and neuronal characteristics, was selected. The prominent advantage is that the size of the synaptic weight is related to the memristance of the AIST memristor. When the voltage applied to both ends of the memristor is greater than its threshold voltage, the resistance value of the memristor will increase or decrease. The AIST model is described as

$$M\left(t\right)=R_{on}\frac{w\left(t\right)}{D}+R_{off}(1-\frac{w\left(t\right)}{D}),$$

(1)

$$\frac{dw\left(t\right)}{dt}=\left\{\begin{array}{cc}{\mu }_v\frac{R_{on}}{D}\frac{i_{off}}{i\left(t\right)-i_0}f\left(w\left(t\right)\right),\hfill & v\left(t\right)>V_{T+}>0\hfill \\ 0,\hfill & V_{T-}\le v\left(t\right)\le V_{T+}\hfill \\ {\mu }_v\frac{R_{on}}{D}\frac{i\left(t\right)}{i_{on}}f\left(w\left(t\right)\right),\hfill & v\left(t\right)<V_{T-}<0\hfill \end{array}\right.$$

(2)

$$\begin{array}{ccc}& & f\left(w\left(t\right)\right)=1-{(\frac{2w\left(t\right)}{D}-1)}^{2p}\hfill \end{array}$$

(3)

where $M\left(t\right)$ is the memristor and $R_{on}$ and $R_{off}$ are the low and high memristance values, respectively. $W\left(t\right)$ is the width of the doped area inside the memristor, and D is the total thickness of the memristor. $i_0$, $i_{on}$, and $i_{off}$ are the current constants passing through, and $v\left(t\right)$ is the voltage applied by the memristor. ${\mu }_v$ is the average ion mobility. $V_{T+}$ and $V_{T-}$ are the positive and negative threshold voltages of the memristor, respectively. $f\left(w\right(t\left)\right)$ is a window function, where p is a positive integer.

The settings of the AIST model parameters are shown in

Table 1. Parameters setting of memristors.

Parameters	${\mathit{M}}_1\mathit{and}{\mathit{M}}_2\mathit{and}{\mathit{M}}_3$	${\mathit{M}}_4\mathit{and}{\mathit{M}}_7\mathit{and}{\mathit{M}}_8$	${\mathit{M}}_5\mathit{and}{\mathit{M}}_6$	${\mathit{M}}_9$
D (nm)	3	3	3	3
$R_{on}(\Omega )$	100	10	50	100
$R_{off}(\Omega )$	$2k$	300	300	$1k$
$R_{init}(\Omega )$	$1k$	200	200	500
$V_{T+}$ (V)	0.1	0.19	0.19	0.1
$V_{T-}$ (V)	−0.1	−0.19	−0.19	−0.1
${\mu }_v({\mathrm{m}}^2 {\mathrm{s}}^{-1} {\Omega }^{-1})$	$1.6\times {10}^{-18}$	$1.6\times {10}^{-17}$	$1.6\times {10}^{-17}$	$1.6\times {10}^{-17}$
$i_{on}$ (A)	1	1	1	1
$i_{off}$ (A)	$1\times {10}^{-5}$	$1\times {10}^{-5}$	$1\times {10}^{-5}$	$1\times {10}^{-5}$
$i_0$ (A)	$1\times {10}^{-3}$	$1\times {10}^{-3}$	$1\times {10}^{-3}$	$1\times {10}^{-3}$
p	10	10	10	10

. The curve changes of the memristor are shown in Figure 1. If the voltage applied to both ends of the memristor is positive and exceeds its threshold voltage, the memristance will decrease. The variation of the memristance varies with different parameters. When the voltage applied to both ends of the memristor is negative and exceeds its threshold voltage, the memristance will increase.

3. Multimodal Mood Consistency and Mood Dependency

The research on neural networks for multimodal mood consistency and mood dependency in this paper is shown in Figure 2. Positive auditory and visual information is expressed by PA and PV. Negative auditory and visual information is represented by NA and NV. Positive and negative emotions are expressed by PE and NE. Process a is the stage of premise. The pleasant bell is the positive auditory information; the cake is the positive visual information; the skull is the negative visual information that frightens people; the thunder and lightning that frighten people represent the negative auditory information. The individual can learn the learning materials of different senses in a period of time. Process b is a single-channel of positive mood consistency. When the individual is in a happy state of mind, the learning efficiency of learning melodious bell materials is higher than that of the normal state of mind. The learning efficiency is lower than the normal state of mind when learning the frightening thunder and sound materials. The learning efficiency of visual materials that make people afraid is lower than that of the normal mood. The learning efficiency of visual materials that make people happy is higher than that of the normal mood. Process c is the mood consistency of single-channel negative emotions. When the individual is in a sad state of mind, the learning efficiency of the cheerful auditory materials is lower than that of the normal state of mind. The learning efficiency of negative auditory and visual materials is higher than that of the normal mood. When learning visual materials that make individuals happy, the learning efficiency is lower than that of normal emotion. Process d is the dual-channel mood consistency. When the individual is in a happy mood and learning positive auditory materials and positive visual materials, the learning efficiency will be higher than that of the single-channel and normal mood. When learning negative auditory and visual materials, the learning efficiency is lower than that of the single-channel and normal mood. When individual learns positive auditory and visual materials in a sad state of mind, the learning efficiency is lower than that in the single-channel and normal state of mind. When learning negative auditory and visual materials, the learning efficiency is higher than that of the single-channel and normal mood. Process e is the mood dependency. After learning the positive materials in a happy mood, the positive materials can be recalled in a happy mood. A happy mood recalls more content and for a longer time than the normal mood.

4. Circuit Design

4.1. Voltage Control Module

The design of the voltage control module is shown in Figure 3, which is used to judge and identify external signals. $N_1$ is the input signal of the learning materials with different properties. If the input signal voltage is 2.1 V, it is positive auditory information. If the input signal voltage is 2.5 V, it is positive visual information. If the voltage of the input signal is 4.8 V, it is a positive visual and auditory input at the same time. If the input signal is negative, it is a negative information input. $S_1$ and $S_2$ are voltage-controlled switches, where the conduction voltage of $S_1$ is 2.9 V and that of $S_2$ is 2.4 V. $O{P}_1$ and $V_1$ are formed into a comparator to ensure that the voltage output by $S_1$ is greater than 2.9 V. The input signal and the output value of $O{P}_1$ pass through the NAND gate $D_1$ and the AND gate $D_2$, and only the auditory signal can be output. The role of $S_2$ is to be turned on during the combination of visual and auditory signals, where the XOR gate $D_3$ is used to prevent the conflict between the auditory signal and the audio–visual signal. The voltage control switch $S_3$ detects whether the auditory signal and the audio–visual signal are lower than the normal value to prevent misjudgment. The memristor $M_1$, operational amplifier $O{P}_2$, and resistor $R_6$ are formed into a proportional operational amplifier and output to the synaptic neuron module. The output voltage of $U_2$ is $U_2=V_{OP1}=-(R_6/M_1)\times V_2$. The voltage of the voltage control switch $S_4$ is 2.4 V, which is used to detect the visual signals. The output voltage of $S_4$ enters two memristors in a reverse parallel manner. The output voltage value of $U_1$ is $U_1=V_{OP3}=-((R_5\times (M_1+M_2))/(M_1\times M_2))\times V_3$. The auditory signal and the visual signal are output to the emotion-generation module through the AND gate $D_4$ and the XOR gate $D_5$.

4.2. Emotion Generation Module

The circuit design of the emotion module is shown in Figure 4. Different emotions with memristors as the core are expressed. $N_3$ is the input signal of different emotional states, and $N_1$ is the input signal of materials with different properties. When the signal input by $N_3$ is a positive mood, $T_5$ will be turned on and $V_6$ will be output. When the signal input by $N_3$ is a negative mood, $T_4$ will be turned on and $V_5$ will be output. $T_6$ and $T_7$ are used for the reverse output of the direction signal. The output voltages of $V_B$ and $V_C$ are $V_B=R_{18}\times V_5/R_{18}+M_7$ and $V_C=M_8\times V_6/R_{20}+M_8$, respectively. When the voltage value of $V_C$ exceeds the threshold voltage of $S_6$, the signal of $V_C$ is output and positive emotions are generated. The signal of $V_B$ is input into memristor $M_9$ and the IN2 port of the mathematical device ABM. The output value of $O{P}_6$ is $V_{OP6}=-(R_{19}/M_9)\times V_B$ and is input into the IN1 of the ABM. The output voltage of the ABM is $V_{ABM}=-V_{IN2}/V_{IN1}\times 6$. $U_5$ is input into the emotion generation module in the presence of the auditory signal and the visual signal. If $U_5$ is at a high level, the voltage $U_4$ passing through $S_3$ is $U_4=V_7+V_{ABM}$. Negative emotions are generated and output to the synaptic neuron module.

4.3. Synaptic Neuron Module

The circuit design of the synaptic neuron module is shown in Figure 5, where $N_2$ is the signal for individual learning. The auditory and visual signals and different emotions are connected through an adder composed of $R_7,R_8,R_9,R_{10},R_{11}$, and $O{P}_4$. The output signal of $O{P}_4$ is transmitted to the memristor $M_4$. $M_4$, $R_{12}$, and $O{P}_5$ are composed of the proportional operation, and the output voltage $V_{OP5}$ is $V_{OP5}=-(R_{12}/M_4)\times V_{OP4}$. The output value of $O{P}_5$ enters $R_{13}$ and $T_1$ through the absolute value module, and the capacitor $C_1$ will be charged. When the voltage of $C_1$ exceeds the threshold voltage of $T_2$, $T_2$ is off and $T_3$ is on. Memristors $M_5$ and $M_6$ are connected in a reverse parallel manner, and $R_{17}$ is used to process the input voltage; finally, the learning signal is obtained. If there is no input signal from the synaptic neuron module, $T_1$ will be connected to discharge $C_1$, and the synaptic neuron circuit will return to its initial state. If the input voltage time is too short and the voltage of $C_1$ does not reach the threshold voltage of 1 V, the circuit will not be activated, indicating that the synaptic neuron circuit can effectively prevent the instantaneous pulse interference of the circuit.

4.4. Complete Circuit

The complete circuit design is shown in Figure 6, in which the voltage control module, the emotion generation module, and the synaptic neuron module are connected to each other. The designed circuit realizes the mood consistency and mood dependency of single the-channel and dual-channel and was verified by Pspice.

5. Simulation and Analysis

5.1. Learning the Initial Situation

The simulation results with only learning materials and no emotional participation are shown in Figure 7. The red line, the blue line, and the green line represent only auditory materials, only visual materials, and audio–visual materials, respectively. The voltage of the auditory signal $U_2$ increases continuously when the auditory material is repeatedly learned. The memristance of memristor $M_1$ decreases continuously until the voltage at the $U_A$ point is greater than 1 V, and the auditory information was learned in 110 s. Similarly, the voltage of $U_1$ increases continuously when the visual materials are learned, and the rate of change is greater than that of the auditory signals. The memristance of memristor $M_2$ also decreases with the change of $U_1$. The visual information was learned in 102 s when the voltage of the $U_A$ point was less than 1 V. The learning efficiency of visual and auditory dual-channel materials is obviously higher than that of single-channel materials. $U_A$ changes faster when audio–visual materials are combined, and the materials were learned in 38 s.

5.2. Single-Channel Mood Consistency

5.2.1. Different Emotional Effects of Single-Channel Positive Materials

The simulation diagram of single-channel passive visual information and passive auditory information learning in a positive emotion and a negative emotion is shown in Figure 8 and Figure 9. Under the influence of a positive emotion, the learning efficiency of positive visual and auditory information will be improved according to the consistency of the mood. In Figure 8, $U_3$ is the signal of the positive emotion, promoting the learning of positive information. The voltage of $U_A$ changes faster than that without emotional influence. Positive auditory materials need 103 s to be learned, and positive visual materials need 86 s to be learned, which are faster than learning without emotional influence. In Figure 9, $U_4$ is the signal of the negative emotion, which will inhibit the learning of positive information. The voltage of $U_A$ changes more slowly than that without emotional influence. Positive auditory materials need 125 s to be learned, and positive visual materials need 118 s to be learned, which are slower than learning without emotional influence.

5.2.2. Different Emotional Effects of Single-Channel Negative Materials

The simulation diagram of single-channel negative auditory information and negative visual information learning under the influence of positive emotions and negative emotions is shown in Figure 10 and Figure 11. In Figure 10, $U_1$ is the auditory signal, $U_3$ is the signal of the negative emotion, and $U_4$ is the signal of the positive emotion. The positive emotion will reduce the learning efficiency of negative auditory information, and the voltage of $U_A$ changes more slowly than when there is no emotional impact. The positive auditory materials needed 122 s to be learned, which is slower than learning without emotional influence. The negative emotion can improve the learning efficiency of negative auditory information, and the voltage of $U_A$ changes faster than that without the influence of emotion. The negative auditory materials needed 103 s to be learned, which is faster than learning without emotional influence.

In Figure 11, $U_1$ is the visual signal, $U_3$ is the signal of the negative emotion, and $U_4$ is the signal of the positive emotion. Similarly, the positive emotion will reduce the learning efficiency of negative visual information, and positive visual materials needed 136 s to be learned. The negative emotion will improve the learning efficiency of negative visual information, and negative auditory materials needed 89 s to be learned.

5.3. Dual-Channel Mood Consistency

5.3.1. Different Emotional Effects of Dual-Channel Positive Materials

The simulation diagram of active material learning under the combination of visual and auditory channels is shown in Figure 12. $U_3$ is the signal of the positive emotion, which can promote the learning of positive materials. Under the influence of positive emotion, the voltage increase of $U_A$ is obvious. The learning time of the channel of audio–visual integration under positive emotion was far less than that of the single-channel, and the materials of audio–visual integration were learned in 28 s. The simulation results in the negative environment are shown in Figure 13. $U_4$ is the signal of the negative emotion, which can inhibit the learning of positive materials. Under the influence of the negative emotion, the voltage of $U_A$ rises slowly. The learning time of the channel of the audio–visual combination under the negative emotion was far less than that of the single-channel, and the materials of the audio–visual combination were learned in 54 s.

5.3.2. Different Emotional Effects of Dual-Channel Negative Materials

The simulation results of dual-channel negative materials in positive and negative emotion learning are shown in Figure 14. $U_1$ and $U_2$ are negative visual and auditory signals, respectively, and $U_3$ and $U_4$ are positive emotional signals and negative emotional signals, respectively. The negative emotion can promote the learning of dual-channel negative materials, and the voltage of $U_A$ rises rapidly. Under the mechanism of mood consistency, negative materials were learned in 66 s. The positive emotion can inhibit the learning of the negative audio–visual materials, and the voltage speed increase of $U_A$ is slower. When the mood is inconsistent, the learning time of negative materials will be longer, which was 24 s.

5.4. Mood Consistency and Mood Dependency

The learning cycle data in different states are shown in

Table 2. Learning cycle data in different states.

No.	Normal State	Positive Emotions	Negative Emotions
1	102 s	86 s	118 s
2	110 s	103 s	125 s
3	38 s	28 s	54 s
4	/	136 s	89 s
5	/	122 s	103 s
6	/	66 s	24 s

1–6 are, respectively, positive single-channel visual signals, positive single-channel auditory signals, positive dual-channel visual and auditory signals, negative single-channel visual signals, negative single-channel auditory signals, negative dual-channel visual and auditory signals.

. The simulation results of mood dependency are shown in Figure 15. Mood consistency is the learning stage. If the normal emotions and learning materials are the same, the learning efficiency will be improved. Mood dependency is the forgetting stage. If the mental state of memory is the same as that of learning, the individual can recall more than normal. In 25 s, when there was only a negative emotion, the resistance of memristor $M_4$ started to rise. The learning content is recalled when there is only a negative emotion. After a period of time, when the voltage value of $U_A$ is less than the threshold voltage of $T_2$, the learning content is forgotten.

As shown in

Table 3. Comparison of the functions implemented by several different works.

Work	Memristor Circuit	Mood Consistency	Mood Dependency	Dual Channel Audio–Visual Integration
[17]	No	Yes	No	No
[21]	Yes	No	No	No
[22]	Yes	No	No	No
[23]	Yes	No	No	No
[19]	Yes	Yes	No	No
this work	Yes	Yes	Yes	Yes

, the functions realized by different works were compared. Compared with the above work [17,19,21,22,23], more functions were realized in our memristor circuit. For example, by discussing the role of the single-channel and dual-channel in mood consistency, we also considered the function of mood dependency.

6. Conclusions

In this paper, a neural network circuit based on memristors for multimode mood consistency and mood dependency was designed. Single-channel visual and auditory mood consistency were achieved, and dual-channel mood consistency was also considered. In addition, the function of mood dependency was realized through a memristor circuit. However, although mood consistency and mood dependency were realized, the influence of external factors on mood dependency was not taken into account. The influence between more channels of coordinated learning and emotion needs further exploration and research. Therefore, we need further research on mood dependency to provide a reference for further brain-like neural networks.