Multimodal Sentiment Analysis Based on Deep Learning Methods Such as Convolutional Neural Networks

Prompt learning has improved the performance of language models by reducing the gap in language model training methods of pre-training and downstream tasks. However, extending prompt learning in language models pre-trained with unimodal data to multimodal sources is difficult as it requires additional deep-learning layers that cannot be attached. In the natural-language emotion-recognition task, improved emotional classification can be expected when using audio and text to train a model rather than only natural-language text. Audio information, such as voice pitch, tone, and intonation, can give more information that is unavailable in text to predict emotions more effectively. Thus, using both audio and text can enable better emotion prediction in speech emotion-recognition models compared to semantic information alone. In this paper, in contrast to existing studies that use multimodal data with an additional layer, we propose a method for improving the performance of speech emotion recognition using multimodal prompt learning with text-based pre-trained models. The proposed method is using text and audio information in prompt learning by employing a language model pre-trained on natural-language text. In addition, we propose a method to improve the emotion-recognition performance of the current utterance using the emotion and contextual information of the previous utterances for prompt learning in speech emotion-recognition tasks. The performance of the proposed method was evaluated using the English multimodal dataset MELD and the Korean multimodal dataset KEMDy20. Experiments using both the proposed methods obtained an accuracy of 87.49%, $F_1$ score of 44.16, and weighted $F_1$ score of 86.28. Full article

Most of the existing quantum-inspired models are based on amplitude-phase embedding to model natural language, which maps words into Hilbert space. In quantum-computing theory, the vectors corresponding to quantum states are all complex values, so there is a gap between these two areas. Presently, complex-valued neural networks have been studied, but their practical applications are few, let alone in the downstream tasks of natural language processing such as sentiment analysis and language modeling. In fact, the complex-valued neural network can use the imaginary part information to embed hidden information and can express more complex information, which is suitable for modeling complex natural language. Meanwhile, quantum-inspired models are defined in Hilbert space, which is also a complex space. So it is natural to construct quantum-inspired models based on complex-valued neural networks. Therefore, we propose a new quantum-inspired model for NLP, ComplexQNN, which contains a complex-valued embedding layer, a quantum encoding layer, and a measurement layer. The modules of ComplexQNN are fully based on complex-valued neural networks. It is more in line with quantum-computing theory and easier to transfer to quantum computers in the future to achieve exponential acceleration. We conducted experiments on six sentiment-classification datasets comparing with five classical models (TextCNN, GRU, ELMo, BERT, and RoBERTa). The results show that our model has improved by 10% in accuracy metric compared with TextCNN and GRU, and has competitive experimental results with ELMo, BERT, and RoBERTa. Full article

Traditional product evaluation research is to collect data through questionnaires or interviews to optimize product design, but the whole process takes a long time to deploy and cannot fully reflect the market situation. Aiming at this problem, we propose a product evaluation prediction model based on multi-level deep feature fusion of online reviews. It mines product satisfaction from the massive reviews published by users on e-commerce websites, and uses this model to analyze the relationship between design attributes and customer satisfaction, design products based on customer satisfaction. Our proposed model can be divided into the following four parts: First, the DSCNN (Depthwise Separable Convolutions) layer and pooling layer are used to combine extracting shallow features from the primordial data. Secondly, CBAM (Convolutional Block Attention Module) is used to realize the dimension separation of features, enhance the expressive ability of key features in the two dimensions of space and channel, and suppress the influence of redundant information. Thirdly, BiLSTM (Bidirectional Long Short-Term Memory) is used to overcome the complexity and nonlinearity of product evaluation prediction, output the predicted result through the fully connected layer. Finally, using the global optimization capability of the genetic algorithm, the hyperparameter optimization of the model constructed above is carried out. The final forecasting model consists of a series of decision rules that avoid model redundancy and achieve the best forecasting effect. It has been verified that the method proposed in this paper is better than the above-mentioned models in five evaluation indicators such as MSE, MAE, RMSE, MAPE and SMAPE, compared with Support Vector Regression (SVR), DSCNN, BiLSTM and DSCNN-BiLSTM. By predicting customer emotional satisfaction, it can provide accurate decision-making suggestions for enterprises to design new products. Full article