Mathematical Foundations of Deep Neural Networks

A special issue of Mathematics (ISSN 2227-7390). This special issue belongs to the section "Mathematics and Computer Science".

Deadline for manuscript submissions: closed (31 May 2023) | Viewed by 11781

Special Issue Editors

School of Software Engineering, South China University of Technology, Guangzhou 510006, China
Interests: deep Learning and computer vision applications
School of Software Engineering, South China University of Technology, Guangzhou 510006, China
Interests: AI chips and systems
Department of Computer Science, Harbin Institute of Technology, Shenzhen 518055, China
Interests: deep Learning; spatial-temporal data mining

Special Issue Information

Dear Colleagues,

Deep learning is popular in various domains, e.g., computer vision, natural language processing, financing, medical applications, etc. Modern deep learning models include a huge number of parameters and demand a high computation power. One of the fundamental issues of machine learning models is their mathematical foundations, which provide insights into the model explanation, model design and network architecture search. In this Special Issue, topics related to the mathematical foundation of deep learning are welcomed, including but not limited to: geometric, topological, Bayesian, and game-theoretic formulations; analytical approaches to exploiting optimal transport theory, optimization theory, approximation theory, information theory, dynamical systems, partial differential equations, and mean field theory; exploring efficient training with small data sets, adversarial learning, reinforcement learning, and closing the decision-action loop; and foundational work on understanding success metrics, privacy safeguards, causal inference, algorithmic fairness, uncertainty quantification, interpretability, and reproducibility.

Prof. Dr. Qingyao Wu
Prof. Dr. Xiaohang Wang
Prof. Dr. Xutao Li
Guest Editors

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Mathematics is an international peer-reviewed open access semimonthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 2600 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Keywords

  • explainable AI
  • model compression
  • network architecture search
  • game-theoretic formulations
  • optimal transport theory
  • optimization theory
  • approximation theory
  • information theory
  • zero/few-shot learning
  • open set learning
  • adversarial learning
  • reinforcement learning
  • metric learning
  • privacy safeguards
  • causal inference
  • algorithmic fairness
  • uncertainty quantification
  • model interpretability

Published Papers (4 papers)

Order results
Result details
Select all
Export citation of selected articles as:

Research

Jump to: Review

15 pages, 348 KiB  
Article
Mathematical Expressiveness of Graph Neural Networks
by Guillaume Lachaud, Patricia Conde-Cespedes and Maria Trocan
Mathematics 2022, 10(24), 4770; https://doi.org/10.3390/math10244770 - 15 Dec 2022
Cited by 3 | Viewed by 2310
Abstract
Graph Neural Networks (GNNs) are neural networks designed for processing graph data. There has been a lot of focus on recent developments of graph neural networks concerning the theoretical properties of the models, in particular with respect to their mathematical expressiveness, that is, [...] Read more.
Graph Neural Networks (GNNs) are neural networks designed for processing graph data. There has been a lot of focus on recent developments of graph neural networks concerning the theoretical properties of the models, in particular with respect to their mathematical expressiveness, that is, to map different graphs or nodes to different outputs; or, conversely, to map permutations of the same graph to the same output. In this paper, we review the mathematical expressiveness results of graph neural networks. We find that according to their mathematical properties, the GNNs that are more expressive than the standard graph neural networks can be divided into two categories: the models that achieve the highest level of expressiveness, but require intensive computation; and the models that improve the expressiveness of standard graph neural networks by implementing node identification and substructure awareness. Additionally, we present a comparison of existing architectures in terms of their expressiveness. We conclude by discussing the future lines of work regarding the theoretical expressiveness of graph neural networks. Full article
(This article belongs to the Special Issue Mathematical Foundations of Deep Neural Networks)
Show Figures

Figure 1

15 pages, 3617 KiB  
Article
Dual Attention Multiscale Network for Vessel Segmentation in Fundus Photography
by Pengshuai Yin, Yupeng Fang and Qilin Wan
Mathematics 2022, 10(19), 3687; https://doi.org/10.3390/math10193687 - 08 Oct 2022
Viewed by 1136
Abstract
Automatic vessel structure segmentation is essential for an automatic disease diagnosis system. The task is challenging due to vessels’ different shapes and sizes across populations. This paper proposes a multiscale network with dual attention to segment various retinal blood vessels. The network injects [...] Read more.
Automatic vessel structure segmentation is essential for an automatic disease diagnosis system. The task is challenging due to vessels’ different shapes and sizes across populations. This paper proposes a multiscale network with dual attention to segment various retinal blood vessels. The network injects a spatial attention module and channel attention module on a feature map, whose size is one-eighth of the input size. The network also uses multiscale input to receive multi-level information, and the network uses the multiscale output to gain more supervision. The proposed method is tested on two publicly available datasets: DRIVE and CHASEDB1. The accuracy, AUC, sensitivity, and specificity on the DRIVE dataset are 0.9615, 0.9866, 0.7709, and 0.9847, respectively. On the CHASEDB1 dataset, the metrics are 0.9800, 0.9892, 0.8215, and 0.9877, respectively. The ablative study further shows effectiveness for each part of the network. Multiscale and dual attention mechanism both improve performance. The proposed architecture is simple and effective. The inference time is 12 ms on a GPU and has potential for real-world applications. The code will be made publicly available. Full article
(This article belongs to the Special Issue Mathematical Foundations of Deep Neural Networks)
Show Figures

Figure 1

13 pages, 321 KiB  
Article
MST-RNN: A Multi-Dimension Spatiotemporal Recurrent Neural Networks for Recommending the Next Point of Interest
by Chunshan Li, Dongmei Li, Zhongya Zhang and Dianhui Chu
Mathematics 2022, 10(11), 1838; https://doi.org/10.3390/math10111838 - 27 May 2022
Cited by 5 | Viewed by 1457
Abstract
With the increasing popularity of location-aware Internet-of-Vehicle services, the next-Point-of-Interest (POI) recommendation has gained significant research interest, predicting where drivers will go next from their sequential movements. Many researchers have focused on this problem and proposed solutions. Machine learning-based methods (matrix factorization, Markov [...] Read more.
With the increasing popularity of location-aware Internet-of-Vehicle services, the next-Point-of-Interest (POI) recommendation has gained significant research interest, predicting where drivers will go next from their sequential movements. Many researchers have focused on this problem and proposed solutions. Machine learning-based methods (matrix factorization, Markov chain, and factorizing personalized Markov chain) focus on a POI sequential transition. However, they do not recommend the user’s position for the next few hours. Neural network-based methods can model user mobility behavior by learning the representations of the sequence data in the high-dimensional space. However, they just consider the influence from the spatiotemporal dimension and ignore many important influences, such as duration time at a POI (Point of Interest) and the semantic tags of the POIs. In this paper, we propose a novel method called multi-dimension spatial–temporal recurrent neural networks (MST-RNN), which extends the ST-RNN and exploits the duration time dimension and semantic tag dimension of POIs in each layer of neural networks. Experiments on real-world vehicle movement data show that the proposed MST-RNN is effective and clearly outperforms the state-of-the-art methods. Full article
(This article belongs to the Special Issue Mathematical Foundations of Deep Neural Networks)
Show Figures

Figure 1

Review

Jump to: Research

37 pages, 548 KiB  
Review
Survey of Optimization Algorithms in Modern Neural Networks
by Ruslan Abdulkadirov, Pavel Lyakhov and Nikolay Nagornov
Mathematics 2023, 11(11), 2466; https://doi.org/10.3390/math11112466 - 26 May 2023
Cited by 4 | Viewed by 5899
Abstract
The main goal of machine learning is the creation of self-learning algorithms in many areas of human activity. It allows a replacement of a person with artificial intelligence in seeking to expand production. The theory of artificial neural networks, which have already replaced [...] Read more.
The main goal of machine learning is the creation of self-learning algorithms in many areas of human activity. It allows a replacement of a person with artificial intelligence in seeking to expand production. The theory of artificial neural networks, which have already replaced humans in many problems, remains the most well-utilized branch of machine learning. Thus, one must select appropriate neural network architectures, data processing, and advanced applied mathematics tools. A common challenge for these networks is achieving the highest accuracy in a short time. This problem is solved by modifying networks and improving data pre-processing, where accuracy increases along with training time. Bt using optimization methods, one can improve the accuracy without increasing the time. In this review, we consider all existing optimization algorithms that meet in neural networks. We present modifications of optimization algorithms of the first, second, and information-geometric order, which are related to information geometry for Fisher–Rao and Bregman metrics. These optimizers have significantly influenced the development of neural networks through geometric and probabilistic tools. We present applications of all the given optimization algorithms, considering the types of neural networks. After that, we show ways to develop optimization algorithms in further research using modern neural networks. Fractional order, bilevel, and gradient-free optimizers can replace classical gradient-based optimizers. Such approaches are induced in graph, spiking, complex-valued, quantum, and wavelet neural networks. Besides pattern recognition, time series prediction, and object detection, there are many other applications in machine learning: quantum computations, partial differential, and integrodifferential equations, and stochastic processes. Full article
(This article belongs to the Special Issue Mathematical Foundations of Deep Neural Networks)
Show Figures

Figure 1

Back to TopTop