Deep Generative Models
A special issue of Entropy (ISSN 1099-4300). This special issue belongs to the section "Signal and Data Analysis".
Deadline for manuscript submissions: closed (30 November 2020)
Special Issue Editors
2. International Laboratory of Stochastic Algorithms and High-Dimensional Inference, National Research University Higher School of Economics, 101000 Moskva, Russia
Interests: statistical machine learning; computational statistics; Markov Chains Monte Carlo; stochastic approximation; Bayesian statistics; statistical signal processing; information theory
Interests: Limit Theorems, Probability, Statistics, Number Theory; high dimensional data analysis; random matrices
Special Issue Information
Dear Colleagues,
Generative models (GM) aim at learning a probabilistic model for high dimensional observations using unsupervised learning techniques. Generative models are used to sample new data points which have the same statistical characteristics as the learning data, but which are not "mere copies". A generative model must capture dependency structures and therefore generalize from the training examples. Although the idea of generative models has long been used, it has achieved tremendous success in just a few years with the introduction of deep neural networks.
Learning deep generative models that are capable of capturing intricate dependence structures from vast amounts of unlabeled data presently appear as one of the major challenges of AI. Recently, different approaches to achieving this goal have been proposed.
A first class of approach is based on the minimization of the cross-entropy (Kullback–Leibler divergence) between the distribution of observations and a model parameterized using neural nets defining an energy-function or a connecting generative with energy-based models (EBM). This approach is appealing yet poses massive computational problems linked to the need to estimate the normalizing constant of the EBM and its gradient. Recent advances have been made in this area, combining complex potential generated by deep neural networks with novel Monte Carlo methods by Markov chains that scale with the dimension of the models and the volume of the data.
A second class of approaches consists in using variational autoencoders (VAE), which combines the principle of auto-encoders (methods capable of learning representations of much smaller dimensions than observations) and variational inference methods. VAEs jointly learn both an algorithm for generating samples from the distribution and the latent space of a much smaller dimension than the observations that summarize the distribution of the observations. VAEs aim to minimize the reconstruction error while regularizing the distribution of the latent representation to match some parametric prior. The ability of VAEs to produce complex distributions is deeply connected with the set of mappings that neural network functions can feasibly learn and by the choice of the variational family which should be “simple enough” for the inference to be reasonably straightforward. Despite recent progresses, there are still a lot to do to make VAEs more effective.
A third class of methods are the generative adversarial networks or GANs, a very elegant idea (described as “the coolest idea in machine learning in the last twenty years” by Y. Le Cun). In the GAN approach, a generative network and a discriminative network compete in a zero-sum game. The generative network produces new data points (using either an energy-based model or a VAE), while the discriminative network tries to discriminate the newly produced data from the training data. Here again, the approaches have been numerous and the progresses achieved in the last 5 years are impressive.
There has been a huge amount of work for generative models, but most of the efforts have been devoted to the i.i.d. scenario (independent sequence of high-dimensional observations), while generative models for sequences (time series) remain much less developed. There have recently been some attempts, but the existing approaches are far from completely satisfactory, and a lot remains to be done.
The objective of this Special Issue is to provide an overview of deep generative methods covering energy-based models, variational auto-encoders, and generative adversarial networks, both in the i.i.d. and the dependent case. The contributions for this Special Issue can be either surveys of recent results in this field, or original theoretical or methodological works. We also wish to open this Special Issue to the applications of generative models and benchmarks of different methods (which are very useful given the current profusion of works in this field).
Prof. Eric Moulines
Prof. Alexey Naumov
Guest Editors
Manuscript Submission Information
Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.
Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Entropy is an international peer-reviewed open access monthly journal published by MDPI.
Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 2600 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.
Keywords
- generative models
- variational auto-encoders
- evidence lower bounds
- energy-based models
- deep latent variable models
- maximum entropy
- macrocanonical sampling
- nonlinear filtering
- (deep) Kalman filter
- generative adversarial network
- scalable Monte Carlo inference
- Monte Carlo Markov chain
- generative models for reinforcement learning (planning, exploration, model-based TL)
- applications of generative models (proteomics, drug discovery, high-energy physics)
