A Factor Analysis Perspective on Linear Regression in the ‘More Predictors than Samples’ Case

Abstract

Linear regression (LR) is a core model in supervised machine learning performing a regression task. One can fit this model using either an analytic/closed-form formula or an iterative algorithm. Fitting it via the analytic formula becomes a problem when the number of predictors is greater than the number of samples because the closed-form solution contains a matrix inverse that is not defined when having more predictors than samples. The standard approach to solve this issue is using the Moore–Penrose inverse or the L2 regularization. We propose another solution starting from a machine learning model that, this time, is used in unsupervised learning performing a dimensionality reduction task or just a density estimation one—factor analysis (FA)—with one-dimensional latent space. The density estimation task represents our focus since, in this case, it can fit a Gaussian distribution even if the dimensionality of the data is greater than the number of samples; hence, we obtain this advantage when creating the supervised counterpart of factor analysis, which is linked to linear regression. We also create its semisupervised counterpart and then extend it to be usable with missing data. We prove an equivalence to linear regression and create experiments for each extension of the factor analysis model. The resulting algorithms are either a closed-form solution or an expectation–maximization (EM) algorithm. The latter is linked to information theory by optimizing a function containing a Kullback–Leibler (KL) divergence or the entropy of a random variable.

1. Introduction

In machine learning, models can be grouped into two categories: probabilistic and nonprobabilistic. Probabilistic models can be classified as generative and discriminative [1]. Examples of classic generative models are naive Bayes and Gaussian mixture models (GMM). Examples of classic discriminative models are linear regression (LR) and logistic regression. The key difference is whether they model the joint probability of the input and the output—generative models—or they just model the conditional probability of the output given the input—discriminative models. For a classification or a regression task, one may argue that what you need is just a discriminative model, but the generative models have their advantages: they can sometimes handle missing data, can easily generate new data, can be extended to be unsupervised or semisupervised, etc. ([2] p. 268).

As one may notice, there are generative models for unsupervised learning that have counterparts in supervised learning, even though this is not widely discussed in the literature. One such example is the GMM ([2] p. 339) with its counterpart, the Gaussian joint Bayes model ([2] p. 102), also known as quadratic discriminant analysis. Their training/fitting algorithms are similar, as one may notice, for example, in [3,4]:

• for Gaussian joint Bayes:

  $${\pi }_j=\frac{1}{n}\sum _{i=1}^n{1}_{\{z_i=j\}}$$

  $${\mu }_j=\frac{{\sum }_{i=1}^n{1}_{\{z_i=j\}}{x}_i}{{\sum }_{i=1}^n{1}_{\{z_i=j\}}}$$

  $${\Sigma }_j=\frac{{\sum }_{i=1}^n{1}_{\{z_i=j\}}(x_i-{\mu }_j){(x_i-{\mu }_j)}^{\top }}{{\sum }_{i=1}^n{1}_{\{z_i=j\}}}$$

  where $x_i$ is an input observation, (${\pi }_j,{\mu }_j,{\Sigma }_j$) are the parameters of a GMM, observable $z_i$ is the class index corresponding to $x_i$, j is a class index, and $1_{\{z_i=j\}}$ is the indicator function, which returns 1 if the condition $z_i=j$ is true and 0 otherwise.
• for expectation–maximization (EM) for the GMM, which optimizes a function concerning a Kullback–Leibler (KL) divergence or the entropy of a random variable—check Appendix A for these details—:
  E step:

  $$\begin{array}{cc}\hfill w_{ij}& =p(z_i=j|x_i;{\pi }^{\prime },{\mu }^{\prime },{\Sigma }^{\prime })\hfill \\ \hfill & =\frac{p\left(x_i\right|z_i=j;{\mu }^{\prime },{\Sigma }^{\prime })p(z_i=j;{\pi }^{\prime })}{{\sum }_{l=1}^{K}p\left(x_i\right|z_i=l;{\mu }^{\prime },{\Sigma }^{\prime })p(z_i=l;{\pi }^{\prime })}\hfill \end{array}$$
  M step:

  $${\pi }_j=\frac{1}{n}\sum _{i=1}^n{w}_{ij}$$

  $${\mu }_j=\frac{{\sum }_{i=1}^n{w}_{ij}{x}_i}{{\sum }_{i=1}^n{w}_{ij}}$$

  $${\Sigma }_j=\frac{{\sum }_{i=1}^n{w}_{ij}(x_i-{\mu }_j){(x_i-{\mu }_j)}^{\top }}{{\sum }_{i=1}^n{w}_{ij}}$$

  where $x_i$ is an input observation, (${\pi }_j,{\mu }_j,{\Sigma }_j$) are the parameters of a GMM, unobservable $z_i$ is the cluster index corresponding to $x_i$, j is a cluster index, and $w_{ij}$ is the probability that $x_i$ belongs to cluster j.

This similarity between GMM and Gaussian joint Bayes is intriguing; hence, we decided to further explore this aspect but starting from other supervised–unsupervised counterparts. As a result, we changed the root model into factor analysis (FA) [5] ([2] p. 381), which is normally used for dimensionality reduction or for density estimation when the dimensionality of the data is greater than the number of samples. Factor analysis is a Gaussian generative model used in unsupervised learning. We aimed at creating its supervised counterpart in order to handle a regression task and then exploit it as much as possible.

After creating the supervised counterpart, we proved a significant property, namely that linear regression is equivalent to (supervised) factor analysis—with one-dimensional latent space—when no constraints are imposed on the covariance matrices.

A linear regression model can be fitted via a closed-form solution or an iterative algorithm. When the number of predictors is greater than the number of samples, there is no closed-form solution. There are other solutions to this problem, as we will see.

We were at the point where we knew that factor analysis was linked to linear regression and that it could be used when the number of samples was lower than the dimensionality of the data—from now on, this is denoted as $D>>n$ or $n<<D$. As a result, we shifted our focus from solely exploiting the factor analysis model to highlighting novel linear regression versions applicable in the $D>>n$ regime—linear regression being a widely known and used model—:

• linear regression when $D>>n$,
• semisupervised linear regression when $D>>n$,
• (semisupervised) linear regression when $D>>n$ with missing data.

The structure of this paper is as follows. In Section 2, we include some theoretical background to enhance the readability of this paper. Section 3 contains related work. In Section 4, we include the models we proposed, starting from factor analysis. Section 5 contains experiments using the proposed models. In Section 6, we conclude the paper and show future directions.

We include the full algorithms in the appendices in a pseudocode format, two of them being instances of the expectation–maximization schema.

2. Theoretical Background

We started our analysis from two core models in machine learning: linear regression and factor analysis. We will discuss the aspects of those two models that are relevant to understanding the next sections of this paper.

2.1. Linear Regression

Proposition 1.

Let $\left\{(x^{\left(i\right)},y^{\left(i\right)})\right|x^{\left(i\right)}\in {\mathbb{R}}^{D\times 1},y^{\left(i\right)}\in \mathbb{R},i\in \{1,\dots ,n\}\}$ be a data set where D is the dimensionality of the input data, $\left\{x^{\left(i\right)}\right|i\in \{1,\dots ,n\}\}$ is the input, and $\left\{y^{\left(i\right)}\right|i\in \{1,\dots ,n\}\}$ is the output. The linear regression model is as follows:

$$Y^{\left(i\right)}=w{x}^{\left(i\right)}+b+{ϵ}^{\left(i\right)}$$

where $Y^{\left(i\right)}$ is a random variable corresponding to $y^{\left(i\right)}$, ${ϵ}^{\left(i\right)}\sim \mathcal{N}(0,{\sigma }^2)$, $\sigma \in {\mathbb{R}}_{+}^{*}$, $w\in {\mathbb{R}}^{1\times D}$, $b\in \mathbb{R}$. Then, the parameters w and b can be estimated via maximum likelihood as follows:

$$\begin{array}{cc}\hfill {\widehat{w}}_{\mathit{LR}}& =\left(n\overline{y}{\overline{x}}^{\top }-Y{X}^{\top }\right){\left(n\overline{x}{\overline{x}}^{\top }-X{X}^{\top }\right)}^{-1}\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill {\widehat{b}}_{\mathit{LR}}& =\overline{y}-{\widehat{w}}_{\mathit{LR}}\overline{x},\hfill \end{array}$$

(2)

or, equivalently, as follows:

$$\begin{array}{c}\hfill \left[\begin{array}{c}{\widehat{w}}_{\mathit{LR}}^{\top }\\ {\widehat{b}}_{\mathit{LR}}\end{array}\right]={\left(\tilde{X}{\tilde{X}}^{\top }\right)}^{-1}\tilde{X}{Y}^{\top }\end{array}$$

(3)

where $\overline{x}={\displaystyle \frac{x^{\left(1\right)}+\cdots +x^{\left(n\right)}}{n}}$, $\overline{y}={\displaystyle \frac{y^{\left(1\right)}+\cdots +y^{\left(n\right)}}{n}}$,

$X=\left[\begin{array}{ccc}{x}^{\left(1\right)}& \dots & x^{\left(n\right)}\end{array}\right]\in {\mathbb{R}}^{D\times n},$$\tilde{X}=\left[\begin{array}{ccc}{\tilde{x}}^{\left(1\right)}& \dots & {\tilde{x}}^{\left(n\right)}\end{array}\right]\in {\mathbb{R}}^{(D+1)\times n},$

${\tilde{x}}^{\left(i\right)}=\left[\begin{array}{c}{x}^{\left(i\right)}\\ 1\end{array}\right]=\left[\begin{array}{c}{x}_1^{\left(i\right)}\\ ⋮\\ x_D^{\left(i\right)}\\ 1\end{array}\right]\in {\mathbb{R}}^{D+1},$ and $Y=\left[\begin{array}{ccc}{y}^{\left(1\right)}& \dots & y^{\left(n\right)}\end{array}\right]\in {\mathbb{R}}^{1\times n}$.

A potential problem with Equation (3) is when $\tilde{X}{\tilde{X}}^{\top }$ is not invertible. Such case arises when $D+1>n$, i.e., there are more predictors than samples. Two standard solutions to this problem are the following:

• Let $A\in {\mathbb{R}}^{a\times b}$ be a matrix. Then, the Moore–Penrose inverse of A can be defined as $A^{+}={lim}_{\alpha \to 0^{+}}{(A^{\top }A+\alpha I)}^{-1}{A}^{\top }$, which, algorithmically, is computed via the singular value decomposition of A ([6] Section 2.9). One may notice that the matrix ${\left(\tilde{X}{\tilde{X}}^{\top }\right)}^{-1}\tilde{X}$ from Equation (3) is just ${\left({\tilde{X}}^{\top }\right)}^{+}$ when $\tilde{X}{\tilde{X}}^{\top }$ is invertible. When it is not, the solution is to replace the matrix ${\left(\tilde{X}{\tilde{X}}^{\top }\right)}^{-1}\tilde{X}$ from Equation (3) with ${\left({\tilde{X}}^{\top }\right)}^{+}$.
• L2 regularization, which results in ridge regression ([2] p. 225). The matrix ${\left(\tilde{X}{\tilde{X}}^{\top }\right)}^{-1}\tilde{X}$ from Equation (3) is replaced with ${\left(\tilde{X}{\tilde{X}}^{\top }+\left[\begin{array}{cccc}\alpha & \dots & 0& 0\\ ⋮& \ddots & ⋮& ⋮\\ 0& \dots & \alpha & 0\\ 0& \dots & 0& 0\end{array}\right]\right)}^{-1}\tilde{X}$, with $\alpha >0$; the bigger the $\alpha$, the more regularization we add to the model, i.e., move away from overfitting. From this point of view, the first solution using the Moore–Penrose inverse can be interpreted as achieving the asymptotically lowest L2 regularization.

2.2. Factor Analysis

The formulas stated in the conclusion of the following Proposition were proved in [5] and are relevant for the factor analysis algorithm—although the matrix $\Psi$ is considered as being diagonal there, the formulas stay the same even if $\Psi$ is not diagonal.

Proposition 2.

Let us consider the following factor analysis model:

$z\sim \mathcal{N}(0,I)$-latent variable, $z\in {\mathbb{R}}^{d\times 1}$

$x|z\sim \mathcal{N}(\mu +\Lambda z,\Psi )$, $x\in {\mathbb{R}}^{D\times 1},\mu \in {\mathbb{R}}^{D\times 1},\Lambda \in {\mathbb{R}}^{D\times d},\mathit{and}\Psi \in {\mathbb{R}}^{D\times D}$ a diagonal matrix. Then:

$\left[\begin{array}{c}x\\ z\end{array}\right]\sim \mathcal{N}\left(\left[\begin{array}{c}\mu \\ 0\end{array}\right],\left[\begin{array}{cc}\Lambda {\Lambda }^{\top }& \Lambda \\ {\Lambda }^{\top }& I\end{array}\right]\right)$

$x\sim \mathcal{N}(\mu ,\Lambda {\Lambda }^{\top }+\Psi )$

$z|x\sim \mathcal{N}({\Lambda }^{\top }{(\Lambda {\Lambda }^{\top }+\Psi )}^{-1}(x-\mu ),I-{\Lambda }^{\top }{(\Lambda {\Lambda }^{\top }+\Psi )}^{-1}\Lambda ).$

A factor analysis model can be fitted via an EM algorithm using the maximum likelihood estimation (MLE) principle, and factor analysis can be used as a density estimation technique when the dimensionality of the data is greater than the number of samples [5].

For the algorithms we developed, we will let $z\sim \mathcal{N}({\mu }_z,{\Sigma }_z)$ and not $z\sim \mathcal{N}(0,I)$, because z becomes observed data, and we want to learn its parameters, and not impose something unrealistic like $z\sim \mathcal{N}(0,I)$. This generalization leads to the following result.

Proposition 3.

Let us consider the following linear Gaussian system:

$z\sim \mathcal{N}({\mu }_z,{\Sigma }_z)$-latent variable, $z\in {\mathbb{R}}^{d\times 1}$, ${\mu }_z\in {\mathbb{R}}^{d\times 1}$, ${\Sigma }_z\in {\mathbb{R}}^{d\times d}$ a symmetric and positive definite matrix,

$x|z\sim \mathcal{N}(\mu +\Lambda z,\Psi )$, $x\in {\mathbb{R}}^{D\times 1},\mu \in {\mathbb{R}}^{D\times 1},\Lambda \in {\mathbb{R}}^{D\times d}$, and $\Psi \in {\mathbb{R}}^{D\times D}$ a symmetric and positive definite matrix. (If Ψ is a diagonal matrix, then we say that we are in the FA case. If Ψ is a scalar matrix, $\Psi =\eta I$, $\eta \in {\mathbb{R}}_{+}^{*}$, then we are in the probabilistic principal component analysis (PPCA) case [7]. If Ψ is any symmetric and positive definite matrix, then we say we are in the unconstrained factor analysis (UncFA) case. If the first two terms are standard (FA and PPCA), the third one is proposed by us—UncFA.) Then:

$$\begin{array}{c}\left[\begin{array}{c}x\\ z\end{array}\right]\sim \mathcal{N}\left(\left[\begin{array}{c}\mu +\Lambda {\mu }_z\\ {\mu }_z\end{array}\right],\left[\begin{array}{cc}\Lambda {\Sigma }_z{\Lambda }^{\top }+\Psi & \Lambda {\Sigma }_z\\ {(\Lambda {\Sigma }_z)}^{\top }& {\Sigma }_z\end{array}\right]\right)\hfill \\ x\sim \mathcal{N}(\mu +\Lambda {\mu }_z,\Lambda {\Sigma }_z{\Lambda }^{\top }+\Psi )\hfill \\ z|x\sim \mathcal{N}({\mu }_z+{\Sigma }_z{\Lambda }^{\top }{(\Lambda {\Sigma }_z{\Lambda }^{\top }+\Psi )}^{-1}(x-\mu -\Lambda {\mu }_z),\hfill \\ {\Sigma }_z-{\Sigma }_z{\Lambda }^{\top }{(\Lambda {\Sigma }_z{\Lambda }^{\top }+\Psi )}^{-1}\Lambda {\Sigma }_z^{\top }).\hfill \end{array}$$

(4)

The proof can be found in ([8] pp. 9–11).

3. Related Work

Although factor analysis is widely used for dimensionality reduction, its supervised counterpart is, to the best of our knowledge, not present in the literature. What is present is a model called supervised principal component analysis or latent factor regression ([2] p. 405). The idea is that not only the input, for a regression task, is generated by a latent variable, as one applies factor analysis to replace the input in the problem with a low dimensional embedding, but also the output. The key idea is that the purpose of supervised principal component analysis is still dimensionality reduction and not at all regression, which is where we want to push the factor analysis model.

There is also a term called linear Gaussian system ([2] p. 119). This was already presented in Section 2, and it generalizes the factor analysis generative process by considering that z has a learnable mean and covariance matrix, but it does not go further.

Factor analysis is strongly related to principal component analysis (PCA) [9] because by imposing a certain constraint in factor analysis, we get a model called probabilistic principal component analysis [7] that can be fitted using a closed-form solution, which, in an asymptotic case, is also the solution for PCA. Probabilistic PCA can be kernelized using a model called the Gaussian process latent variable model (GPLVM) [10]. This model also has supervised counterparts [11], but, as in the case of FA, the supervised extension targets dimensionality reduction, and the idea is similar to the one in supervised PCA.

4. Proposed Models

In this section, we propose three models starting from the FA model, each in a new subsection: simple-supervised factor analysis (S2.FA), simple-semisupervised factor analysis (S3.FA), and missing simple-semisupervised factor analysis (MS3.FA). While S2.FA is applicable in the supervised case, regression, S3.FA is meant to be used in a semisupervised context. MS3.FA handles missing input data in a (semi)supervised scenario.

One important remark that will not be restated in this paper is that all the models (FA, S2.FA, S3.FA, MS3.FA) are fitted by maximizing the likelihood (the MLE principle) of the observed data. Another important observation is regarding the names of our proposed algorithms: the algorithms are called “simple-”—S2 = simple-supervised; S3 = simple-semisupervised; MS3 = missing simple-semisupervised—not only because they constitute a simple adaptation of the factor analysis model, but mostly because we created an adaptation of the (simple-)supervised FA model called (simple-)supervised PPCA, and we did not want this model to be confused with the already existing supervised PCA model in the literature. Simple-supervised probabilistic principal component analysis (S2.PPCA) is not discussed in this paper, but it is implemented and usable in the R package that we developed (https://github.com/aciobanusebi/s2fa; accessed on 31 July 2021) along with other undiscussed but related models: Simple-semisupervised unconstrained factor analysis (S3.UncFA), Simple-semisupervised probabilistic principal component analysis (S3.PPCA), Missing simple-semisupervised unconstrained factor analysis (MS3.UncFA), and Missing simple-semisupervised probabilistic principal component analysis (MS3.PPCA).

4.1. The S2.FA Model

The core of this subsection regards the S2.FA model, but in order to link it with LR, we need to also introduce a similar model to S2.FA: S2.UncFA. This link will make S2.FA a good candidate for replacing LR when $D>>n$. These three ideas—S2.FA and S2.UncFA, S2.UncFA-LR link, and replacing LR via S2.FA—will be expanded below.

4.1.1. The S2.FA Model. S2.FA and S2.UncFA

The first model that we propose is called simple-supervised factor analysis. It is a linear Gaussian system with slight changes:

$z\sim \mathcal{N}({\mu }_z,{\sigma }_z^2)$-observed variable, $z\in \mathbb{R}$, ${\mu }_z\in \mathbb{R}$, ${\sigma }_z\in {\mathbb{R}}_{+}^{*}$

$x|z\sim \mathcal{N}(\mu +\Lambda z,\Psi )$, $x\in {\mathbb{R}}^{D\times 1},\mu \in {\mathbb{R}}^{D\times 1},\Lambda \in {\mathbb{R}}^{D\times 1}$, and $\Psi \in {\mathbb{R}}^{D\times D}$ a diagonal matrix.

If we do not impose the constraint of $\Psi$ being diagonal, we arrive at the simple-supervised unconstrained factor analysis (S2.UncFA):

$z\sim \mathcal{N}({\mu }_z,{\sigma }_z^2)$-observed variable, $z\in \mathbb{R}$, ${\mu }_z\in \mathbb{R}$, ${\sigma }_z\in {\mathbb{R}}_{+}^{*}$

$x|z\sim \mathcal{N}(\mu +\Lambda z,\Psi )$, $x\in {\mathbb{R}}^{D\times 1},\mu \in {\mathbb{R}}^{D\times 1},\Lambda \in {\mathbb{R}}^{D\times 1}$, and $\Psi \in {\mathbb{R}}^{D\times D}$ a symmetric and positive definite matrix.

In contrast with the factor analysis model, which is fitted via an EM algorithm, S2.UncFA and S2.FA are fitted via analytic formulas (see Propositions 4 and 5).

Proposition 4.

Let $\left\{(x^{\left(i\right)},z^{\left(i\right)})\right|x^{\left(i\right)}\in {\mathbb{R}}^{D\times 1},z^{\left(i\right)}\in {\mathbb{R}}^{d\times 1},i\in \{1,\dots ,n\}\}$ be a data set where D is the dimensionality of the input data, d is the dimensionality of the output data (although in the context of this paper $d=1$, we decided to expose more general results—$d\ge 1$—in order for the reader to gain more insight; this is the reason why we write ${\Sigma }_z$ and not just ${\sigma }_z^2$, or ${z^{\left(i\right)}}^{\top }$ and not just $z^{\left(i\right)}$, or ${\overline{z}}^{\top }$ and not just $\overline{z}$, etc.), $\left\{x^{\left(i\right)}\right|i\in \{1,\dots ,n\}\}$ is the input, and $\left\{z^{\left(i\right)}\right|i\in \{1,\dots ,n\}\}$ is the output. We suppose that the data was generated as follows:

$z^{\left(i\right)}\sim \mathcal{N}({\mu }_z,{\Sigma }_z)$, $z^{\left(i\right)}\in {\mathbb{R}}^{d\times 1}$, ${\mu }_z\in {\mathbb{R}}^{d\times 1}$, ${\Sigma }_z\in {\mathbb{R}}^{d\times d}$ a symmetric and positive definite matrix and

$x^{\left(i\right)}|z^{\left(i\right)}\sim \mathcal{N}(\mu +\Lambda z,\Psi )$, $x^{\left(i\right)}\in {\mathbb{R}}^{D\times 1},\mu \in {\mathbb{R}}^{D\times 1},\Lambda \in {\mathbb{R}}^{D\times d}$, while $\Psi \in {\mathbb{R}}^{D\times D}$ is a symmetric and positive definite matrix.

Then, the parameters in the S2.UncFA algorithm (training phase) can be estimated via maximum likelihood using the following closed-form formulas:

$$\begin{array}{cc}\hfill {\widehat{\mu }}_z& =\frac{{\sum }_{i=1}^n{z}^{\left(i\right)}}{n}\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill {\widehat{\Sigma }}_z& =\frac{{\sum }_{i=1}^n(z^{\left(i\right)}-{\widehat{\mu }}_z){(z^{\left(i\right)}-{\widehat{\mu }}_z)}^{\top }}{n}\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill \widehat{\Lambda }& =\left(n\overline{x}{\overline{z}}^{\top }-\sum _{i=1}^n{x}^{\left(i\right)}{z^{\left(i\right)}}^{\top }\right){\left(n\overline{z}{\overline{z}}^{\top }-\sum _{i=1}^n{z}^{\left(i\right)}{z^{\left(i\right)}}^{\top }\right)}^{-1}\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill \widehat{\mu }& =\overline{x}-\widehat{\Lambda }\overline{z}\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill \widehat{\Psi }& =\frac{{\sum }_{i=1}^n(x^{\left(i\right)}-\widehat{\mu }-\Lambda z^{\left(i\right)}){(x^{\left(i\right)}-\widehat{\mu }-\Lambda z^{\left(i\right)})}^{\top }}{n}\hfill \end{array}$$

(9)

where $\overline{x}={\displaystyle \frac{{\sum }_{i=1}^n{x}^{\left(i\right)}}{n}}$ and $\overline{z}={\widehat{\mu }}_z$. For the testing/prediction phase, one uses the formula for $z|x$ from (4).

For more elaborate notations and the proof, see [8] pp. 13–17.

Proposition 5.

[We will denote the parameter $\widehat{\Psi }$ in (9) as ${\widehat{\Psi }}_{S2.\mathit{UncFA}}$.]

For the S2.FA algorithm, (9) is replaced by

$$\widehat{\Psi }=\mathit{diag}\left({\widehat{\Psi }}_{S2.\mathit{UncFA}}\right)$$

(10)

where “$diag$” takes the diagonal of a matrix and returns the corresponding diagonal matrix.

The proof of Equation (10) is relatively simple, and we skip it for brevity. It can be found in [8] pp. 21–23.

For the step-by-step S2.FA algorithm and also for the matrix form of the algorithm, see Appendix B.

4.1.2. The S2.FA Model. The Link between LR and S2.UncFA

Linear regression and S2.UncFA have the same prediction function after fitting, as we claim and prove below.

Proposition 6.

Let $\left\{(x^{\left(i\right)},z^{\left(i\right)})\right|x^{\left(i\right)}\in {\mathbb{R}}^{D\times 1},z^{\left(i\right)}\in {\mathbb{R}}^d,i\in \{1,\dots ,n\}\}$ be a data set where D is the dimensionality of the input data, d is the dimensionality of the output data (The same observation as earlier: in the context of this paper, only the “$d=1$" case is relevant.), $\left\{x^{\left(i\right)}\right|i\in \{1,\dots ,n\}\}$ is the input, and $\left\{z^{\left(i\right)}\right|i\in \{1,\dots ,n\}\}$ is the output.

One can fit an S2.UncFA model and obtain—via the relationships (5)–(9)—${\widehat{\mu }}_z$, ${\widehat{\Sigma }}_z$, $\widehat{\Psi }$, $\widehat{\Lambda }$, $\widehat{\mu }$. Remember that at the test phase (see (4)), the predicted value is

$${\mathit{predicted}}_{S2.\mathit{UncFA}}\left(x^{*}\right)={\widehat{\mu }}_z+{\widehat{\Sigma }}_z{\widehat{\Lambda }}^{\top }{(\widehat{\Lambda }{\widehat{\Sigma }}_z{\widehat{\Lambda }}^{\top }+\widehat{\Psi })}^{-1}(x^{*}-\widehat{\mu }-\widehat{\Lambda }{\widehat{\mu }}_z),\forall x^{*}\in {\mathbb{R}}^{D\times 1}.$$

One can fit a linear regression model and obtain $\widehat{w}$, $\widehat{b}$ from (1) and (2). Remember that at the test phase, the predicted value is

$${\mathit{predicted}}_{\mathit{LR}}\left(x^{*}\right)=\widehat{w}{x}^{*}+\widehat{b},\forall x^{*}\in {\mathbb{R}}^{D\times 1}.$$

Then:

$${\mathit{predicted}}_{S2.\mathit{UncFA}}\left(x^{*}\right)={\mathit{predicted}}_{\mathit{LR}}\left(x^{*}\right),\forall x^{*}\in {\mathbb{R}}^{D\times 1}.$$

Proof. 

Let $X=\left[\begin{array}{c}{x}^{\left(1\right)}\dots x^{\left(n\right)}\end{array}\right]\in {\mathbb{R}}^{D\times n}$ and $Z=\left[\begin{array}{c}{z}^{\left(1\right)}\dots z^{\left(n\right)}\end{array}\right]\in {\mathbb{R}}^{d\times n}$.

We begin by computing $\widehat{\Psi }$.

$$\begin{array}{cc}\hfill \widehat{\Psi }& \stackrel{\left(9\right)}{=}\frac{1}{n}\sum _{i=1}^n(x^{\left(i\right)}-\widehat{\mu }-\widehat{\Lambda }{z}^{\left(i\right)}){(x^{\left(i\right)}-\widehat{\mu }-\widehat{\Lambda }{z}^{\left(i\right)})}^{\top }\hfill \\ \hfill & =\frac{1}{n}\sum _{i=1}^n(x^{\left(i\right)}{x^{\left(i\right)}}^{\top }-x^{\left(i\right)}{\widehat{\mu }}^{\top }-x^{\left(i\right)}{z^{\left(i\right)}}^{\top }{\widehat{\Lambda }}^{\top }-\widehat{\mu }{x^{\left(i\right)}}^{\top }+\widehat{\mu }{\widehat{\mu }}^{\top }+\widehat{\mu }{z^{\left(i\right)}}^{\top }{\widehat{\Lambda }}^{\top }-\hfill \\ \hfill & -\widehat{\Lambda }{z}^{\left(i\right)}{x^{\left(i\right)}}^{\top }+\widehat{\Lambda }{z}^{\left(i\right)}{\widehat{\mu }}^{\top }+\widehat{\Lambda }{z}^{\left(i\right)}{z^{\left(i\right)}}^{\top }{\widehat{\Lambda }}^{\top })\hfill \\ \hfill & =\frac{1}{n}X{X}^{\top }-\overline{x}{\widehat{\mu }}^{\top }-\frac{1}{n}X{Z}^{\top }{\widehat{\Lambda }}^{\top }-\widehat{\mu }{\overline{x}}^{\top }+\widehat{\mu }{\widehat{\mu }}^{\top }+\widehat{\mu }{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }-\frac{1}{n}\widehat{\Lambda }Z{X}^{\top }+\hfill \\ \hfill & +\widehat{\Lambda }\overline{z}{\widehat{\mu }}^{\top }+\frac{1}{n}\widehat{\Lambda }Z{Z}^{\top }{\widehat{\Lambda }}^{\top }\hfill \\ \hfill & =\frac{1}{n}X{X}^{\top }-\frac{1}{n}X{Z}^{\top }{\widehat{\Lambda }}^{\top }-\frac{1}{n}\widehat{\Lambda }Z{X}^{\top }+\frac{1}{n}\widehat{\Lambda }Z{Z}^{\top }{\widehat{\Lambda }}^{\top }-\overline{x}{\widehat{\mu }}^{\top }-\widehat{\mu }{\overline{x}}^{\top }+\widehat{\mu }{\widehat{\mu }}^{\top }+\hfill \\ \hfill & +\widehat{\Lambda }\overline{z}{\widehat{\mu }}^{\top }+\widehat{\mu }{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }.\hfill \end{array}$$

We substitute $\widehat{\mu }$ with $\overline{x}-\widehat{\Lambda }\overline{z}$.

$\overline{x}{\widehat{\mu }}^{\top }=\overline{x}{(\overline{x}-\widehat{\Lambda }\overline{z})}^{\top }=\overline{x}{\overline{x}}^{\top }-\overline{x}{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }$

$\widehat{\mu }{\overline{x}}^{\top }=(\overline{x}-\widehat{\Lambda }\overline{z}){\overline{x}}^{\top }=\overline{x}{\overline{x}}^{\top }-\widehat{\Lambda }\overline{z}{\overline{x}}^{\top }$

$\widehat{\mu }{\widehat{\mu }}^{\top }=(\overline{x}-\widehat{\Lambda }\overline{z}){(\overline{x}-\widehat{\Lambda }\overline{z})}^{\top }=\overline{x}{\overline{x}}^{\top }-\overline{x}{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }-\widehat{\Lambda }\overline{z}{\overline{x}}^{\top }+\widehat{\Lambda }\overline{z}{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }$

$\widehat{\Lambda }\overline{z}{\widehat{\mu }}^{\top }=\widehat{\Lambda }\overline{z}{(\overline{x}-\widehat{\Lambda }\overline{z})}^{\top }=\widehat{\Lambda }\overline{z}{\overline{x}}^{\top }-\widehat{\Lambda }\overline{z}{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }$

$\widehat{\mu }{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }=(\overline{x}-\widehat{\Lambda }\overline{z}){\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }=\overline{x}{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }-\widehat{\Lambda }\overline{z}{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }$

We return to compute $\widehat{\Psi }$:

$$\begin{array}{cc}\hfill \widehat{\Psi }& =\frac{1}{n}X{X}^{\top }-\frac{1}{n}X{Z}^{\top }{\widehat{\Lambda }}^{\top }-\frac{1}{n}\widehat{\Lambda }Z{X}^{\top }+\frac{1}{n}\widehat{\Lambda }Z{Z}^{\top }{\widehat{\Lambda }}^{\top }-\overline{x}{\overline{x}}^{\top }+\cancel{\overline{x}{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }}-\cancel{\overline{x}{\overline{x}}^{\top }}+\hfill \\ \hfill & +\cancel{\widehat{\Lambda }\overline{z}{\overline{z}}^{\top }}+\cancel{\overline{x}{\overline{x}}^{\top }}-\cancel{\overline{x}{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }}-\cancel{\widehat{\Lambda }\overline{z}{\overline{z}}^{\top }}+\cancel{\widehat{\Lambda }\overline{z}{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }}+\widehat{\Lambda }\overline{z}{\overline{x}}^{\top }-\cancel{\widehat{\Lambda }\overline{z}{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }}+\hfill \\ \hfill & +\overline{x}{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }-\widehat{\Lambda }\overline{z}{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }\hfill \\ \hfill & =\frac{1}{n}X{X}^{\top }-\frac{1}{n}X{Z}^{\top }{\widehat{\Lambda }}^{\top }-\frac{1}{n}\widehat{\Lambda }Z{X}^{\top }+\frac{1}{n}\widehat{\Lambda }Z{Z}^{\top }{\widehat{\Lambda }}^{\top }-\overline{x}{\overline{x}}^{\top }+\widehat{\Lambda }\overline{z}{\overline{x}}^{\top }+\hfill \\ \hfill & +\overline{x}{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }-\widehat{\Lambda }\overline{z}{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }.\hfill \end{array}$$

(11)

We continue by computing $\widehat{\Lambda }{\widehat{\Sigma }}_z{\widehat{\Lambda }}^{\top }$.

$$\begin{array}{cc}\hfill & \widehat{\Lambda }{\widehat{\Sigma }}_z{\widehat{\Lambda }}^{\top }\stackrel{\left(6\right)}{=}\widehat{\Lambda }\left(\frac{1}{n}Z{Z}^{\top }-\overline{z}{\overline{z}}^{\top }\right){\widehat{\Lambda }}^{\top }=\frac{1}{n}\widehat{\Lambda }Z{Z}^{\top }{\widehat{\Lambda }}^{\top }-\widehat{\Lambda }\overline{z}{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }.\hfill \end{array}$$

(12)

We observe that the above term (see (12)) is also included in $\widehat{\Psi }$ (see (11)).

$$\begin{array}{cc}\hfill {\widehat{\Sigma }}_z{\widehat{\Lambda }}^{\top }& \stackrel{\left(7\right)}{=}\left(\frac{1}{n}Z{Z}^{\top }-\overline{z}{\overline{z}}^{\top }\right){(n\overline{z}{\overline{z}}^{\top }-Z{Z}^{\top })}^{-1}(n\overline{z}{\overline{x}}^{\top }-Z{X}^{\top })\hfill \\ \hfill & =\left(\frac{1}{n}Z{Z}^{\top }-\overline{z}{\overline{z}}^{\top }\right)\frac{-1}{n}{\left(\frac{1}{n}Z{Z}^{\top }-\overline{z}{\overline{z}}^{\top }\right)}^{-1}(n\overline{z}{\overline{x}}^{\top }-Z{X}^{\top })\hfill \\ \hfill & =\frac{1}{n}Z{X}^{\top }-\overline{z}{\overline{x}}^{\top }.\hfill \end{array}$$

(13)

Since ${\left(\widehat{\Lambda }{\widehat{\Sigma }}_z{\widehat{\Lambda }}^{\top }\right)}^{\top }=\widehat{\Lambda }{\widehat{\Sigma }}_z^{\top }{\widehat{\Lambda }}^{\top }=\widehat{\Lambda }{\widehat{\Sigma }}_z{\widehat{\Lambda }}^{\top }\Rightarrow \widehat{\Lambda }{\widehat{\Sigma }}_z{\widehat{\Lambda }}^{\top }$ is symmetric.

We have that:

$$\begin{array}{c}\hfill \widehat{\Lambda }{\widehat{\Sigma }}_z{\widehat{\Lambda }}^{\top }=\widehat{\Lambda }\left({\widehat{\Sigma }}_z{\widehat{\Lambda }}^{\top }\right)\stackrel{\left(13\right)}{=}\widehat{\Lambda }\left(\frac{1}{n}Z{X}^{\top }-\overline{z}{\overline{x}}^{\top }\right)=\frac{1}{n}\widehat{\Lambda }Z{X}^{\top }-\widehat{\Lambda }\overline{z}{\overline{x}}^{\top }.\end{array}$$

(14)

We also have that:

$$\begin{array}{cc}\hfill \widehat{\Lambda }{\widehat{\Sigma }}_z{\widehat{\Lambda }}^{\top }& ={\left(\widehat{\Lambda }{\widehat{\Sigma }}_z{\widehat{\Lambda }}^{\top }\right)}^{\top }\stackrel{\left(13\right)}{=}{\left(\frac{1}{n}\widehat{\Lambda }Z{X}^{\top }-\widehat{\Lambda }\overline{z}{\overline{x}}^{\top }\right)}^{\top }=\frac{1}{n}X{Z}^{\top }{\widehat{\Lambda }}^{\top }-\overline{x}{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }.\hfill \end{array}$$

(15)

We continue by computing $\widehat{\Lambda }{\widehat{\Sigma }}_z{\widehat{\Lambda }}^{\top }+\widehat{\Psi }$.

As we have already noticed above, $\widehat{\Lambda }{\widehat{\Sigma }}_z{\widehat{\Lambda }}^{\top }$ is also included in $\widehat{\Psi }$ (see (11) and (12)). In the computation of $\widehat{\Lambda }{\widehat{\Sigma }}_z{\widehat{\Lambda }}^{\top }+\widehat{\Psi }$, we replace $\widehat{\Lambda }{\widehat{\Sigma }}_z{\widehat{\Lambda }}^{\top }$ once with (14) and then with (15). We get:

$$\begin{array}{cc}\hfill \widehat{\Lambda }{\widehat{\Sigma }}_z{\widehat{\Lambda }}^{\top }+\widehat{\Psi }& \underset{\left(14\right)\left(15\right)}{\overset{\left(11\right)\left(12\right)}{=}}\frac{1}{n}X{X}^{\top }-\cancel{\frac{1}{n}X{Z}^{\top }{\widehat{\Lambda }}^{\top }}-\cancel{\frac{1}{n}\widehat{\Lambda }Z{X}^{\top }}-\overline{x}{\overline{x}}^{\top }+\cancel{\widehat{\Lambda }\overline{z}{\overline{x}}^{\top }}+\hfill \\ & +\cancel{\overline{x}{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }}+\cancel{\frac{1}{n}\widehat{\Lambda }Z{X}^{\top }}-\cancel{\widehat{\Lambda }\overline{z}{\overline{x}}^{\top }}+\cancel{\frac{1}{n}X{Z}^{\top }{\widehat{\Lambda }}^{\top }}-\cancel{\overline{x}{\overline{z}}^{\top }{\widehat{\Lambda }}^{\top }}\hfill \\ & =\frac{1}{n}X{X}^{\top }-\overline{x}{\overline{x}}^{\top }.\hfill \end{array}$$

(16)

Observation: The result is exactly the maximum likelihood estimate of the covariance matrix $\Sigma$ of the input data set if $x\sim \mathcal{N}(\mu ,\Sigma )$: ${\Sigma }_{\mathrm{MLE}}=\frac{1}{n}X{X}^{\top }-\overline{x}{\overline{x}}^{\top }$. This is natural because according to the relationship (4) we have $x\sim \mathcal{N}(\mu ,\Lambda {\Sigma }_z{\Lambda }^{\top }+\Psi )$, and there are enough free parameters in $\Lambda {\Sigma }_z{\Lambda }^{\top }+\Psi$, i.e., $dD+\frac{d^2-d}{2}+d+\frac{D^2-D}{2}+D$ free parameters—$dD$ in $\Lambda$, $\frac{d^2-d}{2}$ in ${\Sigma }_z$, $\frac{D^2-D}{2}$ in $\Psi$, for it to become ${\Sigma }_{\mathrm{MLE}}=\frac{1}{n}X{X}^{\top }-\overline{x}{\overline{x}}^{\top }$, since $\Sigma$ has $\frac{D^2-D}{2}+D$ free parameters.

We return to the initial computation:

${\mathit{predicted}}_{S2.\mathit{UncFA}}\left(x^{*}\right)=$

$\stackrel{\left(4\right)}{=}{\widehat{\mu }}_z+{\widehat{\Sigma }}_z{\widehat{\Lambda }}^{\top }{(\widehat{\Lambda }{\widehat{\Sigma }}_z{\widehat{\Lambda }}^{\top }+\widehat{\Psi })}^{-1}(x^{*}-\widehat{\mu }-\widehat{\Lambda }{\widehat{\mu }}_z)$

$\stackrel{\left(5\right)\left(13\right)\left(16\right)}{=}\overline{z}+\left(\frac{1}{n}Z{X}^{\top }-\overline{z}{\overline{x}}^{\top }\right){\left(\frac{1}{n}X{X}^{\top }-\overline{x}{\overline{x}}^{\top }\right)}^{-1}(x^{*}-\overline{x}+\cancel{\widehat{\Lambda }\overline{z}}-\cancel{\widehat{\Lambda }\overline{z}})$

$=\left(\frac{1}{n}Z{X}^{\top }-\overline{z}{\overline{x}}^{\top }\right){\left(\frac{1}{n}X{X}^{\top }-\overline{x}{\overline{x}}^{\top }\right)}^{-1}{x}^{*}+\overline{z}-\left(\frac{1}{n}Z{X}^{\top }-\overline{z}{\overline{x}}^{\top }\right){\left(\frac{1}{n}X{X}^{\top }-\overline{x}{\overline{x}}^{\top }\right)}^{-1}\overline{x}$

$=\left(n\overline{z}{\overline{x}}^{\top }-Z{X}^{\top }\right){\left(n\overline{x}{\overline{x}}^{\top }-X{X}^{\top }\right)}^{-1}{x}^{*}+\overline{z}-\left(n\overline{z}{\overline{x}}^{\top }-Z{X}^{\top }\right){\left(n\overline{x}{\overline{x}}^{\top }-X{X}^{\top }\right)}^{-1}\overline{x}$

$\stackrel{\left(1\right)}{=}\widehat{w}{x}^{*}+\overline{z}-\widehat{w}\overline{x}$

$\stackrel{\left(2\right)}{=}\widehat{w}{x}^{*}+\widehat{b}$

$={\mathit{predicted}}_{\mathit{LR}}\left(x^{*}\right),\forall x^{*}\in {\mathbb{R}}^{D\times 1}.$ □

4.1.3. The S2.FA Model. A New Approach for LR When $D>>n$

Since FA can be used to estimate the density of a data set when $D>>n$, and S2.UncFA is equivalent to LR, we consider S2.FA as a new approach to extend LR when $D>>n$ besides the two solutions mentioned in Section 2.

4.2. The S3.FA Model

Factor analysis is a classic generative unsupervised model. Its supervised counterpart is S2.FA as shown in the previous subsection. Those two can be merged into a semisupervised model that we propose here, named simple-semisupervised factor analysis:

$z\sim \mathcal{N}({\mu }_z,{\sigma }_z^2)$-either observed or latent variable, $z\in \mathbb{R}$, ${\mu }_z\in \mathbb{R}$, ${\sigma }_z\in {\mathbb{R}}_{+}^{*}$

$x|z\sim \mathcal{N}(\mu +\Lambda z,\Psi )$, $x\in {\mathbb{R}}^{D\times 1},\mu \in {\mathbb{R}}^{D\times 1},\Lambda \in {\mathbb{R}}^{D\times 1}$, and $\Psi \in {\mathbb{R}}^{D\times D}$ a diagonal matrix.

If we were to speak about Gaussian naive Bayes and the GMM, good hints for combining these two supervised–unsupervised counterparts into a semisupervised model can be found in [12]. We applied those hints for our supervised–unsupervised counterparts—S2.FA and FA—and created an EM algorithm to fit an S3.FA model. For the step-by-step algorithm and the matrix form of the algorithm, see Appendix C. For more elaborate notations and the proof, see [8] pp. 26–34.

4.3. The MS3.FA Model

The algorithm that fits an S3.FA model can be adapted also for the case when not all the components of x are known. We call the resulted model missing simple-semisupervised factor analysis:

$z\sim \mathcal{N}({\mu }_z,{\sigma }_z^2)$-either observed or latent variable, $z\in \mathbb{R}$, ${\mu }_z\in \mathbb{R}$, ${\sigma }_z\in {\mathbb{R}}_{+}^{*}$

$x|z\sim \mathcal{N}(\mu +\Lambda z,\Psi )$, $x\in {\mathbb{R}}^{D\times 1},\mu \in {\mathbb{R}}^{D\times 1},\Lambda \in {\mathbb{R}}^{D\times 1}$, and $\Psi \in {\mathbb{R}}^{D\times D}$ a diagonal matrix; each component of x: $x_1,\dots ,x_D$ is either observed or latent.

The resulting algorithm that fits a MS3.FA model is an EM algorithm. For the step-by-step algorithm, see Appendix D. For more elaborate notations and the proof, see [8] pp. 34–37.

5. Experiments

In this section, we include the experiments we carried out on data with $D>>n$ using the S2.FA, S3.FA, and MS3.FA models, comparing them with other methods. In all the experiments, we computed errors between the real values and the predicted values; the metric we used is mean squared error (MSE):

$$\mathrm{MSE}=\frac{1}{N}\sum _{i=1}^N{({\mathrm{real}}_i-{\mathrm{predicted}}_i)}^2,$$

where N is the number of the unknown elements whose real and predicted values are ${\mathrm{real}}_i$ and ${\mathrm{predicted}}_i$, respectively; an unknown element represents an output number for S2.FA and S3.FA or an input/output number for MS3.FA. We ran each experiment five times and computed a 95% confidence interval using the t-distribution. Furthermore, in each experiment we used the same three data sets:

• Gas sensor array under flow modulation data set (http://archive.ics.uci.edu/ml/datasets/Gas+sensor+array+under+flow+modulation; accessed on 31 July 2021) [13]: 58 observations, 432 input attributes;
• atp1d—the airline ticket price; 1D refers to the fact that the target price is in the next day—(https://www.openml.org/d/41475; accessed on 31 July 2021) [14]: 337 observations, 411 input attributes; 370 after preprocessing: see below;
• m5spec—corn measured on a NIR spectrometer: mp5 instrument—(http://www.eigenvector.com/data/Corn; accessed on 31 July 2021): 80 observations, 700 input attributes.

All three of these data sets have multiple outputs, but for each data set, we selected only the first output column that appears in the text data file and used it as the output: ace_conc for gas sensor array under flow modulation data set, LBL_ALLminpA_fut_001 for atp1d, the first column in the propvals file for m5spec.

We preprocessed each data set simply by dropping the constant columns. Only the second data set has constant columns: from 432 columns, we obtain 370.

5.1. The S2.FA Model: Experiment

The experiment concerning S2.FA covers the comparison of the three solutions presented so far for LR when $D>>n$:

• Moore–Penrose inverse
• ridge regression—L2 regularization
• S2.FA.

Each data set was split into a training part—$80\%$—and a testing part—$20\%$. If the model had hyperparameters, ridge regression, the training part was also split into a new training part—$60\%$ of the whole data set—and a validation part—$20\%$ of the whole data set—in order to be able to set the hyperparameters (for ridge regression, we used a simple technique: pick $\alpha \in \{{10}^2,{10}^{1.9},{10}^{1.8},\dots ,{10}^{-1.9},{10}^{-2}\}$, which attains the minimum validation error); after setting the hyperparameters, we train a new model on the initial training part—$80\%$ of the whole data set—and obtain the final model. All of the MSE errors are reported on the testing part and shown in

Table 1. Simple-supervised factor analysis (S2.FA) experiment: mean squared error (MSE) 95% confidence intervals on three data sets using three methods for regression when $D>>n$; the best MSE means are marked in bold.

Data Set/Method	Moore–Penrose	Ridge Regression	S2.FA
Gas sensor array under flow modulation	0.0251 ± 0.0254	0.0062 ± 0.007	0.0452 ± 0.0208
atp1d	94,627.7239 ± 80,183.0076	27,770.9253 ± 42,887.5216	4724.2957 ± 1616.3341
m5spec	0.00004 ± 0.00001	0.02676 ± 0.01372	0.37344 ± 0.25025

.

As one may notice, the best method is different for each data set, so our general advice is to use all the methods on a given data set and pick the best one.

5.2. The S3.FA Model: Experiment

The experiment concerning S3.FA includes an analysis of algorithms for semisupervised regression when $D>>n$:

• Moore–Penrose inverse—a supervised method
• S2.FA—a supervised method
• S3.FA—a semisupervised method
• label propagation [15]—a semisupervised method: we used the function
  sslLabelProp in the SSL R package [16] with the parameter alpha set to 1.

Each data set was split into a training part—$80\%$—and a testing part—$20\%$. We retained from the training part $5\%$, $10\%$, $15\%$, $20\%$, $25\%$, …, $100\%$ of the output labels. For the supervised methods, we used only the labeled data in the training set, and for the semisupervised methods, we used the full training set when fitting the model. We initialized the S3.FA method with the fitted parameters returned by the S2.FA algorithm. All of the MSE errors are reported on the testing part and shown in Figure 1, Figure 2 and Figure 3—inspired from [17]—and

Table 2. S3.FA experiment: MSE 95% confidence intervals on three data sets using four methods for semisupervised regression when $D>>n$; $p\in \{5\%,10\%,15\%,30\%,50\%,70\%\}$ of the training output labels are retained; the best MSE means are marked in bold.

Data Set	Method	$\mathit{p}=5\%$	$\mathit{p}=10\%$	$\mathit{p}=15\%$
Gas sensor array under flow modulation	Moore–Penrose	0.034 ± 0.0437	0.0292 ± 0.0421	0.0267 ± 0.0326
	S2.FA	0.2511 ± 0.3953	0.0899 ± 0.2367	0.0285 ± 0.0333
	S3.FA	3.9799 ± 6.7087	0.349 ± 0.6626	0.2294 ± 0.1561
	Label Propagation	0.0853 ± 0.0073	0.0825 ± 0.0051	0.0708 ± 0.013
atp1d	Moore–Penrose	3763.2939 ± 749.0966	3079.5042 ± 1842.7676	3428.7567 ± 1722.6437
	S2.FA	2706.9906 ± 477.3245	2339.3504 ± 324.8581	2279.0802 ± 99.5443
	S3.FA	3771.605 ± 1563.6543	2972.1137 ± 639.6724	2633.0816 ± 274.7409
	Label Propagation	110,820.3235 ± 0	110,820.3235 ± 0	110,820.3235 ± 0
m5spec	Moore–Penrose	0.4602 ± 0.354	0.3609 ± 0.7631	0.0187 ± 0.0221
	S2.FA	0.7003 ± 0.3834	0.4269 ± 0.5441	0.4999 ± 0.5172
	S3.FA	0.8133 ± 0.6366	1.653 ± 3.9497	0.5118 ± 0.5503
	Label Propagation	0.2175 ± 0.0707	0.1939 ± 0.0706	0.2343 ± 0.0624
Data Set	Method	$\mathit{p}=\mathbf{30}\%$	$\mathit{p}=\mathbf{50}\%$	$\mathit{p}=\mathbf{70}\%$
Gas sensor array under flow modulation	Moore–Penrose	0.0313 ± 0.0259	0.0192 ± 0.0145	0.0132 ± 0.0075
	S2.FA	0.0588 ± 0.0576	0.0233 ± 0.0224	0.0279 ± 0.0116
	S3.FA	0.645 ± 1.3044	0.1666 ± 0.1059	0.0912 ± 0.0443
	Label Propagation	0.0668 ± 0.0086	0.0675 ± 0.018	0.0605 ± 0.0122
atp1d	Moore–Penrose	6401.7587 ± 3602.9182	7907.484 ± 4449.2312	36,041.874 ± 21,300.704
	S2.FA	2444.9001 ± 404.2393	2439.162 ± 108.3605	2399.7948 ± 160.1038
	S3.FA	2760.6602 ± 391.9217	2659.6472 ± 112.9795	2521.9861 ± 222.3765
	Label Propagation	110,820.3235 ± 0	110,820.3235 ± 0	110,820.3235 ± 0
m5spec	Moore–Penrose	0.00057 ± 0.00085	0.00009 ± 0.00008	0.00009 ± 0.00004
	S2.FA	0.37449 ± 0.12508	0.35796 ± 0.07275	0.3995 ± 0.03164
	S3.FA	0.37865 ± 0.12808	0.36181 ± 0.07463	0.40419 ± 0.03415
	Label Propagation	0.19391 ± 0.02754	0.17424 ± 0.00857	0.17752 ± 0.01545

. The figures contain all the results from using $5\%$ to $100\%$ of the output labels, but we include less information in the table for brevity.

We notice that on the selected data sets, S3.FA returns poorer results even than S2.FA, which uses only the labeled data. As in the previous experiment, the best method is also data-dependent. The models that show a greater amount of variability compared to the others are S3.FA—in two data sets—and Moore–Penrose—in one data set. Moreover, as expected, the errors tend to decrease as the percentage of labeled training data points increases; the plots do not help us in this regard, but this decrease can be seen numerically in Table 2.

5.3. The MS3.FA Model: Experiment

The experiment concerning MS3.FA includes a comparison of two different types of algorithms for data imputation when $D>>n$:

• Mean imputation: for a given attribute (input column), compute its mean ignoring the missing values, then replace the missing data on that attribute with this computed mean
• MS3.FA.

We also tried two other R packages: mice [18] and Amelia [19], but they could not be applied successfully on our data sets perhaps because they have a peculiarity: $D>>n$.

For each data set, we removed $10\%$, $20\%$, $30\%$, $40\%$, $50\%$, and $60\%$ of the input (We could have added missing data also in the output, but we wanted to focus on the missing input data scenario and not on the semisupervised case.) cells and imputed those via the above mentioned algorithms. The results are presented in Figure 4, Figure 5 and Figure 6 and in

Table 3. MS3.FA experiment: MSE 95% confidence intervals on three data sets using two methods for imputing missing data when $D>>n$; $p\in \{10\%,20\%,30\%,40\%,50\%,60\%\}$ of the input data are removed; the best MSE means are marked in bold.

Data Set	Method	$\mathit{p}=10\%$	$\mathit{p}=20\%$	$\mathit{p}=30\%$
Gas sensor array under flow modulation	Mean	0.0108 ± 0.0022	0.0241 ± 0.0027	0.0359 ± 0.0011
	MS3.FA	0.00603 ± 0.00029	0.01176 ± 0.00081	0.01747 ± 0.00094
atp1d	Mean	3239.9757 ± 68.2511	6831.0884 ± 105.6119	10,077.1798 ± 117.4693
	MS3.FA	1807.7748 ± 105.03	3697.1146 ± 124.2197	5276.899 ± 104.6252
m5spec	Mean	0.00013 ± 0.00001	0.00026 ± 0	0.00039 ± 0.00001
	MS3.FA	0.14835 ± 0.000002	0.148433 ± 0.000002	0.148518 ± 0.000002
Data Set	Method	$\mathit{p}=\mathbf{40}\%$	$\mathit{p}=\mathbf{50}\%$	$\mathit{p}=\mathbf{60}\%$
Gas sensor array under flow modulation	Mean	0.0494 ± 0.0042	0.0597 ± 0.0015	0.0731 ± 0.0012
	MS3.FA	0.02372 ± 0.00159	0.0299 ± 0.00047	0.0364 ± 0.0011
atp1d	Mean	13,423.3625 ± 252.7735	16,899.3166 ± 223.6971	20,156.2493 ± 62.1299
	MS3.FA	7071.6793 ± 143.3232	8921.5319 ± 165.5759	10,558.5723 ± 53.1182
m5spec	Mean	0.000521 ± 0.000005	0.000658 ± 0.000008	0.00080 ± 0.000005
	MS3.FA	0.1486 ± 0.00001	0.14869 ± 0.00001	0.148781 ± 0.000001

.

From these results, we discover that MS3.FA is better than mean imputation on two data sets, and, as expected, the error increases as the percentage of missing data increases.

6. Conclusions and Future Work

The initial purpose of this paper was to extend an already existing model: factor analysis. We developed its supervised counterpart (S2.FA) and noticed that the unconstrained version (S2.UncFA) is equivalent to linear regression. Because FA is applied in density estimation when the dimensionality of the data is greater than the number of samples, and because of the already mentioned equivalence, the purpose of the paper became to analyze this new method of applying LR when $D>>n$, i.e., via S2.FA. Since FA and S2.FA are generative models and are unsupervised–supervised counterparts, we combined both into a new model S3.FA as an extension of LR to semisupervised learning when $D>>n$. The final extension regards missing data; it is called MS3.FA. We developed an R package (s2fa) with these algorithms; it can be found on GitHub. The experimental parts included several comparisons in the $D>>n$ scenario:

• of S2.FA with other techniques extending LR to the $D>>n$ case,
• of S3.FA with other (semi)supervised regression methods,
• of MS3.FA with another data imputation algorithm.

The bottom line is that we do not necessarily recommend S3.FA for semisupervised regression since our results suggest that it gives poor results, but we encourage the consideration of S2.FA for regression and MS3.FA for missing data imputation as algorithms to be compared with others on a given data set.

As for future work, we could further explore the S2.FA, S3.FA, and MS3.FA algorithms when z is a real vector, not just a real number. Moreover, we can experiment with the PPCA version of the algorithms. Questions regarding the time complexity—empirical or not—can be also addressed; we expect the fitting time to be impractical if the number of columns is large. Because there are models such as mixture of factor analyzers [20] and mixture of linear regression models ([21] Section 14.5.1), another research direction involves mixtures of S2.FAs. Another idea would be to investigate the memory resources required by the algorithms when the data set increases and also to consider scalable systems such as Spark [22] for implementation.