Analytic Function Approximation by Path-Norm-Regularized Deep Neural Networks

Abstract

We show that neural networks with an absolute value activation function and with network path norm, network sizes and network weights having logarithmic dependence on $1/\epsilon$ can $\epsilon$-approximate functions that are analytic on certain regions of ${\mathbb{C}}^d$.

1. Introduction

Deep neural networks have found broad applications in many areas and disciplines, such as computer vision, speech and audio recognition and natural language processing. Two of the main characteristics of a given class of neural networks are its complexity and approximating capability. Once the activation function is selected, a class of networks is determined by the specification of the network architecture (namely, its depth and width) and the choice of network weights. Hence, the estimation of the complexity of a given class is carried out by regularizing (one of) those parameters, and the approximation properties of obtained regularized classes of networks are then investigated.

The capability of shallow networks of depth 1 to approximate continuous functions is shown in the universal approximation theorem ([1]), and approximations of integrable functions by networks with fixed width are presented in [2]. Network-architecture-constrained approximations of analytic functions are given in [3], where it is shown that ReLU networks with depth depending logarithmically on $1/\epsilon$ and width $d+4$ can $\epsilon$-approximate analytic functions on the closed subcubes of ${(-1,1)}^d$.

The weight regularization of networks is usually carried out by imposing an $l_p$-related constraint on network weights, $p\ge 0$. The most popular types of such constraints include the $l_0$, $l_1$ and the path norm regularizations (see, respectively, [4,5,6] and references therein). Approximations of $\beta$-smooth functions on ${[0,1]}^d$ by $l_0$-regularized sparse ReLU networks are given in [5,7], and exponential rates of approximations of analytic functions by $l_0$-regularized networks are derived in [8].

Path-norm-regularized classes of deep ReLU networks are considered in [4], where, together with other characteristics, the Rademacher complexities of those classes are estimated. The network size independence of those estimates makes the path norm regularization particularly remarkable. As the estimation only uses the Lipschitz continuity (with Lipschitz constant 1), the idempotency and the non-negative homogeneity of the ReLU function, it can be extended to networks with the absolute value activation function. Network characteristics similar to the path norm are also considered in the works [9,10], where they are called, respectively, a variation and a basis-path norm, and statistical features of classes of networks are described in terms of those characteristics.

The objective of the present paper is the construction of path-norm-regularized networks that exponentially fast approximate analytic functions. Our goal is to achieve such convergence rates with activations that are idempotent, non-negative homogeneous and Lipschitz continuous with Lipschitz constant 1 so that the constructed path-norm-regularized networks fall within the scope of network classes studied in [4]. It turns out that networks with an absolute value activation function may suit this goal better than the networks with an ReLU activation function. More precisely, we show that analytic functions can be $\epsilon$-approximated by networks with an absolute value activation function $a\left(x\right)$ and with the path norm, the depth, the width and the weights all depending logarithmically on $1/\epsilon$. Such an approximation holds (i) on any subset ${(0,1-\delta ]}^d\subset {(0,1)}^d$ for analytic functions on ${(0,1)}^d$ with absolutely convergent power series; (ii) on the whole hypercube ${[0,1]}^d$ for functions that can be analytically continued to certain subsets of ${\mathbb{C}}^d$. Note that, as the network weights, as well as the total number of weights, depend logarithmically on $1/\epsilon$, then the $l_1$ weight norms of the constructed approximating deep networks are also of logarithmic dependence on $1/\epsilon$.

Note that the absolute value activation function considered in this paper is among the common built-in activation functions of the software-based neural network evolving method NEAT-Python ([11]). Training algorithms for networks with an absolute value activation function are developed in the works [12,13]. In addition, the VC-dimensions and the structures of the loss surfaces of neural networks with piecewise linear activation functions, including the absolute value function, are described in the works [14,15].

Notation: For a matrix $W\in {\mathbb{R}}^{d_1\times d_2}$, we denote by $\left|W\right|\in {\mathbb{R}}^{d_1\times d_2}$ the matrix obtained by taking the absolute values of the entries of W: ${\left|W\right|}_{ij}=\left|W_{ij}\right|$. For brevity of presentation, we will say that the matrix $\left|W\right|$ is the absolute value of the matrix W (note that, in the literature, there are also other definitions of the notion of an absolute value of a matrix). The path norm of a neural network f is denoted by ${\parallel f\parallel }_{\times }$. For $\mathrm{x}=(x_1,\dots ,x_d)\in {\mathbb{R}}^d$ and $\mathbf{k}=(k_1,\dots ,k_d)\in {\mathbb{N}}_0^d$, the degree of the monomial ${\mathrm{x}}^{\mathbf{k}}=x_1^{k_1}·\cdots ·x_d^{k_d}$ is defined to be ${\parallel \mathbf{k}\parallel }_1={\sum }_{i=1}^d{k}_i$. To assure that the matrix–vector multiplications are able to be accomplished, the vectors from ${\mathbb{R}}^d$, according to the context, may be treated as matrices either from ${\mathbb{R}}^{d\times 1}$ or from ${\mathbb{R}}^{1\times d}$.

2. The Class of Approximant Networks

Neural networks are constituted of weight matrices, biases and nonlinear activation functions acting neuron-wise in the hidden layers. The biases, also called shift vectors, can be omitted by adding a fixed coordinate 1 to the input vector and correspondingly modifying the weight matrices. As the definition of the path norm of networks does not assume the presence of shift vectors, we will add a coordinate 1 to the input vector $\mathrm{x}$ and will consider classes of neural networks of the form

$${\mathcal{F}}_{\alpha }(L,\mathbf{p})=\{f:{[0,1]}^p\to {\mathbb{R}}^{p_{L+1}}|f\left(\mathrm{x}\right)=W_L\circ \alpha \circ W_{L-1}\circ \alpha \circ \cdots \circ \alpha \circ W_0(1,\mathrm{x})\},$$

where $W_i\in {\mathbb{R}}^{p_{i+1}\times p_i}$ are the weight matrices, $i=0,\dots ,L,$ and $\mathbf{p}=(p_0,p_1,\dots ,p_{L+1})$ is the width vector, with $p_0=p+1$. The number of hidden layers L determines the depth of networks from ${\mathcal{F}}_{\alpha }(L,\mathbf{p})$ and, in each layer, the activation function $\alpha :\mathbb{R}\to \mathbb{R}$ acts element-wise on the input vector. For $f\in {\mathcal{F}}_{\alpha }(L,\mathbf{p})$ given by

$$f\left(\mathrm{x}\right)=W_L\circ \alpha \circ W_{L-1}\circ \alpha \circ \cdots \circ \alpha \circ W_0(1,\mathrm{x}),$$

(1)

let

$${\parallel f\parallel }_{\times }:=\parallel \prod _{i=0}^L\left|W_i\right|{\parallel }_1$$

(2)

be the path norm of f, where ${\parallel ·\parallel }_1$ denotes the $l_1$ norm of the $p_0(=p+1)$ dimensional vector ${\prod }_{i=0}^L\left|W_i\right|$ obtained as a product of absolute values of the weight matrices of f. For $B>0$, let

$${\mathcal{F}}_{\alpha }(L,\mathbf{p},B)=\{f\in {\mathcal{F}}_{\alpha }{(L,\mathbf{p}),\parallel f\parallel }_{\times }\le B\}$$

be a path-norm-regularized subclass of ${\mathcal{F}}_{\alpha }(L,\mathbf{p})$. As the results obtained in [4] indicate, the path norm regularizations are particularly well-suited for networks whose activation function $\alpha$ is

• Lipschitz continuous with Lipschitz constant 1;
• Idempotent, that is, $\alpha \left(\alpha \right(x\left)\right)=\alpha \left(x\right)$, $x\in \mathbb{R}$;
• Non-negative homogeneous, that is, $\alpha \left(cx\right)=c\alpha \left(x\right),$ for $c\ge 0$, $x\in \mathbb{R}$.

We therefore aim to choose an activation $\alpha$ possessing those properties such that analytic functions can be approximated by networks from ${\mathcal{F}}_{\alpha }(L,\mathbf{p},B)$ with a small path norm constraint B. The most popular activation functions satisfying the above conditions are the ReLU function $\sigma \left(x\right)=max\{0,x\}$ and the absolute value function $a\left(x\right)=\left|x\right|$. Below, we show that, with the absolute value activation function, the path norms of approximant networks may be significantly smaller than the path norms of the ReLU networks.

The standard technique of neural network function approximation relies on approximating the product function $(x,y)\mapsto xy$, which then allows us to approximate monomials and polynomials of any desired degree. In [7], the approximation of the product $xy=({(x+y)}^2-x^2-y^2)/2$ is achieved by approximating the function $x\mapsto x^2$. The latter is based on the observation that, for the triangle wave

$$g_s\left(x\right)=\underset{s\mathrm{times}}{\underbrace{g\circ g\circ \cdots \circ g}}\phantom{1}\left(x\right),$$

(3)

where $g:[0,1]\to [0,1]$ is defined by

$$g\left(x\right)=\left\{\begin{array}{cc}2x,\hfill & 0\le x<1/2,\hfill \\ 2(1-x),\hfill & 1/2\le x\le 1,\hfill \end{array}\right.$$

and for any positive integer m,

$$|x^2-f_m\left(x\right)|\le 2^{-2m-2},$$

where

$$f_m\left(x\right):=x-\sum _{s=1}^m\frac{g_s\left(x\right)}{2^{2s}}.$$

(4)

The approximation of $x^2$ by networks with the ReLU activation function $\sigma \left(x\right)$ then follows from the representation

$$g\left(x\right)=2\sigma \left(x\right)-4\sigma (x-1/2).$$

(5)

Thus, in this case, we will obtain matrices containing weights 2 and 4, which will make the path norm of approximant networks big. Note that the same approach is also used in [3] for constructing ReLU network approximations of analytic functions. In [5], the approximation of the product

$$\begin{array}{c}\hfill xy=h\left(\frac{x-y+1}{2}\right)-h\left(\frac{x+y}{2}\right)+\frac{x+y}{2}-\frac{1}{4}\end{array}$$

is achieved by approximating the function $h\left(x\right):=x(1-x)$, which, in turn, is based on the observation that, for the triangle wave

$$\begin{array}{c}\hfill R^k=T^k\circ T^{k-1}\circ \cdots \circ T^1,\end{array}$$

where $T^k:[0,2^{2-2k}]\to [0,2^{-2k}]$ is defined by

$$T^k\left(x\right):=\sigma (x/2)-\sigma (x-2^{1-2k}),$$

(6)

and for any positive integer m,

$$\begin{array}{c}\hfill |h\left(x\right)-\sum _{k=1}^m{R}^k\left(x\right)|\le 2^{-m},x\in [0,1].\end{array}$$

Although in the representation (6), the coefficients (weights) are all in $[-1,1]$, the approximant ${\sum }_{k=1}^m{R}^k\left(x\right)$ in this case does not have the factors $2^{-2s}$ presented in the approximant $f_m\left(x\right)$ in (4), which, again, will result in big values of path norms. Therefore, in order to take advantage of the presence of those reducing weights, we would like to represent the function $g\left(x\right)$ in (5) by a linear combination of activation functions with smaller coefficients. This is possible if, instead of $\sigma \left(x\right)$, we deploy the absolute value activation function $a\left(x\right)$. Indeed, in this case, we have that $g\left(x\right)$ can be represented on $[0,1]$ as

$$g\left(x\right)=1-2a(x-1/2).$$

(7)

In the next section, we use the above representation (7) to show that analytic functions can be $\epsilon$-approximated by networks from ${\mathcal{F}}_a(L,\mathbf{p},B)$ with each of ${L,\parallel \mathbf{p}\parallel }_{\infty }$ and B, as well as the network weights having logarithmic dependence on $1/\epsilon$. As all networks will have the same activation function $a\left(x\right)=\left|x\right|$, in the following, the subscript a will be omitted.

3. Results

We first construct a neural network with activation function $a\left(x\right)$, that, for the given $\gamma ,m\in \mathbb{N}$, simultaneously approximates all d-dimensional monomials of a degree less than $\gamma$ up to an error of ${\gamma }^2{4}^{-m}$. The depth of this network has order $m{log}_2\gamma$ and its width is of order $m{\gamma }^{d+1}$. Moreover, the entries of the product of the absolute values of matrices of the network have an order of at most ${\gamma }^5$ (note the independence of m).

For $\gamma >0$, let $C_{d,\gamma }$ denote the number of d-dimensional monomials ${\mathrm{x}}^{\mathbf{k}}$ with degree ${\parallel \mathbf{k}\parallel }_1<\gamma$. Then, $C_{d,\gamma }<{(\gamma +1)}^d$ and the following holds:

Lemma 1.

There is a neural network Mon${}_{m,\gamma }^d\in \mathcal{F}(L,\mathbf{p})$ with $L\le \lceil {log}_2\gamma \rceil (2m+5)+2$, $p_0=d+1$, $p_{L+1}=C_{d,\gamma }$ and ${\parallel \mathbf{p}\parallel }_{\infty }\le 6\gamma (m+2)C_{d,\gamma }$ such that

$$\parallel {\mathrm{Mon}}_{m,\gamma }^d\left(\mathrm{x}\right)-{\left({\mathrm{x}}^{\mathbf{k}}\right)}_{{\parallel \mathbf{k}\parallel }_1<\gamma }{\parallel }_{\infty }\le {\gamma }^2{4}^{-m},\mathrm{x}\in {[0,1]}^d.$$

Moreover, the entries of the $C_{d,\gamma }\times (d+1)$-dimensional matrix obtained by multiplying the absolute values of matrices presented in ${\mathrm{Mon}}_{m,\gamma }^d$ are all bounded by $144{(\gamma +1)}^5$.

Taking in the above lemma $\gamma ,m=\lceil {log}_2\frac{1}{\epsilon }\rceil$, we obtain a neural network from $\mathcal{F}(L,\mathbf{p})$, with L and ${\parallel \mathbf{p}\parallel }_{\infty }$ having logarithmic dependence on $1/\epsilon$, which simultaneously approximates the monomials of a degree at most of $\gamma$ with error $\epsilon$ (up to a logarithmic factor). Moreover, the entries of the product of absolute values of matrices of this network will also have logarithmic dependence on $1/\epsilon$. Below, we use this property to construct a neural network approximation of analytic and analytically continuable functions with an approximation error $\epsilon$ and with network parameters having logarithmic order.

Theorem 1.

Let $f\left(\mathrm{x}\right)={\sum }_{\mathbf{k}\in {\mathbb{N}}_0^d}{a}_{\mathbf{k}}{\mathrm{x}}^{\mathbf{k}}$ be an analytic function on ${(0,1)}^d$ with ${\sum }_{\mathbf{k}\in {\mathbb{N}}_0^d}\left|a_{\mathbf{k}}\right|\le F$. Then, for any $\epsilon ,\delta \in (0,1)$, there is a constant $C=C(d,F)$ and a neural network $F_{\epsilon }\in \mathcal{F}(L,\mathbf{p},B)$ with $L\le C\left({log}_2\frac{1}{\delta }\right)\left({log}_2^2\frac{1}{\epsilon }\right),$${\parallel \mathbf{p}\parallel }_{\infty }\le \frac{C}{{\delta }^{d+1}}{\left({log}_2\frac{1}{\epsilon }\right)}^{d+2}$ and

$$B\le {10}^{4}dF{\left(\frac{{log}_2((2F+16)/\epsilon )}{\delta }\right)}^5,$$

such that

$$|F_{\epsilon }\left(\mathrm{x}\right)-f\left(\mathrm{x}\right)|\le \frac{\epsilon }{{\delta }^2},\mathit{forall}\mathrm{x}\in {(0,1-\delta ]}^d.$$

Note that an exponential convergence rate of deep ReLU network approximants on subintervals ${(0,1-\delta ]}^d$ is also given in [3]. In our case, however, not only the depth and the width but also the path norm $\parallel F_{\epsilon }{\parallel }_{\times }$ of the constructed network $F_{\epsilon }$ have logarithmic dependence on $1/\epsilon$. Note that, in the above theorem, as $\delta$ approaches to 0, both ${\parallel \mathbf{p}\parallel }_{\infty }$ and B, as well as the approximation error, grow polynomially on $1/\delta$. In the next theorem, we use the properties of Chebyshev series to derive an exponential convergence rate on the whole hypercube ${[0,1]}^d$. Recall that the Chebyshev polynomials are defined as $T_0\left(x\right)=1$, $T_1\left(x\right)=x$ and

$$T_{n+1}\left(x\right)=2x{T}_n\left(x\right)-T_{n-1}\left(x\right).$$

Chebyshev polynomials play an important role in the approximation theory ([16]), and, in particular, it is known ([17], Theorem 3.1) that if f is Lipschitz continuous on $[-1,1]$, then it has a unique representation as an absolutely and uniformly convergent Chebyshev series

$$f\left(x\right)=\sum _{k=0}^{\infty }{a}_k{T}_k\left(x\right).$$

Moreover, in case f can be analytically continued to an ellipse $E_{\rho }\subset \mathbb{C}$ with foci $-1$ and 1 and with the sum of semimajor and semiminor axes equal to $\rho >1$, then the partial sums of the above Chebyshev series converge to f with a geometric rate and the coefficients $a_k$ also decay with a geometric rate. This result was first derived by Bernstein in [18] and its extension to the multivariate case was given in [19]. Note that the condition $z\in E_{\rho }$ implies that $z^2\in N_{1,h^2}$, where $h=(\rho -{\rho }^{-1})/2$ and, for $d,a>0$, $N_{d,a}\subset \mathbb{C}$ denotes an open ellipse with foci 0 and d and the leftmost point $-a$. For $F>0$, $\rho >1$ and $h=(\rho -{\rho }^{-1})/2$, let ${\mathcal{A}}^d(\rho ,F)$ be the space of functions $f:{[0,1]}^d\to \mathbb{R}$ that can be analytically continued to the region $\{\mathbf{z}\in {\mathbb{C}}^d:z_1^2+\cdots +z_d^2\in N_{d,h^2}\}$ and are bounded there by F. Using the extension of Bernstein’s theorem to the multivariate case, we obtain

Lemma 2.

Let $\rho \ge 2^{\sqrt{d}}$. For $f\in {\mathcal{A}}^d(\rho ,F)$, there is a constant $C=C(d,\rho ,F)$ and a polynomial

$$p\left(\mathrm{x}\right)=\sum _{{\parallel \mathbf{k}\parallel }_1\le \gamma }{b}_{\mathbf{k}}{\mathrm{x}}^{\mathbf{k}},\mathrm{x}\in {[0,1]}^d,$$

with

$$|b_{\mathbf{k}}{|\le C(\gamma +1)}^d$$

(8)

and

$$|f\left(\mathrm{x}\right)-p\left(\mathrm{x}\right)|\le C{\rho }^{-\gamma /\sqrt{d}},\mathit{forall}\mathrm{x}\in {[0,1]}^d.$$

Combining Lemma 1 and Lemma 2, we obtain the following.

Theorem 2.

Let $\epsilon \in (0,1)$ and let $\rho \ge 2^{\sqrt{d}}$. For $f\in {\mathcal{A}}^d(\rho ,F)$, there is a constant $C=C(d,\rho ,F)$ and a neural network $F_{\epsilon }\in \mathcal{F}(L,\mathbf{p},B)$ with $L\le C{log}_2^2\frac{1}{\epsilon },$${\parallel \mathbf{p}\parallel }_{\infty }\le C{\left({log}_2\frac{1}{\epsilon }\right)}^{d+2}$ and $B\le C{\left({log}_2\frac{1}{\epsilon }\right)}^{2d+5}$ such that

$$|F_{\epsilon }{\left(\mathrm{x}\right)-f\left(\mathrm{x}\right)|\le \epsilon ,\mathit{forall}\mathrm{x}\in [0,1]}^d.$$

We conclude this part by estimating the $l_1$ weight regularization of networks constructed in Theorem 2. First, the total number of weights in those networks is bounded by ${(L+1)\parallel \mathbf{p}\parallel }_{\infty }^2=O{\left({log}_2\frac{1}{\epsilon }\right)}^{2d+6}.$ From (7), it follows that all of the weights of network ${\mathrm{Mon}}_{m,\gamma }^d$ from Lemma 1 are in $[-2,2]$. In Theorem 2, the network $F_{\epsilon }$ is obtained by adding to a network ${\mathrm{Mon}}_{m,\gamma }^d,$ with $\gamma =m=O\left({log}_2\frac{1}{\epsilon }\right)$, a layer with coefficients of partial sums of power series of an approximated function. Thus, using (8), we obtain that the $l_1$ weight norm of the network $F_{\epsilon }$ constructed in Theorem 2 has order $O{\left({log}_2\frac{1}{\epsilon }\right)}^{4d+6}$.

4. Proofs

In the following proofs, $I_k$ denotes an identity matrix of size $k\times k$ and all of the networks have activation $a\left(x\right)=\left|x\right|$. The proof of Lemma 1 is based on the following two lemmas.

Lemma 3.

For any positive integer m, there exists a neural network Mult${}_m\in \mathcal{F}(2m+3,\mathbf{p})$, with $p_0=3$, $p_{L+1}=1$ and ${\parallel \mathbf{p}\parallel }_{\infty }=3m+2,$ such that

$$|{\mathrm{Mult}}_m(x,y)-xy|\le 3·2^{-2m-3},\mathrm{for}\mathrm{all}x,y\in [0,1],$$

(9)

and the product of absolute values of the matrices presented in Mult${}_m$ is equal to

$$\left(3\sum _{k=1}^m\frac{2^k-1}{2^{2k}},2-2^{-m},2-2^{-m}\right).$$

Proof. 

For $k\ge 2$, let $R_k$ denote a row of length k with a first entry equal to $-1/2$, last entry equal to 1 and all other entries equal to 0. Let $A_k$ be a matrix of size $(k+1)\times k$ obtained by adding the $(k+1)$-th row $R_k$ to the identity matrix $I_k$. That is,

In addition, let $B_k$ denote a matrix of size $k\times k$ given by

It then follows from (7) that

$$B_{m+2}\circ a\circ A_{m+1}\circ \cdots \circ B_3\circ a\circ A_2\left(\genfrac{}{}{0pt}{}{1}{x}\right)=\left(\begin{array}{c}1\\ x\\ g_1\left(x\right)\\ g_2\left(x\right)\\ ·\\ ·\\ ·\\ g_m\left(x\right)\end{array}\right),$$

where $g_s\left(x\right)$ is the function defined in (3), $s=1,\dots ,m$. Thus, if $S_{m+2}$ is a row of length $m+2$ defined as

$$S_{m+2}=\left(0,1,-\frac{1}{2^{2·1}},-\frac{1}{2^{2·2}},\cdots ,-\frac{1}{2^{2·m}}\right),$$

then

$$S_{m+2}\circ a\circ B_{m+2}\circ a\circ A_{m+1}\circ \cdots \circ a\circ B_3\circ a\circ A_2\left(\genfrac{}{}{0pt}{}{1}{x}\right)=f_m\left(x\right),$$

where $f_m$ is defined by (4). We have that

$$|S_{m+2}|·|B_{m+2}|·|A_{m+1}|·\cdots ·|B_3|·|A_2|=\left(\sum _{k=1}^m\frac{2^{k+1}-2}{2^{2k}},2-2^{-m}\right).$$

As $xy=\frac{1}{2}\left({(x+y)}^2-x^2-y^2\right),$ then, in the first layer of ${\mathrm{Mult}}_m$, we will obtain a vector

$$\left(\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 1& 0& 0\\ 0& 0& 1\\ 1& 0& 0\\ 0& 1& 1\end{array}\right)\left(\begin{array}{c}1\\ x\\ y\end{array}\right):=C\left(\begin{array}{c}1\\ x\\ y\end{array}\right)=\left(\begin{array}{c}1\\ x\\ 1\\ y\\ 1\\ x+y\end{array}\right)$$

and will then apply the network in a parallel manner from the first part of the proof to each of the pairs (1, x),(1, y) and (1, x + y). More precisely, for a given matrix M of size p × q, let $\tilde{M}$ be a matrix of size 3p × 3q defined as

$$\tilde{M}=\left(\begin{array}{ccc}M& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& M& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& M\end{array}\right).$$

Then, for the network

$${\mathrm{Mult}}_m(x,y)=\left(-\frac{1}{2},-\frac{1}{2},\frac{1}{2}\right)\circ a\circ {\tilde{S}}_{m+2}\circ a\circ {\tilde{B}}_{m+2}\circ a\circ {\tilde{A}}_{m+1}\circ \cdots \circ {\tilde{B}}_3\circ a\circ {\tilde{A}}_2\circ a\circ C\left(\begin{array}{c}1\\ x\\ y\end{array}\right)$$

we have that

$${\mathrm{Mult}}_m(x,y)=\frac{1}{2}(f_m(x+y)-f_m\left(x\right)-f_m\left(y\right)),$$

which, together with $|f_m\left(x\right)-x^2|<2^{-2m-2}$ and the triangle inequality, implies (9). It remains to be noted that the product of absolute values of the matrices presented in ${\mathrm{Mult}}_m$ is equal to

$$\left(\frac{1}{2},\frac{1}{2},\frac{1}{2}\right)·|{\tilde{S}}_{m+2}|·|{\tilde{B}}_{m+2}|·|{\tilde{A}}_{m+1}|·\cdots ·|{\tilde{B}}_3|·|{\tilde{A}}_2|·|C|=\left(3\sum _{k=1}^m\frac{2^k-1}{2^{2k}},2-2^{-m},2-2^{-m}\right),$$

which completes the proof of the lemma. □

Lemma 4.

For any positive integer m, there exists a neural network Mult${}_m^r\in \mathcal{F}(L,\mathbf{p})$, with $L=(2m+5)\lceil {log}_{2}r\rceil +1$, $p_0=r+1$, $p_{L+1}=1$ and ${\parallel \mathbf{p}\parallel }_{\infty }\le 6r(m+2)+1,$ such that

$$|{\mathrm{Mult}}_m^r\left(\mathrm{x}\right)-\prod _{i=1}^r{x}_i|\le r^2{4}^{-m}for all\mathrm{x}=(x_1,\cdots ,x_r)\in {[0,1]}^r,$$

and, for the $(r+1)$-dimensional vector $J_m^r$ obtained by multiplication of absolute values of matrices presented in ${\mathrm{Mult}}_m^r$, we have that $\parallel J_m^r{\parallel }_{\infty }\le 144r^4$.

Proof. 

First, for a given $k\in \mathbb{N}$, we construct a network $N_m^k\in \mathcal{F}(L,\mathbf{p})$ with $L=2m+4,$$p_0=2k+1$ and $p_{L+1}=k+1,$ such that

$$N_m^k(x_1,x_2,\dots ,x_{2k-1},x_{2k})=(1,{\mathrm{Mult}}_m(x_1,x_2),\dots ,{\mathrm{Mult}}_m(x_{2k-1},x_{2k})).$$

In the first layer, we obtain a vector for which the first coordinate is 1 followed by triples $(1,x_{2l-1},x_{2l})$$l=1,\dots ,k,$ that is, the vector $(1,1,x_1,x_2,1,x_3,x_4,\dots ,1,x_{2k-1},x_{2k})$. $N_m^k$ is then obtained by applying in parallel the network ${\mathrm{Mult}}_m$ to each triple $(1,x_{2l-1},x_{2l})$ while keeping the first coordinate equal to 1. The product of absolute values of the matrices presented in this construction is a matrix of size $(k+1)\times (2k+1)$ having a form

$$\left(\begin{array}{cccccccccc}1& 0& 0& 0& 0& 0& \dots & 0& 0& 0\\ a_m& b_m& b_m& 0& 0& 0& \dots & 0& 0& 0\\ a_m& 0& 0& b_m& b_m& 0& \dots & 0& 0& 0\\ ·& ·& ·& ·& ·& ·& ·& ·& ·& ·\\ a_m& 0& 0& 0& 0& 0& \dots & 0& b_m& b_m\end{array}\right),$$

where $a_m=3{\sum }_{k=1}^m\frac{2^k-1}{2^{2k}}$ and $b_m=2-2^{-m}$ are the coordinates obtained in the previous lemma. Let us now construct the network ${\mathrm{Mult}}_m^r$. The first hidden layer of ${\mathrm{Mult}}_m^r$ computes

$$(1,x_1,\dots ,x_r)\mapsto (1,x_1,\dots ,x_r,\underset{2^q-r}{\underbrace{1,1,\dots ,1}}\phantom{1}),$$

where $q=\lceil {log}_{2}r\rceil$. We then subsequently apply the networks $N_m^{2^q},N_m^{2^{q-1}},\dots ,N_m^2$ and, in the last layer, we multiply the outcome by $(0,1)$. From Lemma 3 and triangle inequality, we have that $|{\mathrm{Mult}}_m(x,y)-tz|\le 3·2^{-2m-3}+|x-t|+|y-z|,$ for $x,y,t,z\in [0,1]$. Hence, by induction on q, we obtain that $|{\mathrm{Mult}}_m^r\left(\mathrm{x}\right)-{\prod }_{i=1}^r{x}_i|\le 3^q{2}^{-2m-3}\le 3r^2{2}^{-2m-3}\le r^2{4}^{-m}$.

Note that the product of absolute values of matrices in each network $N_m^k$ has the above form, that is, in each row, it has at most three nonzero values, each of which is less than 2. As the matrices given in the first and the last layer of ${\mathrm{Mult}}_m^r$ also satisfy this property, then each entry of the product of absolute values of all matrices of ${\mathrm{Mult}}_m^r$ will not exceed ${12}^{q+2}\le 144r^4$. □

Proof of Lemma 1.

We have that, if ${\parallel \mathbf{k}\parallel }_1=0$, then ${\mathrm{x}}^{\mathbf{k}}=1$, and if ${\parallel \mathbf{k}\parallel }_1=1$, then $\mathbf{k}$ has only one non-zero coordinate, say, $k_j$, which is equal to 1 and ${\mathrm{x}}^{\mathbf{k}}=x_j$. Denote $N=C_{d,\gamma }-d-1$ and let ${\mathbf{k}}^1,\dots ,{\mathbf{k}}^N$ be the multi-indices satisfying $1<\parallel {\mathbf{k}}^i{\parallel }_1<\gamma ,$$i=1,\dots ,N$. For $\mathbf{k}=(k_1,\dots ,k_d)$ with ${\parallel \mathbf{k}\parallel }_1>1$, denote by ${\mathrm{x}}_{\mathbf{k}}$ the $(\parallel \mathbf{k}{\parallel }_1+1)$-dimensional vector of the form

$${\mathrm{x}}_{\mathbf{k}}=(1,\underset{k_1}{\underbrace{x_1,\dots ,x_1}}\phantom{1},\dots ,\underset{k_d}{\underbrace{x_d,\dots ,x_d}}\phantom{1}).$$

The first layer of ${\mathrm{Mon}}_{m,\gamma }^d$ computes the $\left(d+1+{\sum }_{i=1}^N(\parallel {\mathbf{k}}^i{\parallel }_1+1)\right)$-dimensional vector

$${(1,\mathrm{x},{\mathrm{x}}_{{\mathbf{k}}^1},\dots ,{\mathrm{x}}_{{\mathbf{k}}^N})}^{⊺}$$

by multiplying the input vector by matrix $\mathsf{\Gamma }$ of size $\left(d+1+{\sum }_{i=1}^N(\parallel {\mathbf{k}}^i{\parallel }_1+1)\right)\times (r+1)$. In the following layers, we do not change the first $d+1$ coordinates (by multiplying them by $I_{d+1}$), and, to each ${\mathrm{x}}_{{\mathbf{k}}^i}$, we apply in parallel the network ${\mathrm{Mult}}_m^{\parallel {\mathbf{k}}^i{\parallel }_1}$. Recall that, in Lemma 4, $J_m^r$ denotes the $(r+1)$-dimensional vector obtained from the product of absolute values of the matrices of ${\mathrm{Mult}}_m^r$. We then have that the product of the absolute values of the matrices of ${\mathrm{Mon}}_{m,\gamma }^d$ has the form

As the matrix $\mathsf{\Gamma }$ only contains entries 0 and 1, then, applying Lemma 4, we obtain that the entries of M are bounded by

$$\underset{1\le i\le N}{max}\left|\right|J_m^{\parallel {\mathbf{k}}^i{\parallel }_1}|{|}_1\le 144{(\gamma +1)}^5.$$

□

Proof of Theorem 1.

Let $\gamma =⌈\frac{{log}_2((2F+16)/\epsilon )}{{log}_2{(1-\delta )}^{-1}}⌉.$ Then, for $\mathrm{x}\in {(0,1-\delta ]}^d$, we have that

$$|f\left(\mathrm{x}\right)-\sum _{{\parallel \mathbf{k}\parallel }_1<\gamma }{a}_{\mathbf{k}}{\mathrm{x}}^{\mathbf{k}}|=\left|\sum _{{\parallel \mathbf{k}\parallel }_1\ge \gamma }{a}_{\mathbf{k}}{\mathrm{x}}^{\mathbf{k}}\right|\le {(1-\delta )}^{\gamma }F\le \frac{\epsilon F}{2F+16}\le \frac{\epsilon }{2}\le \frac{\epsilon }{2{\delta }^2}.$$

(10)

Applying Lemma 1 with $m=\lceil {log}_2\frac{4F+16}{\epsilon }\rceil$, we obtain that, for all $\mathrm{x}\in {[0,1]}^d$

$$\begin{array}{cc}& \parallel {\mathrm{Mon}}_{m,\gamma }^d\left(\mathrm{x}\right)-{\left({\mathrm{x}}^{\mathbf{k}}\right)}_{{\parallel \mathbf{k}\parallel }_1<\gamma }{\parallel }_{\infty }\le {\gamma }^2{4}^{-m}\le \left(\frac{4}{{log}_2^2{(1-\delta )}^{-1}}\right)\left({log}_2^2\frac{2F+16}{\epsilon }\right){\left(\frac{\epsilon }{4F+16}\right)}^2\hfill \\ \hfill & \le \frac{4(2F+16){\epsilon }^2}{{\delta }^2\epsilon {(4F+16)}^2}\le \frac{\epsilon }{2F{\delta }^2},\hfill \end{array}$$

(11)

where we used the inequalities ${log}_2{(1-\delta )}^{-1}\ge \delta ,\delta \in (0,1),$ and ${log}_2^{2}r\le r$ for $r\ge 16$. In order to approximate the partial sum ${\sum }_{{\parallel \mathbf{k}\parallel }_1\le \gamma }{a}_{\mathbf{k}}{\mathrm{x}}^{\mathbf{k}},$ we add one last layer with the coefficients of that partial sum to the network ${\mathrm{Mon}}_{m,\gamma +1}^d$. As the sum of absolute values of those coefficients is bounded by F, then, combining (10) and (11), for the obtained network $F_{\epsilon }$ we obtain

$$|F_{\epsilon }\left(\mathrm{x}\right)-f\left(\mathrm{x}\right)|\le \frac{\epsilon }{{\delta }^2},for all\mathrm{x}\in {(0,1-\delta ]}^d.$$

From Lemma 1 it follows that

$$\parallel F_{\epsilon }{\parallel }_{\times }\le 144(d+1)F{(\gamma +1)}^5\le {10}^{4}dF{\left(\frac{{log}_2((2F+16)/\epsilon )}{\delta }\right)}^5.$$

□

Let us now present the result from [19] that will be used to derive Lemma 2. First, if $f\in {\mathcal{A}}^d(\rho ,F)$, then ([20], Theorem 4.1) f has a unique representation as an absolutely and uniformly convergent multivariate Chebyshev series

$$f\left(\mathrm{x}\right)=\sum _{k_1=0}^{\infty }\dots \sum _{k_d=0}^{\infty }{a}_{k_1,\dots ,k_d}{T}_{k_1}\left(x_1\right)\dots T_{k_d}\left(x_d\right),\mathrm{x}\in {[0,1]}^d.$$

Note that, for $\mathbf{k}:=(k_1,\dots ,k_d)$, the degree of a d-dimensional polynomial $T_{k_1}\left(x_1\right)\dots T_{k_d}\left(x_d\right)$ is ${\parallel \mathbf{k}\parallel }_1=k_1+\cdots +k_d$. Then, for any non-negative integers $n_1,\dots ,n_d,$ the partial sum

$$p\left(\mathrm{x}\right)=\sum _{k_1=0}^{n_1}\dots \sum _{k_d=0}^{n_d}{a}_{\mathbf{k}}{T}_{k_1}\left(x_1\right)\dots T_{k_d}\left(x_d\right)$$

(12)

is a polynomial truncation of the multivariate Chebyshev series of f of degree $d\left(p\right)=n_1+\cdots +n_d$. It is shown in [19] that

Theorem 3.

For $f\in {\mathcal{A}}^d(\rho ,F)$, there is a constant $C=C(d,\rho ,F)$ such that the multivariate Chebyshev coefficients of f satisfy

$$|a_{\mathbf{k}}|\le C{\rho }^{-{\parallel \mathbf{k}\parallel }_2}$$

(13)

and, for the polynomial truncations p of the multivariate Chebyshev series of f, we have that

$$\underset{d\left(p\right)\le \gamma }{inf}{\parallel f\left(\mathrm{x}\right)-p\left(\mathrm{x}\right)\parallel }_{{[0,1]}^d}\le C{\rho }^{-\gamma /\sqrt{d}}.$$

Proof of Lemma 2.

Note that, from the recursive definition of the Chebyshev polynomials, it follows that, for any $k\ge 0$, the coefficients of the Chebyshev polynomial $T_k\left(x\right)$ are all bounded by $2^k$. Let p now be a polynomial given by (12) with degree $d\left(p\right)\le \gamma$. As the number of summands in the right-hand side of (12) is bounded by ${(\gamma +1)}^d$, then, using (13), we obtain that p can be rewritten as

$$p\left(\mathrm{x}\right)=\sum _{{\parallel \mathbf{k}\parallel }_1\le \gamma }{b}_{\mathbf{k}}{\mathrm{x}}^{\mathbf{k}},$$

with

$$|b_{\mathbf{k}}{|\le C(\gamma +1)}^d{2}^{{\parallel \mathbf{k}\parallel }_1}{\rho }^{-{\parallel \mathbf{k}\parallel }_2}\le C{(\gamma +1)}^d{2}^{\sqrt{d}{\parallel \mathbf{k}\parallel }_2}{\rho }^{-{\parallel \mathbf{k}\parallel }_2}\le C{(\gamma +1)}^d,$$

where the last inequality follows from the condition $\rho \ge 2^{\sqrt{d}}$. □

Proof of Theorem 2.

The proof follows from Lemmas 1 and 2 by taking $\gamma =m=\lceil {log}_2\frac{1}{\epsilon }\rceil$ and adding, to the network ${\mathrm{Mon}}_{m,\gamma +1}^d$, the last layer with the coefficients of the polynomial $p\left(\mathrm{x}\right)$ from Lemma 2. For the obtained network $F_{\epsilon }$ we have that

$$\parallel F_{\epsilon }{\parallel }_{\times }\le 144C(d+1)C_{d,\gamma +1}{(\gamma +2)}^d{(\gamma +2)}^5\le 144C(d+1){(\gamma +2)}^{2d+5},$$

where C is the constant from Lemma 2. □

5. Discussion

Although various activation functions, including the ReLU, sigmoid and the Gaussian function, have already been used in the literature for neural network approximations of smooth and analytic functions (see [3,8,21]), approximating properties of neural networks with an absolute value activation function, which is a built-in activation function of software-based neural network evolving methods (such as NEAT-Python, [11]), has been barely covered previously. Whereas the algorithms developed in the works [12,13] allow us to train neural networks with an absolute value activation function, in the present paper, we study the capabilities of those networks to approximate analytic functions. While popular types of constraints imposed on approximating neural networks are either controlling the $l_p$ norms of network weights or adjusting their architectures, in the present work, we study approximating properties of neural networks with regularized path norms and show that networks with an absolute value activation function and with network path norms having logarithmic dependence on $1/\epsilon$ can $\epsilon$-approximate functions that are analytic on certain regions of ${\mathbb{C}}^d$. The sizes and the weights of constructed networks also have logarithmic dependence on $1/\epsilon$.