On the Asymptotic Optimality of a Low-Complexity Coding Strategy for WSS, MA, and AR Vector Sources

Abstract

In this paper, we study the asymptotic optimality of a low-complexity coding strategy for Gaussian vector sources. Specifically, we study the convergence speed of the rate of such a coding strategy when it is used to encode the most relevant vector sources, namely wide sense stationary (WSS), moving average (MA), and autoregressive (AR) vector sources. We also study how the coding strategy considered performs when it is used to encode perturbed versions of those relevant sources. More precisely, we give a sufficient condition for such perturbed versions so that the convergence speed of the rate remains unaltered.

1. Introduction

In [1], Kolmogorov gave a formula for the rate distortion function (RDF) of Gaussian vectors and for the RDF of Gaussian wide sense stationary (WSS) sources. In [2], Pearl presented an upper bound for the RDF of finite-length data blocks of any Gaussian WSS source and proved that such a bound tends to the RDF of the source when the length of the data block grows. However, he did not propose a coding strategy to achieve his bound for a given block length. In [3], we presented a tighter upper bound for the RDF of finite-length data blocks of any Gaussian WSS source, and we proposed a low-complexity coding strategy to achieve our bound. Obviously, since such a bound is tighter than the one given by Pearl, it also tends to the RDF of the source when the length of the data block grows. In [4], we generalized our low-complexity coding strategy to encode (compress) finite-length data blocks of any Gaussian vector source. Moreover, in [4], we also gave a sufficient condition for the vector source in order to make such a coding strategy asymptotically optimal. We recall that a coding strategy is asymptotically optimal if its rate tends to the RDF of the source as the length of the data block grows. Such a sufficient condition requires the Gaussian vector source to be asymptotically WSS (AWSS). The definition of the AWSS process was first introduced in [5], Section 6, and extended to vector processes in [6], Definition 7.1. However, the convergence speed of the rate of the coding strategy considered (i.e., how fast the rate of the coding strategy tends to the RDF of the AWSS vector source) was not studied in [4].

In this paper, we present a less restrictive sufficient condition for the vector source to make the coding strategy considered asymptotically optimal. Moreover, we study the convergence speed of the rate of such a coding strategy when it is used to encode the most relevant vector sources, namely, WSS, moving average (MA), and autoregressive (AR) vector sources. In this paper, we also study how the coding strategy considered performs when it is used to encode perturbed versions of those relevant sources. Specifically, we give a sufficient condition for such perturbed versions so that the convergence speed of the rate remains unaltered.

The study of the convergence speed in any information-theoretic problem is not an easy task. To study the aforementioned convergence speed, we first need to derive new mathematical results on block Toeplitz matrices and new mathematical results on the correlation matrices of the WSS, MA, and AR vector processes. These new mathematical results are useful not only to study the convergence speed in the information-theoretic problem considered, but also in other problems. In fact, as an example, in Appendix H, we use such mathematical results to study the convergence speed in a statistical signal processing problem on filtering WSS vector processes.

The paper is organized as follows. In Section 2, we give several new mathematical results on block Toeplitz matrices. In Section 3, using the results obtained in Section 2, we give several new mathematical results on the correlation matrices of WSS, MA, and AR vector processes. In Section 4, we recall the low-complexity coding strategy presented in [4], and using the results obtained in Section 3, we study the asymptotic optimality of such a coding strategy when it is used to encode WSS, MA, and AR vector sources. In Section 4, we also study how the coding strategy considered performs when it is used to encode perturbed versions of those sources. Finally, in Section 5, some conclusions are presented.

2. Several New Results on Block Toeplitz Matrices

In this section, we present new results on the product of block Toeplitz matrices, on the inverse of a block Toeplitz matrix, and on block circulant matrices. These results will be used in Section 3. We begin by introducing some notation.

2.1. Notation

In this paper, $\mathbb{N}$, $\mathbb{Z}$, $\mathbb{R}$, and $\mathbb{C}$ denote the set of natural numbers (that is, the set of positive integers), the set of integer numbers, the set of real numbers, and the set of complex numbers, respectively. ${\mathbb{C}}^{M\times N}$ is the set of all $M\times N$ complex matrices. $I_N$ stands for the $N\times N$ identity matrix. $0_{M\times N}$ denotes the $M\times N$ zero matrix. $V_n$ is the $n\times n$ Fourier unitary matrix, i.e.,

$${\left[V_n\right]}_{j,k}=\frac{1}{\sqrt{n}}{\mathrm{e}}^{-\frac{2\pi (j-1)(k-1)}{n}\mathrm{i}},j,k\in \{1,\dots ,n\},$$

with $\mathrm{i}$ being the imaginary unit. We denote by ${\lambda }_1\left(A\right),\dots ,{\lambda }_n\left(A\right)$ the eigenvalues of an $n\times n$ Hermitian matrix A arranged in decreasing order. ∗ denotes the conjugate transpose. ⊗ is the Kronecker product. ${\parallel ·\parallel }_2$ and ${\parallel ·\parallel }_F$ are the spectral norm and the Frobenius norm, respectively.

If $n\in \mathbb{N}$ and $A_j\in {\mathbb{C}}^{M\times N}$ for all $j\in \{1,\dots ,n\}$, then $\mathrm{diag}(A_1,\dots ,A_n)$ is the $n\times n$ block diagonal matrix whose $M\times N$ blocks are given by:

$${\left[\mathrm{diag}(A_1,\dots ,A_n)\right]}_{j,k}={\delta }_{j,k}{A}_j,j,k\in \{1,\dots ,n\},$$

where $\delta$ is the Kronecker delta. We also denote by ${\mathrm{diag}}_{1\le k\le n}\left(A_k\right)$ the matrix $\mathrm{diag}(A_1,\dots ,A_n)$.

If $n\in \mathbb{N}$ and $F:\mathbb{R}\to {\mathbb{C}}^{M\times N}$ is a continuous $2\pi$-periodic function, $T_n\left(F\right)$ stands for the $n\times n$ block Toeplitz matrix generated by F whose $M\times N$ blocks are given by:

$${\left[T_n\left(F\right)\right]}_{j,k}={\mathsf{F}}_{j-k},j,k\in \{1,\dots ,n\},$$

where ${\left\{{\mathsf{F}}_k\right\}}_{k\in \mathbb{Z}}$ is the sequence of Fourier coefficients of F, that is,

$${\mathsf{F}}_k=\frac{1}{2\pi }{\int }_0^{2\pi }{\mathrm{e}}^{-k\omega \mathrm{i}}F\left(\omega \right)d\omega \forall k\in \mathbb{Z}.$$

We denote by $C_n\left(F\right)$ the $n\times n$ block circulant matrix with $M\times N$ blocks defined as:

$$C_n\left(F\right)=(V_n\otimes I_M){\mathrm{diag}}_{1\le k\le n}\left(F\left(\frac{2\pi (k-1)}{n}\right)\right){(V_n\otimes I_N)}^{*}.$$

If $A_n\in {\mathbb{C}}^{nM\times nN}$, then $C_{A_n}$ is the $n\times n$ block circulant matrix with the $M\times N$ blocks given by:

$$C_{A_n}=(V_n\otimes I_M){\mathrm{diag}}_{1\le k\le n}\left({\left[{(V_n\otimes I_M)}^{*}{A}_n(V_n\otimes I_N)\right]}_{k,k}\right){(V_n\otimes I_N)}^{*}.$$

We denote by ${\widehat{C}}_n\left(F\right)$ the $n\times n$ block circulant matrix with the $M\times N$ blocks defined as ${\widehat{C}}_n\left(F\right)=C_{T_n\left(F\right)}$.

If $F\left(\omega \right)$ is Hermitian for all $\omega \in \mathbb{R}$ (or equivalently, $T_n\left(F\right)$ is Hermitian for all $n\in \mathbb{N}$ (see, e.g., [6], Theorem 4.4), then $infF$ denotes ${inf}_{\omega \in [0,2\pi ]}{\lambda }_N\left(F\left(\omega \right)\right)$. We recall that (see [7], Proposition 3):

$$\underset{n\in \mathbb{N}}{inf}{\lambda }_{nN}\left(T_n\left(F\right)\right)=infF=\underset{\omega \in [0,2\pi ]}{min}{\lambda }_N\left(F\left(\omega \right)\right).$$

(1)

2.2. Product of Block Toeplitz Matrices

We begin this subsection with a result on the entries of the block Toeplitz matrices generated by the product of two functions, which is a direct consequence of the Parseval theorem.

Lemma 1.

Consider two continuous $2\pi$-periodic functions $F:\mathbb{R}\to {\mathbb{C}}^{M\times N}$ and $G:\mathbb{R}\to {\mathbb{C}}^{N\times K}$. Let ${\left\{{\mathsf{F}}_k\right\}}_{k\in \mathbb{Z}}$ and ${\left\{{\mathsf{G}}_k\right\}}_{k\in \mathbb{Z}}$ be the sequences of Fourier coefficients of F and G, respectively. Then:

$${\left[T_n\left(FG\right)\right]}_{j,k}=\sum _{h=-\infty }^{\infty }{\mathsf{F}}_{j-h}{\mathsf{G}}_{h-k}$$

for all $n\in \mathbb{N}$ and $j,k\in \{1,\dots ,n\}$.

Proof. 

See Appendix A. □

We can now give a result on the product of two block Toeplitz matrices when one of them is generated by a trigonometric polynomial. We recall that an $M\times N$ trigonometric polynomial of degree $p\in \mathbb{N}\cup \left\{0\right\}$ is a function $F:\mathbb{R}\to {\mathbb{C}}^{M\times N}$ of the form:

$$F\left(\omega \right)=\sum _{k=-p}^p{\mathrm{e}}^{k\omega \mathrm{i}}{\mathsf{A}}_k\forall \omega \in \mathbb{R},$$

(2)

where ${\mathsf{A}}_k\in {\mathbb{C}}^{M\times N}$ with $\left|k\right|\le p$. It can be easily proven (see, e.g., [6], Example 4.3) that the sequence of the Fourier coefficients ${\left\{{\mathsf{F}}_k\right\}}_{k\in \mathbb{Z}}$ of the continuous $2\pi$-periodic function F in Equation (2) is given by:

$${\mathsf{F}}_k=\left\{\begin{array}{cc}{\mathsf{A}}_k\hfill & \mathrm{if} \left|k\right|\le p,\hfill \\ 0_{M\times N}\hfill & \mathrm{if} \left|k\right|>p.\hfill \end{array}\right.$$

Lemma 2. 

Let F, G, ${\left\{{\mathsf{F}}_k\right\}}_{k\in \mathbb{Z}}$, and ${\left\{{\mathsf{G}}_k\right\}}_{k\in \mathbb{Z}}$ be as in Lemma 1.

1. 

If F is a trigonometric polynomial of degree p, then:

$${[T_n\left(F\right)T_n\left(G\right)-T_n\left(FG\right)]}_{j,k}=\left\{\begin{array}{cc}-\sum _{h=j-p}^0{\mathsf{F}}_{j-h}{\mathsf{G}}_{h-k}\hfill & if j\le p,\hfill \\ 0_{M\times K}\hfill & if p+1\le j\le n-p,\hfill \\ -\sum _{h=n+1}^{j+p}{\mathsf{F}}_{j-h}{\mathsf{G}}_{h-k}\hfill & if j\ge n-p+1,\hfill \end{array}\right.$$

(3)

and:

$$\parallel T_n\left(F\right)T_n\left(G\right)-T_n{\left(FG\right)\parallel }_F\le \sqrt{p(p+1)\left(\frac{1}{2\pi }{\int }_0^{2\pi }{\parallel F\left(\omega \right)\parallel }_F^{2}d\omega \right)\left(\frac{1}{2\pi }{\int }_0^{2\pi }{\parallel G\left(\omega \right)\parallel }_F^{2}d\omega \right)}$$

(4)

for all $n\in \mathbb{N}$ and $j,k\in \{1,\dots ,n\}$.

2. 

If G is a trigonometric polynomial of degree q, then:

$${[T_n\left(F\right)T_n\left(G\right)-T_n\left(FG\right)]}_{j,k}=\left\{\begin{array}{cc}-\sum _{h=k-q}^0{\mathsf{F}}_{j-h}{\mathsf{G}}_{h-k}\hfill & if k\le q,\hfill \\ 0_{M\times K}\hfill & if q+1\le k\le n-q,\hfill \\ -\sum _{h=n+1}^{k+q}{\mathsf{F}}_{j-h}{\mathsf{G}}_{h-k}\hfill & if k\ge n-q+1,\hfill \end{array}\right.$$

(5)

and:

$$\parallel T_n\left(F\right)T_n\left(G\right)-T_n{\left(FG\right)\parallel }_F\le \sqrt{q(q+1)\left(\frac{1}{2\pi }{\int }_0^{2\pi }{\parallel F\left(\omega \right)\parallel }_F^{2}d\omega \right)\left(\frac{1}{2\pi }{\int }_0^{2\pi }{\parallel G\left(\omega \right)\parallel }_F^{2}d\omega \right)}$$

(6)

for all $n\in \mathbb{N}$ and $j,k\in \{1,\dots ,n\}$.

3. 

If F is a trigonometric polynomial of degree p and G is a trigonometric polynomial of degree q, then:

$$T_n\left(F\right)T_n\left(G\right)-T_n\left(FG\right)=\left(\begin{array}{ccc}{\xi }_1(F,G)& 0_{pM\times (n-2q)K}& 0_{pM\times qK}\\ 0_{(n-2p)M\times qK}& 0_{(n-2p)M\times (n-2q)K}& 0_{(n-2p)M\times qK}\\ 0_{pM\times qK}& 0_{pM\times (n-2q)K}& {\xi }_2(F,G)\end{array}\right)$$

and:

$$\parallel T_n\left(F\right)T_n\left(G\right)-T_n{\left(FG\right)\parallel }_F=\sqrt{\parallel {\xi }_1{(F,G)\parallel }_F^2+{\parallel {\xi }_2(F,G)\parallel }_F^2}$$

(7)

for all $n\ge max\{2p,2q\}$, where ${\xi }_1(F,G),{\xi }_2(F,G)\in {\mathbb{C}}^{pM\times qK}$ are given by:

$${\left[{\xi }_1(F,G)\right]}_{j,k}=-\sum _{h=max\{j-p,k-q\}}^0{\mathsf{F}}_{j-h}{\mathsf{G}}_{h-k}$$

and:

$${\left[{\xi }_2(F,G)\right]}_{j,k}=-\sum _{h=1}^{min\{j,k\}}{\mathsf{F}}_{j-p-h}{\mathsf{G}}_{h+q-k}$$

for all $j\in \{1,\dots ,p\}$ and $k\in \{1,\dots ,q\}$.

Proof. 

See Appendix B. □

2.3. Inverse of a Block Toeplitz Matrix

Lemma 3. 

Let $F:\mathbb{R}\to {\mathbb{C}}^{N\times N}$ be a trigonometric polynomial of degree p.

1. 

If $F\left(\omega \right)$ is invertible for all $\omega \in \mathbb{R}$ and $\left\{T_n\left(F\right)\right\}$ is stable (i.e., $T_n\left(F\right)$ is invertible for all $n\in \mathbb{N}$ and $\{\parallel {\left(T_n\left(F\right)\right)}^{-1}{\parallel }_2\}$ is bounded), then:

$$\parallel {\left(T_n\left(F\right)\right)}^{-1}-T_n\left(F^{-1}\right){\parallel }_F\le \underset{m\in \mathbb{N}}{sup}{\parallel {\left(T_m\left(F\right)\right)}^{-1}\parallel }_2\sqrt{p(p+1)\left(\frac{1}{2\pi }{\int }_0^{2\pi }{\parallel F\left(\omega \right)\parallel }_F^{2}d\omega \right)\left(\frac{1}{2\pi }{\int }_0^{2\pi }{\parallel {\left(F\left(\omega \right)\right)}^{-1}\parallel }_F^{2}d\omega \right)}$$

for all $n\in \mathbb{N}$.

2. 

If $F\left(\omega \right)$ is positive definite for all $\omega \in \mathbb{R}$, then:

$$\parallel {\left(T_n\left(F\right)\right)}^{-1}-T_n\left(F^{-1}\right){\parallel }_F\le \frac{1}{infF}\sqrt{p(p+1)\left(\frac{1}{2\pi }{\int }_0^{2\pi }{\parallel F\left(\omega \right)\parallel }_F^{2}d\omega \right)\left(\frac{1}{2\pi }{\int }_0^{2\pi }{\parallel {\left(F\left(\omega \right)\right)}^{-1}\parallel }_F^{2}d\omega \right)}$$

(8)

for all $n\in \mathbb{N}$.

Proof. 

See Appendix C. □

2.4. Block Circulant Matrices

Lemma 4.

Consider $A_n,B_n\in {\mathbb{C}}^{nM\times nN}$. Then:

$$\parallel C_{A_n}-C_{B_n}{\parallel }_F\le {\parallel A_n-B_n\parallel }_F$$

and:

$$\parallel A_n-C_{A_n}{\parallel }_F\le 2\parallel A_n-B_n{\parallel }_F+{\parallel B_n-C_{B_n}\parallel }_F.$$

(9)

Moreover, if $B_n$ is an $n\times n$ block circulant matrix with $M\times N$ blocks, then:

$$C_{B_n}=B_n$$

(10)

and:

$$\parallel A_n-C_{A_n}{\parallel }_F\le 2{\parallel A_n-B_n\parallel }_F.$$

(11)

Proof. 

See Appendix D. □

Lemma 5.

Let $F:\mathbb{R}\to {\mathbb{C}}^{M\times N}$ be a trigonometric polynomial of degree p. Then:

$$\parallel T_n\left(F\right)-{\widehat{C}}_n{\left(F\right)\parallel }_F\le {\parallel T_n\left(F\right)-C_n\left(F\right)\parallel }_F=\sqrt{\sum _{k=1}^{p}k\left(\parallel {\mathsf{F}}_k{\parallel }_F^2+{\parallel {\mathsf{F}}_{-k}\parallel }_F^2\right)}$$

for all $n>2p$. Furthermore,

$$\underset{n\to \infty }{lim}\parallel T_n\left(F\right)-{\widehat{C}}_n{\left(F\right)\parallel }_F=\underset{n\to \infty }{lim}{\parallel T_n\left(F\right)-C_n\left(F\right)\parallel }_F.$$

Proof. 

See Appendix E. □

3. Several New Results on the Correlation Matrices of Certain Random Vector Processes

Let $\left\{x_n\right\}$ be a (complex) random N-dimensional vector process, that is $x_n$ is a (complex) random N-dimensional (column) vector for all $n\in \mathbb{N}$. In this section, we study the boundedness of the sequence $\left\{{∥E\left(x_{n:1}{x}_{n:1}^{*}\right)-C_{E\left(x_{n:1}{x}_{n:1}^{*}\right)}∥}_F\right\}$ when $\left\{x_n\right\}$ is a WSS, MA, or AR vector process, where:

$$x_{n:1}=\left(\begin{array}{c}{x}_n\\ x_{n-1}\\ ⋮\\ x_1\end{array}\right),n\in \mathbb{N},$$

and E denotes expectation.

3.1. WSS Vector Processes

In this subsection, we review the concept of the WSS vector process, and we prove that the sequence $\left\{{∥E\left(x_{n:1}{x}_{n:1}^{*}\right)-C_{E\left(x_{n:1}{x}_{n:1}^{*}\right)}∥}_F\right\}$ is bounded when $\left\{x_n\right\}$ is a WSS vector process whose power spectral density (PSD) is a trigonometric polynomial.

Definition 1.

Let $X:\mathbb{R}\to {\mathbb{C}}^{N\times N}$ be continuous and $2\pi$-periodic. A random N-dimensional vector process $\left\{x_n\right\}$ is said to be WSS with PSD X if it has constant mean (i.e., $E\left(x_{n_1}\right)=E\left(x_{n_2}\right)$ for all $n_1,n_2\in \mathbb{N})$ and $\left\{E\left(x_{n:1}{x}_{n:1}^{*}\right)\right\}=\left\{T_n\left(X\right)\right\}$.

Lemma 6.

If $\left\{x_n\right\}$ is a WSS vector process whose PSD is a trigonometric polynomial, then $\left\{{∥E\left(x_{n:1}{x}_{n:1}^{*}\right)-C_{E\left(x_{n:1}{x}_{n:1}^{*}\right)}∥}_F\right\}$ is bounded.

Proof. 

This is a direct consequence of Lemma 5. □

3.2. VMA Processes

In this subsection, we review the concept of the MA vector (VMA) process, and we prove that the sequence $\left\{{∥E\left(x_{n:1}{x}_{n:1}^{*}\right)-C_{E\left(x_{n:1}{x}_{n:1}^{*}\right)}∥}_F\right\}$ is bounded when $\left\{x_n\right\}$ is a VMA process of finite order.

Definition 2.

A zero-mean random N-dimensional vector process $\left\{x_n\right\}$ is said to be a VMA process if:

$$x_n=w_n+\sum _{k=1}^{n-1}{\mathsf{G}}_{-k}{w}_{n-k}\forall n\in \mathbb{N},$$

(12)

where ${\mathsf{G}}_{-k}\in {\mathbb{C}}^{N\times N}$ for all $k\in \mathbb{N}$ and $\left\{w_n\right\}$ is a zero-mean WSS N-dimensional vector process whose PSD is an $N\times N$ positive semidefinite matrix Λ. If there exists $q\in \mathbb{N}$ such that ${\mathsf{G}}_{-k}=0_{N\times N}$ for all $k>q$, then $\left\{x_n\right\}$ is called a VMA process of (finite) order q or a VMA$\left(q\right)$ process.

Lemma 7.

If $\left\{x_n\right\}$ is a VMA$\left(q\right)$ process as in Definition 2, then $\left\{{∥E\left(x_{n:1}{x}_{n:1}^{*}\right)-C_{E\left(x_{n:1}{x}_{n:1}^{*}\right)}∥}_F\right\}$ is bounded.

Proof. 

See Appendix F. □

3.3. VAR Processes

In this subsection, we review the concept of the AR vector (VAR) process, and we study the boundedness of the sequence $\left\{{∥E\left(x_{n:1}{x}_{n:1}^{*}\right)-C_{E\left(x_{n:1}{x}_{n:1}^{*}\right)}∥}_F\right\}$ when $\left\{x_n\right\}$ is a VAR process of finite order.

Definition 3.

A zero-mean random N-dimensional vector process $\left\{x_n\right\}$ is said to be a VAR process if:

$$x_n=w_n-\sum _{k=1}^{n-1}{\mathsf{F}}_{-k}{x}_{n-k}\forall n\in \mathbb{N},$$

(13)

where ${\mathsf{F}}_{-k}\in {\mathbb{C}}^{N\times N}$ for all $k\in \mathbb{N}$ and $\left\{w_n\right\}$ is a zero-mean WSS N-dimensional vector process whose PSD is an $N\times N$ positive definite matrix Λ. If there exists $p\in \mathbb{N}$ such that ${\mathsf{F}}_{-k}=0_{N\times N}$ for all $k>p$, then $\left\{x_n\right\}$ is called a VAR process of (finite) order p or a VAR$\left(p\right)$ process.

Lemma 8.

Let $\left\{x_n\right\}$ be a VAR$\left(p\right)$ process as in Definition 3. Suppose that $F\left(\omega \right)=I_N+{\sum }_{k=1}^p{\mathrm{e}}^{-k\omega \mathrm{i}}{\mathsf{F}}_{-k}$ is invertible for all $\omega \in \mathbb{R}$ and $\{\parallel {\left(T_n\left(F\right)\right)}^{-1}{\parallel }_2\}$ is bounded. Then, $\left\{{∥E\left(x_{n:1}{x}_{n:1}^{*}\right)-C_{E\left(x_{n:1}{x}_{n:1}^{*}\right)}∥}_F\right\}$ is bounded.

Proof. 

See Appendix G. □

4. On the Asymptotic Optimality of a Low-Complexity Coding Strategy for Gaussian Vector Sources

4.1. Low-Complexity Coding Strategy Considered

In [1], Kolmogorov gave a formula for the RDF of a real zero-mean Gaussian N-dimensional vector $\mathrm{x}$ with a positive definite correlation matrix $E\left(\mathrm{x}{\mathrm{x}}^{\top }\right)$, namely,

$$R_{\mathrm{x}}\left(D\right)=\frac{1}{N}\sum _{k=1}^{N}max\left\{0,\frac{1}{2}ln\frac{{\lambda }_k\left(E\left(\mathrm{x}{\mathrm{x}}^{\top }\right)\right)}{\theta }\right\}\forall D\in \left(0,\frac{\mathrm{tr}\left(E\left(\mathrm{x}{\mathrm{x}}^{\top }\right)\right)}{N}\right],$$

where ⊤ stands for the transpose, $\mathrm{tr}$ denotes the trace, and $\theta$ is a real number satisfying:

$$D=\frac{1}{N}\sum _{k=1}^{N}min\left\{\theta ,{\lambda }_k\left(E\left(\mathrm{x}{\mathrm{x}}^{\top }\right)\right)\right\}.$$

We recall that $R_{\mathrm{x}}\left(D\right)$ can be thought of as the minimum rate (measured in nats) at which $\mathrm{x}$ can be encoded (compressed) in order to be able to recover it with a mean squared error (MSE) per dimension no larger than a given distortion D, that is:

$$\frac{E\left({∥\mathrm{x}-\tilde{\mathrm{x}}∥}_2^2\right)}{N}\le D,$$

where $\tilde{\mathrm{x}}$ denotes the estimation of $\mathrm{x}$.

If $D\in \left(0,{\lambda }_N\left(E\left(\mathrm{x}{\mathrm{x}}^{\top }\right)\right)\right]$, an optimal coding strategy to achieve $R_{\mathrm{x}}\left(D\right)$ is to encode ${\left[\mathrm{z}\right]}_{1,1},\dots ,{\left[\mathrm{z}\right]}_{N,1}$ separately with $E(\parallel {\left[\mathrm{z}\right]}_{k,1}-\widetilde{{\left[\mathrm{z}\right]}_{k,1}}{\parallel }_2^2)\le D$ for all $k\in \{1,\dots ,N\}$, where $\mathrm{z}=U^{\top }\mathrm{x}$ with U being a real orthogonal eigenvector matrix of $E\left(\mathrm{x}{\mathrm{x}}^{\top }\right)$ (see [8], Corollary 1). Observe that in order to obtain U, we need to know the correlation matrix $E\left(\mathrm{x}{\mathrm{x}}^{\top }\right)$. This coding strategy also requires an optimal coding method for real Gaussian random variables.

In [4], Theorem 3, we gave a low-complexity coding strategy for any Gaussian N-dimensional vector source $\left\{x_n\right\}$. According to that strategy, to encode a finite-length data block $x_{n:1}$ of such a source, we first compute the block discrete Fourier transform (DFT) of $x_{n:1}$:

$$y_{n:1}=\left(V_n^{*}\otimes I_N\right)x_{n:1},$$

(14)

and then, we encode $y_{\lceil \frac{n}{2}\rceil },\dots ,y_n$ separately (i.e., if n is even, we encode $y_{\frac{n}{2}},\widehat{y_{\frac{n}{2}+1}},\dots ,\widehat{y_{n-1}},y_n$ separately, and if n is odd, we encode $\widehat{y_{\frac{n+1}{2}}},\dots ,\widehat{y_{n-1}},y_n$ separately) with:

$$\frac{E\left({∥\widehat{y_k}-\widetilde{\widehat{y_k}}∥}_2^2\right)}{2N}\le \frac{D}{2},k\in \left\{⌈\frac{n}{2}⌉,\dots ,n-1\right\}\backslash \left\{\frac{n}{2}\right\},$$

and:

$$\frac{E\left({∥y_k-\widetilde{y_k}∥}_2^2\right)}{N}\le D,k\in \left\{\frac{n}{2},n\right\}\cap \mathbb{N},$$

where $\lceil x\rceil$ denotes the smallest integer higher than or equal to $x\in \mathbb{R}$ and:

$$\widehat{z}=\left(\begin{array}{c}\mathrm{Re}\left(z\right)\\ \mathrm{Im}\left(z\right)\end{array}\right)=\left(\begin{array}{c}\mathrm{Re}\left({\left[z\right]}_{1,1}\right)\\ ⋮\\ \mathrm{Re}\left({\left[z\right]}_{N,1}\right)\\ \mathrm{Im}\left({\left[z\right]}_{1,1}\right)\\ ⋮\\ \mathrm{Im}\left({\left[z\right]}_{N,1}\right)\end{array}\right)\forall z\in {\mathbb{C}}^{N\times 1}$$

with $\mathrm{Re}$ and $\mathrm{Im}$ being the real part and the imaginary part of a complex number, respectively.

As our coding strategy requires the computation of the block DFT, its computational complexity is $O(nNlogn)$ whenever the fast Fourier transform (FFT) algorithm is used. We recall that the computational complexity of the optimal coding strategy for $x_{n:1}$ is $O\left(n^2{N}^2\right)$ since it requires the computation of $U_n^{\top }{x}_{n:1}$, where $U_n$ is a real orthogonal eigenvector matrix of $E\left(x_{n:1}{x}_{n:1}^{\top }\right)$. Observe that such an eigenvector matrix $U_n$ also needs to be computed, which further increases the complexity. Hence, the main advantage of our coding strategy is that it notably reduces the computational complexity of coding $x_{n:1}$. Moreover, our coding strategy does not require the knowledge of $E\left(x_{n:1}{x}_{n:1}^{\top }\right)$. It only requires the knowledge of $E\left(\widehat{y_k}{\widehat{y_k}}^{\top }\right)$, with $k\in \{\lceil \frac{n}{2}\rceil ,\dots ,n\}$.

We finish this subsection by reviewing a result that provides an upper bound for the distance between $R_{x_{n:1}}\left(D\right)$ and the rate of our coding strategy ${\tilde{R}}_{x_{n:1}}\left(D\right)$ (see [4], Theorem 3).

Theorem 1.

Consider $n,N\in \mathbb{N}$. Let $x_k$ be a random N-dimensional vector for all $k\in \{1,\dots ,n\}$. Suppose that $x_{n:1}$ is a real zero-mean Gaussian vector with a positive definite correlation matrix (or equivalently, ${\lambda }_{nN}\left(E\left(x_{n:1}{x}_{n:1}^{\top }\right)\right)>0$). Let $y_{n:1}$ be the random vector given by Equation (14). If $D\in \left(0,{\lambda }_{nN}\left(E\left(x_{n:1}{x}_{n:1}^{\top }\right)\right)\right]$, then:

$$\begin{array}{c}\hfill 0\le {\tilde{R}}_{x_{n:1}}\left(D\right)-R_{x_{n:1}}\left(D\right)\le \frac{1}{2}ln\left(1+\frac{{∥E\left(x_{n:1}{x}_{n:1}^{\top }\right)-C_{E\left(x_{n:1}{x}_{n:1}^{\top }\right)}∥}_F}{\sqrt{nN}{\lambda }_{nN}\left(E\left(x_{n:1}{x}_{n:1}^{\top }\right)\right)}\right),\end{array}$$

(15)

where:

$${\tilde{R}}_{x_{n:1}}\left(D\right)=\left\{\begin{array}{cc}\frac{R_{y_{\frac{n}{2}}}\left(D\right)+2{\sum }_{k=\frac{n}{2}+1}^{n-1}{R}_{\widehat{y_k}}\left(\frac{D}{2}\right)+R_{y_n}\left(D\right)}{n}\hfill & \mathit{if} n \mathit{is} \mathit{even},\hfill \\ \frac{2{\sum }_{k=\frac{n+1}{2}}^{n-1}{R}_{\widehat{y_k}}\left(\frac{D}{2}\right)+R_{y_n}\left(D\right)}{n}\hfill & \mathit{if} n \mathit{is} \mathit{odd}.\hfill \end{array}\right.$$

4.2. On the Asymptotic Optimality of the Low-Complexity Coding Strategy Considered

In this subsection, we study the asymptotic optimality of our coding strategy for Gaussian vector sources. We begin by presenting a new result that provides a sufficient condition for the source to make such a coding strategy asymptotically optimal.

Theorem 2.

Let $\left\{x_n\right\}$ be a real zero-mean Gaussian N-dimensional vector process. Suppose that ${inf}_{n\in \mathbb{N}}{\lambda }_{nN}\left(E\left(x_{n:1}{x}_{n:1}^{\top }\right)\right)>0$ and ${lim}_{n\to \infty }\frac{{∥E\left(x_{n:1}{x}_{n:1}^{\top }\right)-C_{E\left(x_{n:1}{x}_{n:1}^{\top }\right)}∥}_F}{\sqrt{n}}=0$. If $D\in \left(0,{inf}_{n\in \mathbb{N}}{\lambda }_{nN}\left(E\left(x_{n:1}{x}_{n:1}^{\top }\right)\right)\right]$, then:

$$\underset{n\to \infty }{lim}\left({\tilde{R}}_{x_{n:1}}\left(D\right)-R_{x_{n:1}}\left(D\right)\right)=0.$$

Hence, if $\left\{R_{x_{n:1}}\left(D\right)\right\}$ is convergent, then:

$$\underset{n\to \infty }{lim}{\tilde{R}}_{x_{n:1}}\left(D\right)=\underset{n\to \infty }{lim}{R}_{x_{n:1}}\left(D\right).$$

(16)

Proof. 

From Equation (15), we have:

$$\begin{array}{cc}\hfill 0& \le {\tilde{R}}_{x_{n:1}}\left(D\right)-R_{x_{n:1}}\left(D\right)\le \frac{1}{2}ln\left(1+\frac{{∥E\left(x_{n:1}{x}_{n:1}^{\top }\right)-C_{E\left(x_{n:1}{x}_{n:1}^{\top }\right)}∥}_F}{\sqrt{nN}{inf}_{m\in \mathbb{N}}{\lambda }_{mN}\left(E\left(x_{m:1}{x}_{m:1}^{\top }\right)\right)}\right)\forall n\in \mathbb{N},\hfill \end{array}$$

and therefore, Theorem 2 is proven. □

We recall that ${lim}_{n\to \infty }{R}_{x_{n:1}}\left(D\right)$ is the RDF of the source $\left\{x_n\right\}$.

In [4], Theorem 4, we gave a more restrictive sufficient condition for the source to make the coding strategy considered asymptotically optimal. Specifically, in [4], Theorem 4, we proved that Equation (16) holds if $\left\{x_n\right\}$ is AWSS. However, the convergence speed of the rate of the coding strategy considered (i.e., how fast the rate of the coding strategy tends to the RDF of the AWSS vector source) was not studied in [4]. We now study the convergence speed of the rate of such a coding strategy when it is used to encode the most relevant vector sources, namely WSS vector sources, VMA sources, and VAR sources. It should be mentioned that this convergence speed depends on the sequence $\left\{{∥E\left(x_{n:1}{x}_{n:1}^{\top }\right)-C_{E\left(x_{n:1}{x}_{n:1}^{\top }\right)}∥}_F\right\}$ whose boundedness is studied in Section 3 for these three types of vector sources.

Theorem 3.

Let $\left\{x_n\right\}$ be a real zero-mean Gaussian WSS N-dimensional vector process whose PSD X is a trigonometric polynomial. Suppose that $infX>0$ (or equivalently, $det\left(X\right(\omega \left)\right)\ne 0$ for all $\omega \in \mathbb{R}$). If $D\in \left(0,infX\right]$, there exists $K\in [0,\infty )$ such that:

$$\begin{array}{cc}\hfill 0& \le {\tilde{R}}_{x_{n:1}}\left(D\right)-R_{x_{n:1}}\left(D\right)\le \frac{1}{2}ln\left(1+\frac{K}{\sqrt{n}}\right)\forall n\in \mathbb{N}.\hfill \end{array}$$

(17)

Proof. 

As $\left\{T_n\left(X\right)\right\}=\left\{E\left(x_{n:1}{x}_{n:1}^{*}\right)\right\}$, $T_n\left(X\right)$ is positive semidefinite for all $n\in \mathbb{N}$. Consequently, from [7], Proposition 3, $X\left(\omega \right)$ is positive semidefinite for all $\omega \in \mathbb{R}$. Therefore, applying Equation (1), $infX>0$ if and only if $det\left(X\right(\omega \left)\right)\ne 0$ for all $\omega \in \mathbb{R}$. Equation (17) is a direct consequence of Equation (1), Theorem 1, and Lemma 6. □

Theorem 4.

Let $\left\{x_n\right\}$ be a VMA(q) process as in Definition 2. Suppose that $det\left(\mathsf{\Lambda }\right)\ne 0$ and $\{\parallel {\left(T_n\left(G\right)\right)}^{-1}{\parallel }_2\}$ is bounded with $G\left(\omega \right)=I_N+{\sum }_{k=1}^q{\mathrm{e}}^{-k\omega \mathrm{i}}{\mathsf{G}}_{-k}$ for all $\omega \in \mathbb{R}$. If $\left\{x_n\right\}$ is real and Gaussian, and $D\in \left(0,{inf}_{n\in \mathbb{N}}{\lambda }_{nN}\left(E\left(x_{n:1}{x}_{n:1}^{\top }\right)\right)\right]$, there exists $K\in [0,\infty )$ such that:

$$\begin{array}{cc}\hfill 0& \le {\tilde{R}}_{x_{n:1}}\left(D\right)-R_{x_{n:1}}\left(D\right)\le \frac{1}{2}ln\left(1+\frac{K}{\sqrt{n}}\right)\forall n\in \mathbb{N}.\hfill \end{array}$$

Proof. 

Since $det\left(T_n\left(G\right)\right)=1$ for all $n\in \mathbb{N}$, from Equation (A3), we have:

$$\begin{array}{cc}\hfill & {\lambda }_{nN}\left(E\left(x_{n:1}{x}_{n:1}^{\top }\right)\right)=\frac{1}{{∥{\left(E\left(x_{n:1}{x}_{n:1}^{\top }\right)\right)}^{-1}∥}_2}=\frac{1}{{∥{\left(T_n\left(G\right)T_n\left(\mathsf{\Lambda }\right){\left(T_n\left(G\right)\right)}^{*}\right)}^{-1}∥}_2}\hfill \\ \hfill & =\frac{1}{{∥{\left({\left(T_n\left(G\right)\right)}^{-1}\right)}^{*}{T}_n\left({\mathsf{\Lambda }}^{-1}\right){\left(T_n\left(G\right)\right)}^{-1}∥}_2}\ge \frac{1}{{∥{\left({\left(T_n\left(G\right)\right)}^{-1}\right)}^{*}∥}_2{∥T_n\left({\mathsf{\Lambda }}^{-1}\right)∥}_2{∥{\left(T_n\left(G\right)\right)}^{-1}∥}_2}\hfill \\ \hfill & =\frac{{\lambda }_N\left(\mathsf{\Lambda }\right)}{{∥{\left(T_n\left(G\right)\right)}^{-1}∥}_2^2}\ge \frac{{\lambda }_N\left(\mathsf{\Lambda }\right)}{{\left({sup}_{m\in \mathbb{N}}{∥{\left(T_m\left(G\right)\right)}^{-1}∥}_2\right)}^2}>0\forall n\in \mathbb{N}.\hfill \end{array}$$

Hence, ${inf}_{n\in \mathbb{N}}{\lambda }_{nN}\left(E\left(x_{n:1}{x}_{n:1}^{\top }\right)\right)>0$. Theorem 1 and Lemma 7 prove Theorem 4. □

Theorem 5.

Let $\left\{x_n\right\}$ be a VAR(p) process as in Definition 3. Suppose that $F\left(\omega \right)=I_N+{\sum }_{k=1}^p{\mathrm{e}}^{-k\omega \mathrm{i}}{\mathsf{F}}_{-k}$ is invertible for all $\omega \in \mathbb{R}$ and $\{\parallel {\left(T_n\left(F\right)\right)}^{-1}{\parallel }_2\}$ is bounded. If $\left\{x_n\right\}$ is real and Gaussian and $D\in \left(0,{inf}_{n\in \mathbb{N}}{\lambda }_{nN}\left(E\left(x_{n:1}{x}_{n:1}^{\top }\right)\right)\right]$, there exists $K\in [0,\infty )$ such that:

$$\begin{array}{cc}\hfill 0& \le {\tilde{R}}_{x_{n:1}}\left(D\right)-R_{x_{n:1}}\left(D\right)\le \frac{1}{2}ln\left(1+\frac{K}{\sqrt{n}}\right)\forall n\in \mathbb{N}.\hfill \end{array}$$

Proof. 

As $det\left(T_n\left(F\right)\right)=1$ for all $n\in \mathbb{N}$, applying Equation (A4) and [6], Theorem 4.3, yields:

$$\begin{array}{cc}\hfill {\lambda }_{nN}\left(E\left(x_{n:1}{x}_{n:1}^{\top }\right)\right)& =\frac{1}{{∥{\left(E\left(x_{n:1}{x}_{n:1}^{\top }\right)\right)}^{-1}∥}_2}=\frac{1}{{∥{\left({\left(T_n\left(F\right)\right)}^{-1}{T}_n\left(\mathsf{\Lambda }\right){\left({\left(T_n\left(F\right)\right)}^{*}\right)}^{-1}\right)}^{-1}∥}_2}\hfill \\ \hfill & =\frac{1}{{∥{\left(T_n\left(F\right)\right)}^{*}{T}_n\left({\mathsf{\Lambda }}^{-1}\right)T_n\left(F\right)∥}_2}\ge \frac{1}{{∥{\left(T_n\left(F\right)\right)}^{*}∥}_2{∥T_n\left({\mathsf{\Lambda }}^{-1}\right)∥}_2{∥T_n\left(F\right)∥}_2}\hfill \\ \hfill & =\frac{{\lambda }_N\left(\mathsf{\Lambda }\right)}{{∥T_n\left(F\right)∥}_2^2}\ge \frac{{\lambda }_N\left(\mathsf{\Lambda }\right)}{{\left({sup}_{m\in \mathbb{N}}{∥T_m\left(F\right)∥}_2\right)}^2}>0\forall n\in \mathbb{N}.\hfill \end{array}$$

Thus, ${inf}_{n\in \mathbb{N}}{\lambda }_{nN}\left(E\left(x_{n:1}{x}_{n:1}^{\top }\right)\right)>0$. Theorem 1 and Lemma 8 prove Theorem 5. □

4.3. On How the Low-Complexity Coding Strategy Considered Performs under Perturbations

In this subsection, we study how the low-complexity coding strategy considered performs when it is used to encode a perturbed version, $\left\{z_n\right\}$, of a WSS, MA, or AR vector source $\left\{x_n\right\}$. Observe that if $\left\{{∥E\left(z_{n:1}{z}_{n:1}^{\top }\right)-E\left(x_{n:1}{x}_{n:1}^{\top }\right)∥}_F\right\}$ is bounded, from Equation (9), we conclude that our coding strategy can also be used to optimally encode $\left\{z_n\right\}$, and the convergence speed of the rate remains unaltered.

We now present three numerical examples that show how the coding strategy considered performs in the presence of a perturbation. In all of them, $N=2$ and:

$$E\left(z_{n:1}{z}_{n:1}^{\top }\right)=E\left(x_{n:1}{x}_{n:1}^{\top }\right)+\left(\begin{array}{cc}{0}_{2n-2\times 2n-2}& 0_{2n-2\times 2}\\ 0_{2\times 2n-2}& I_2\end{array}\right)\forall n\in \mathbb{N}.$$

Obviously, $\left\{{∥E\left(z_{n:1}{z}_{n:1}^{\top }\right)-E\left(x_{n:1}{x}_{n:1}^{\top }\right)∥}_F\right\}$ is bounded since ${∥E\left(z_{n:1}{z}_{n:1}^{\top }\right)-E\left(x_{n:1}{x}_{n:1}^{\top }\right)∥}_F=\sqrt{2}$ for all $n\in \mathbb{N}$. The three vector sources $\left\{x_n\right\}$ considered in our numerical examples are the zero-mean WSS vector source in [9], Section 4, the VMA(1) source in [10], Example 2.1, and the VAR(1) source in [10], Example 2.3. In [9], Section 4, the Fourier coefficients of the PSD X are:

$${\mathsf{X}}_0=\left(\begin{array}{cc}2.0002& 0.7058\\ 0.7058& 2.0000\end{array}\right),{\mathsf{X}}_{-1}={\mathsf{X}}_1^{*}=\left(\begin{array}{cc}-0.3542& 0.1016\\ 0.1839& -0.2524\end{array}\right),{\mathsf{X}}_{-2}={\mathsf{X}}_2^{*}=\left(\begin{array}{cc}-0.0923& 0.0153\\ 0.1490& 0.0696\end{array}\right),$$

$${\mathsf{X}}_{-3}={\mathsf{X}}_3^{*}=\left(\begin{array}{cc}-0.1443& -0.0904\\ 0.0602& 0.0704\end{array}\right),{\mathsf{X}}_{-4}={\mathsf{X}}_4^{*}=\left(\begin{array}{cc}-0.0516& -0.0603\\ 0& 0\end{array}\right),$$

and ${\mathsf{X}}_k=0_{2\times 2}$ with $\left|k\right|>4$. In [10], Example 2.1, ${\mathsf{G}}_{-1}$ and $\mathsf{\Lambda }$ are given by:

$$\left(\begin{array}{cc}-0.8& -0.7\\ 0.4& -0.6\end{array}\right)$$

(18)

and:

$$\left(\begin{array}{cc}4& 1\\ 1& 2\end{array}\right),$$

(19)

respectively. In [10], Example 2.3, ${\mathsf{F}}_{-1}$ and $\mathsf{\Lambda }$ are given by Equations (18) and (19), respectively.

Figure 1a, Figure 2a and Figure 3a show $R_{x_{n:1}}\left(D\right)$ and ${\tilde{R}}_{x_{n:1}}\left(D\right)$ for the three vector sources $\left\{x_n\right\}$ considered by assuming that they are Gaussian. Figure 1b, Figure 2b and Figure 3b show $R_{z_{n:1}}\left(D\right)$ and ${\tilde{R}}_{z_{n:1}}\left(D\right)$ for these three vector sources. In Figure 1, Figure 2 and Figure 3, $n\le 100$ and $D=0.001$. The figures bear the evidence of the fact that the rate of the low-complexity coding strategy considered tends to the RDF of the source even in the presence of a perturbation.

5. Conclusions

In [4], we proposed a low-complexity coding strategy to encode finite-length data blocks of any Gaussian vector source. In this paper, we proved that the convergence speed of the rate of our coding strategy is $O\left(\frac{1}{\sqrt{n}}\right)$ when it is used to encode the most relevant vector sources, namely WSS, MA, and AR vector sources. This means that the rate of our coding strategy will be close enough to the RDF of the source even if the length n of the data blocks is relatively small. Therefore, we conclude that our coding strategy is not only low-complexity and asymptotically optimal, but also low-latency. These three features make our coding strategy very useful in practical coding applications.