Strong Ergodicity in Nonhomogeneous Markov Systems with Chronological Order

Abstract

In the present, we study the problem of strong ergodicity in nonhomogeneous Markov systems. In the first basic theorem, we relax the fundamental assumption present in all studies of asymptotic behavior. That is, the assumption that the inherent inhomogeneous Markov chain converges to a homogeneous Markov chain with a regular transition probability matrix. In addition, we study the practically important problem of the rate of convergence to strong ergodicity for a nonhomogeneous Markov system (NHMS). In a second basic theorem, we provide conditions under which the rate of convergence to strong ergodicity is geometric. With these conditions, we in fact relax the basic assumption present in all previous studies, that is, that the inherent inhomogeneous Markov chain converges to a homogeneous Markov chain with a regular transition probability matrix geometrically fast. Finally, we provide an illustrative application from the area of manpower planning.

1. Introductory Notes

Consider a stochastic system that has a population of members categorized in different states. Three types of movements are possible in the system. Firstly, movements of members are probable among the states of the system; secondly, members are leaving the system from the various states; and thirdly, new members are entering the system to replace leavers and to expand the population. When the various movements of the system are modeled by a nonhomogeneous Markov chain, we call such a system a nonhomogeneous Markov system (NHMS).

An NHMS is actually a generalization of the classical Markov chain where we have one particle moving among the states without the possibility of leaving the system and probably being replaced by another with possibly different characteristics. Of great importance is the vector of absolute probabilities which consists of the probabilities of the particle to be in any state of the Markov chain. On the other hand, in an NHMS, we have a population of particles categorized according to their characteristics in the various states. Particles are leaving the population from all the states, and new particles are entering the population to replace them and to expand the population. Of great importance is the vector of the expected relative population structure. Hence, the problems to be solved are a lot harder, and new strategies and tools are used other than the simple Markov chain. The roots of the motive for the development of the theory of NHMS, which was first introduced in Vassiliou [1], could be summarized in the use of Markov models in manpower systems. This started with the work of Young and Almond [2], Young [3,4], and Bartholomew [5,6], and it was extended in the works of Young and Vassiliou [7], Vassiliou [8,9], and McClean [10,11]. In the book by Vassiliou [12], one can find the evolution of the theory of NHMS and the large diversity of its developments in various directions, that is, NHMS in discrete and continuous time, stochastic control in NHMS, Laws of Large Numbers for NHMS, Perturbations theory, NHMS in a stochastic environment, Markov systems, and others. In Section 5.4 of [12], there is a synopsis of real and potential applications of NHMS which illustrates the breadth of applications and some of the reasons why the entire theory is central to these processes. Work on manpower planning using results from NHMS and new areas has continued throughout the years up to nowadays, for example, Garg, et al. [13,14,15], Ugwugo and McClean [16], Vassiliadis [17,18], Georgiou, et al. [19], Guerry [20,21], Pollard [22], Esquivel, et al. [23,24,25].

In Section 2 of the present study, we define and describe the NHMS in discrete time and space in a compact but hopefully readable way for the reader who comes in contact for the first time with these processes. We also state the expected relative population structure in the various states as a function of the parameters of the population that could be estimated from the available data. In Section 3, we start with some basic definitions of concepts and mathematical tools, as well as useful known results that will be used in what follows. The novel part of this section is Theorem 1, where we study strong ergodicity for NHMS by relaxing the basic assumption present in all studies of strong ergodicity for NHMS. That is, we will not assume that the inherent inhomogeneous Markov chain converges as time goes to infinity to a homogeneous Markov chain with a regular (it consists of one communicating class of states, which is aperiodic) transition probability matrix or, equivalently, that the inhomogeneous Markov chain is strongly ergodic. In Section 3, we start with some basic definitions of concepts and mathematical tools, as well as useful known results that will be used in what follows. The novel part of this section is Theorem 4, where we prove under what conditions the rate of convergence of strong ergodicity in a NHMS is geometrically fast. This is an important question in NHMS due to its large practical value. In Theorem 4, we relax the basic assumption present in all studies of the rate of convergence to its asymptotic behavior for a NHMS. That is, we will not assume that the inherent inhomogeneous Markov chain converge as time goes to infinity to a homogeneous Markov chain with a regular transition probability matrix geometrically fast. Finally, in Section 5, we provide an illustrative application from the area of manpower planning.

2. The NHMS in Discrete Time and Space

Let a population consist of any kind of entities and let us denote by $T\left(t\right)$ for $t=1,2,\dots$, the total number of memberships at time t, that is, at the end of the interval $\left(t-1,t\right]$, which are being held by its members. At every point of time that a member leaves the population, the membership is being transferred to a new member. For example, members could be patients in a hospital and memberships the beds they occupy. It is assumed that the total number of memberships are known or the sequence ${\left\{T\left(t\right)\right\}}_{t=0}^{\infty }$ is a realization of a known stochastic process depending on the application. The memberships are distributed in a finite number of states and let $S=\left\{1,2,\dots ,k\right\}$ be the state space. Important aspect of an NHMS (see Vassiliou [12], Section 5.2) is the population structure, that is, the vector of random variables

$$\mathbf{N}\left(t\right)=\left[N_1\left(t\right),N_2\left(t\right),\dots ,N_k\left(t\right)\right],$$

where $N_i\left(t\right)$ is the number of memberships in state i at time t. Also, very important is the relative population structure, which is the vector of random variables $\mathbf{q}\left(t\right)=\mathbf{N}\left(t\right)/T\left(t\right)$. We denote by $\mathbf{P}\left(t\right)$ the transition probability matrix of the internal transitions of the members of the population during the interval $\left(t-1,t\right]$, that is, the t-th interval. Also, we have probable leavers from the states in S in every time interval t and let us denote by

$${\mathbf{p}}_{k+1}\left(t\right)=\left[p_{1,k+1}\left(t\right),p_{2,k+1}\left(t\right),\dots ,p_{k,k+1}\left(t\right)\right],$$

where the state $k+1$ represents the external environment. Finally, we have new entrants of memberships to the population in order to replace leavers and to expand the population. Let us collect the probabilities of allocation of the memberships to the various states in the t-th interval in the following stochastic vector

$${\mathbf{p}}_0\left(t\right)=\left[p_{01}\left(t\right),p_{02}\left(t\right),\dots ,p_{0k}\left(t\right)\right],$$

where the state 0 represents the new entrants with their memberships waiting to be allocated in the various states. Note that in what follows we assume that $\Delta T\left(t\right)=T\left(t\right)-T\left(t-1\right)\ge 0$.

The transition probability matrix of the memberships $\mathbf{Q}\left(t\right)$ in the t-th interval can be shown (Vassiliou [12], p. 193), and it is given by

$$\mathbf{Q}\left(t\right)=\mathbf{P}\left(t\right)+{\mathbf{p}}_{k+1}\left(t\right){\mathbf{p}}_0\left(t\right).$$

(1)

We call the inhomogeneous Markov chain defined uniquely by the sequence of transition probability matrices ${\left\{\mathbf{Q}\left(t\right)\right\}}_{t=0}^{\infty }$ the embedded or inherent nonhomogeneous Markov chain of NHMS. A population or any physical phenomenon that could be modeled in the above described way is defined to be a nonhomogeneous Markov system.

Now, as previously, we define the relative population structure for a population which started at time s and is at time t to be $\mathbf{q}\left(s,t\right)=\mathbf{N}\left(s,t\right)/T\left(t\right)$, where $\mathbf{N}\left(s,t\right)$ is the population structure for the population. It could be proved that (Georgiou and Vassiliou [26] and Vassiliou [12], p. 195)

$$\mathbb{E}\left[\mathbf{q}\left(s,t\right)\right]=\mathbb{E}\left[\mathbf{q}\left(s,t-1\right)\right]\alpha \left(t-1\right)\mathbf{Q}\left(t\right)+b\left(t-1\right){\mathbf{p}}_0\left(t\right),$$

(2)

where s is the initial time and, therefore, $\mathbf{q}\left(s\right)$ is the initial relative population structure which is known and

$$\alpha \left(t-1\right)=\frac{T\left(t-1\right)}{T\left(t\right)}\mathrm{and}b\left(t-1\right)=\frac{T\left(t\right)-T\left(t-1\right)}{T\left(t\right)},$$

(3)

and where $\mathbf{q}\left(s,t\right)$ is the relative population structure at time t for the system that started with initial relative population structure $\mathbf{q}\left(s\right)$, which apparently is a random variable, and we denote by $\mathbb{E}\left[\mathbf{q}\left(s,t\right)\right]$ its expected value.

From (2), recursively, we obtain

$$\mathbb{E}\left[\mathbf{q}\left(s,t\right)\right]=\mathbf{q}\left(s\right)\frac{T\left(s\right)}{T\left(t\right)}\mathbf{Q}\left(s,t\right)+\frac{1}{T\left(t\right)}\sum _{\tau =1}^t\Delta T\left(s+t\right){\mathbf{p}}_0\left(s+t\right)\mathbf{Q}\left(s+\tau ,t\right),$$

(4)

where

$$\mathbf{Q}\left(s,t\right)=\mathbf{Q}\left(s+1\right)\mathbf{Q}\left(s+2\right)\dots \mathbf{Q}\left(t\right).$$

3. Strong Ergodicity in NHMS with Chronological Order

The asymptotic behavior of NHMS and of nonhomogeneous Markov chains, as well as of homogeneous Markov chains, has been one of the central problems for many years, as can be seen in Refs. [12,27,28,29,30,31]. The asymptotic behavior of NHMS started with Vassiliou [1,32], and an updated evolution of these theorems and their variants could be found in Vassiliou [12]. In the present section, we provide and prove a basic theorem for strong ergodicity in NHMS when the transition probabilities matrices of the inherent Markov chain ${\left\{\mathbf{Q}\left(t\right)\right\}}_{t=0}^{\infty }$ are given in chronological order; that is, it is assumed that the time order of the elements of the sequence ${\left\{\mathbf{Q}\left(t\right)\right\}}_{t=0}^{\infty }$ is given and will not be changed. In Theorem 1 we relax the basic assumption present in all studies of asymptotic behavior for NHMS. That is, we will not assume that the inherent inhomogeneous Markov chain converge as time goes to infinity to a homogeneous Markov chain with a regular (it consists of one communicating class of states which is aperiodic) transition probability matrix or, equivalently, that the inhomogeneous Markov chain is strongly ergodic. We start with some basic definitions and results, which will be useful in what follows.

For what follows, we assume a complete probability space $\left(\Omega ,\mathcal{F},\mathbb{P}\right)$ and consider an NHMS in discrete time and space. Therefore, we will not repeat this in every Definition, Lemma, Proposition, or Theorem.

Definition 1.

We say that the NHMS is strongly ergodic if and only if there exists a stochastic vector ψ such that

$$\underset{\nu \to \infty }{lim}∥\mathbb{E}\left[\mathbf{q}\left(t,t+\nu \right)\right]-\mathit{\psi }∥=0.\mathrm{for}t=0,1,2,\dots ,$$

(5)

where $∥.∥$ from now on is any vector norm, except if it is otherwise stated.

Definition 2.

If $\mathbf{Q}$ is any finite stochastic matrix with state space $S=\left\{1,2,\dots ,k\right\}$, then the Dobrushin ergodicity coefficient that is known is given by

$${\tau }_1\left(\mathbf{Q}\right)=\frac{1}{2}\underset{i,r}{max}\sum _{j=1}^k\left|q_{ij}-q_{kj}\right|.$$

(6)

We now define another class of ergodicity coefficients which will be generated by different norms.

Definition 3

(Seneta [27]). Let the set

$$D^n=\left\{\mathbf{x}:\mathbf{x}\in {\mathbb{R}}^n,\mathbf{x}\ge \mathbf{0},\mathbf{x}{\mathbf{1}}^{\top }=1\right\},$$

and by $d\left(\dots \right)$ any metric on this set. Then, the quantity

$$\tau \left(\mathbf{P}\right)=\underset{\mathbf{x},\mathbf{y}\in D^n}{sup}\frac{d\left(\mathbf{xP},\mathbf{yP}\right)}{d\left(\mathbf{x},\mathbf{y}\right)}\mathit{with}\mathbf{x}\ne \mathbf{y},$$

(7)

for any stochastic matrix, $\mathbf{P}$ is called a coefficient of ergodicity.

Remark 1.

All matrix norms on ${\mathbb{R}}^n$ provide an appropriate metric on $D^n$ via $d\left(\mathbf{x},\mathbf{y}\right)=∥\mathbf{x}-\mathbf{y}∥$. Then for any stochastic matrix $\mathbf{P}$ we get

$${\tau }_{∥.∥}\left(\mathbf{P}\right)=\underset{\mathbf{x},\mathbf{y}\in D^n}{sup}\frac{∥\left(\mathbf{x}-\mathbf{y}\right)\mathbf{P}∥}{∥\mathbf{x}-\mathbf{y}∥}\mathit{with}\mathbf{x}\ne \mathbf{y}.$$

(8)

We call ${\tau }_{∥.∥}\left(\mathbf{P}\right)$ the coefficient of ergodicity induced by the norm $∥.∥$. It is proved that (8) can be written equivalently (Vassiliou [12], p. 118) as follows:

$${\tau }_{∥.∥}\left(\mathbf{P}\right)=\underset{\begin{array}{c}∥z∥=1\\ \mathbf{z}{\mathbf{1}}^{\top }=1\end{array}}{sup}∥\mathbf{zP}∥\mathit{over}\mathit{all}\mathbf{z}\in {\mathbb{R}}^n.$$

(9)

When the $L_1$ norm is used, that is, for an $n\times n$ matrix $\mathbf{A}$ with elements from $\mathbb{C}$, we obtain that

$${∥\mathbf{A}∥}_1=\sum _{i,j=1}^n\left|a_{ij}\right|,$$

(10)

then

$${\tau }_1\left(\mathbf{P}\right)=\underset{\begin{array}{c}{∥z∥}_1=1\\ \mathbf{z}{\mathbf{1}}^{\top }=1\end{array}}{sup}{∥\mathbf{zP}∥}_1\mathit{over}\mathit{all}\mathbf{z}\in {\mathbb{R}}^n.$$

(11)

We now give the definitions of strong and weak ergodicity for a nonhomogeneous Markov chain:

Definition 4.

Consider an inhomogeneous Markov chain ${\left\{X_t\right\}}_{t=0}^{\infty }$ in discrete time and space. We say that ${\left\{X_t\right\}}_{t=0}^{\infty }$ with a sequence of transition probability matrices ${\left\{\mathbf{Q}\left(t\right)\right\}}_{t=0}^{\infty }$ is strongly ergodic if there exists a stable stochastic matrix $\mathbf{Q}$ such that for every t

$$\underset{\nu \to \infty }{lim}∥\mathbf{Q}\left(t,t+\nu \right)-\mathbf{Q}∥=0.$$

(12)

If the limit is zero uniformly in t we say that ${\left\{X_t\right\}}_{t=0}^{\infty }$ or equivalently ${\left\{\mathbf{Q}\left(t\right)\right\}}_{t=0}^{\infty }$ is uniformly strongly ergodic.

Definition 5.

Consider an inhomogeneous Markov chain ${\left\{X_t\right\}}_{t=0}^{\infty }$ in discrete time and space. We say that ${\left\{X_t\right\}}_{t=0}^{\infty }$ with sequence of transition probability matrices ${\left\{\mathbf{Q}\left(t\right)\right\}}_{t=0}^{\infty }$ is weakly ergodic if for all states $i,j,r,t$

$$q_{ir}\left(t,t+\nu \right)-q_{jr}\left(t,t+\nu \right)\to 0\mathit{as}\nu \to \infty .$$

(13)

Note that in (13), ${lim}_{\nu \to \infty }{q}_{ij}\left(t,t+\nu \right)$ is not actually necessary to exist.

Remark 2.

Note that equivalently a non-homogeneous Markov chain is weakly ergodic if ${\tau }_1\left(\mathbf{Q}\left(t,t+\nu \right)\right)<1$ for every t.

We now state the following Lemma the proof of which exists in Vassiliou ([12], p. 119).

Lemma 1.

The coefficient of ergodicity generated by any metric as in (7) or induced by any vector norm on ${\mathbb{R}}^n$ has the following properties:

(1)

$\tau \left({\mathbf{P}}_1{\mathbf{P}}_2\right)\le \tau \left({\mathbf{P}}_1\right)\tau \left({\mathbf{P}}_2\right)\mathit{for}\mathit{any}{\mathbf{P}}_1,{\mathbf{P}}_2\mathit{stochastic}\mathit{matrices}.$

(2)

$\mathit{For}\mathit{any}\mathit{stochastic}\mathit{matrix}\mathbf{P},\tau \left(\mathbf{P}\right)=0\mathit{if}\mathit{and}\mathit{only}\mathit{if}rank\left(\mathbf{P}\right)=1.$

The following Lemma proof, which can be found in Paz [33], is useful in what follows:

Lemma 2.

If $\mathbf{P}$ is a stochastic matrix and if $\mathbf{R}$ is any real matrix such that $\mathbf{R}{\mathbf{1}}^{\top }=\mathbf{0}$ and in addition $∥\mathbf{R}∥<\infty$, then

$$∥\mathbf{RP}∥\le ∥\mathbf{R}∥{\tau }_1\left(\mathbf{P}\right).$$

We now state and prove one of the basic theorems of the present paper.

Theorem 1.

Let there be the complete probability space $\left(\Omega ,\mathcal{F},\mathbb{P}\right)$ and consider an NHMS in discrete time and space. We assume that

$$\underset{t\to \infty }{lim}T\left(t\right)=T\mathit{with}\Delta T\left(t\right)\ge 0.$$

(14)

Let ${\left\{Y\left(t\right)\right\}}_{t=0}^{\infty }$ be the inherent nonhomogeneous Markov chain of the movement of memberships. If ${\left\{Y\left(t\right)\right\}}_{t=0}^{\infty }$ is weakly ergodic with ${\left\{\mathbf{Q}\left(t\right)\right\}}_{t=0}^{\infty }$, the sequence of transition probabilities, and in addition, there exists a stochastic vector

$$\mathit{\psi }\left(t\right)=\left[{\psi }_1\left(t\right),{\psi }_2\left(t\right),\dots ,{\psi }_k\left(t\right)\right],$$

(15)

which is the left eigenvector of $\mathbf{Q}\left(t\right)$ for $t=0,1,2,\dots$, that is

$$\mathit{\psi }\left(t\right)=\mathit{\psi }\left(t\right)\mathbf{Q}\left(t\right)\mathrm{for}t=0,1,2,\dots ,$$

(16)

and

$$\sum _{t=0}^{\infty }∥\mathit{\psi }\left(t+1\right)-\mathit{\psi }\left(t\right)∥<\infty ,$$

(17)

then the NHMS is strongly ergodic.

Proof. 

From (17) we obtain that there exists a stochastic vector $\mathit{\psi }$ such that

$$\underset{t\to \infty }{lim}∥\mathit{\psi }\left(t\right)-\mathit{\psi }∥=0.$$

(18)

From (16) and (18), we obtain that there exists a stochastic matrix $\mathbf{Q}$ such that

$$\underset{t\to \infty }{lim}∥\mathbf{Q}\left(t\right)-\mathbf{Q}∥=0\mathrm{with}\mathbf{Q}{\mathbf{1}}^{\top }={\mathbf{1}}^{\top }.$$

(19)

Denote by $\mathbf{\Psi }\left(t\right)$ to be the stable matrix with row the stochastic vector $\mathit{\psi }\left(t\right)$ and $\mathbf{\Psi }$ the stable matrix with row $\mathit{\psi }$. Then, we have

$$∥\mathbf{\Psi }\left(t+1\right)-\mathbf{\Psi }\left(t\right)∥=∥\mathit{\psi }\left(t+1\right)-\mathit{\psi }\left(t\right)∥\mathrm{and}∥\mathbf{\Psi }\left(t\right)-\mathbf{\Psi }∥=∥\mathit{\psi }\left(t\right)-\mathit{\psi }∥.$$

(20)

We now show that

$$\underset{\nu \to \infty }{lim}∥\mathbf{Q}\left(t,t+\nu \right)-\mathbf{\Psi }∥=0.$$

(21)

We have that

$$\mathbf{Q}\left(t,t+\nu \right)-\mathbf{\Psi }=\mathbf{Q}\left(t,t+r\right)\mathbf{Q}\left(t+r,t+\nu \right)-\mathbf{\Psi }\left(t+r\right)\mathbf{Q}\left(t+r,t+\nu \right)+$$

$$\mathbf{\Psi }\left(t+r\right)\mathbf{Q}\left(t+r,t+\nu \right)-\mathbf{\Psi }\left(t+\nu -1\right)+\mathbf{\Psi }\left(t+\nu -1\right)-\mathbf{\Psi }.$$

(22)

Taking norms on (22) we obtain that

$$∥\mathbf{Q}\left(t,t+\nu \right)-\mathbf{\Psi }∥\le ∥\mathbf{Q}\left(t,t+r\right)\mathbf{Q}\left(t+r,t+\nu \right)-\mathbf{\Psi }\left(t+r\right)\mathbf{Q}\left(t+r,t+\nu \right)∥$$

$$+∥\mathbf{\Psi }\left(t+r\right)\mathbf{Q}\left(t+r,t+\nu \right)-\mathbf{\Psi }\left(t+\nu -1\right)∥+∥\mathbf{\Psi }\left(t+\nu -1\right)-\mathbf{\Psi }∥.$$

(23)

We now have

$$N_1\left(t,\nu \right)=∥\mathbf{Q}\left(t,t+r\right)\mathbf{Q}\left(t+r,t+\nu \right)-\mathbf{\Psi }\left(t+r\right)\mathbf{Q}\left(t+r,t+\nu \right)∥\le$$

$$∥[\mathbf{Q}\left(t,t+r\right)-\mathbf{\Psi }\left(t+r\right)]\mathbf{Q}\left(t+r,t+\nu \right)∥\le (\mathrm{Due}\mathrm{to}\mathrm{Lemma}2)\le$$

$$∥\mathbf{Q}\left(t,t+r\right)-\mathbf{\Psi }\left(t+r\right)∥{\tau }_1\left(\mathbf{Q}\left(t+r,t+\nu \right)\right)\le 2{\tau }_1\left(\mathbf{Q}\left(t+r,t+\nu \right)\right),$$

(24)

where ${\tau }_1\left(\mathbf{Q}\left(t+r,t+\nu \right)\right)$ is less than one due to weak ergodicity of ${\left\{Y\left(t\right)\right\}}_{t=0}^{\infty }$ the inherent nonhomogeneous Markov chain. Also for fixed r, we can always choose $\nu$ such that

$$N_1\left(t,\nu \right)\le \frac{\epsilon }{3},\mathrm{with}\epsilon >0\mathrm{a}\mathrm{small}\mathrm{number}.$$

(25)

We now have that

$$\mathbf{\Psi }\left(t+r\right)\mathbf{Q}\left(t+r,t+\nu \right)=\left(\mathrm{due}\mathrm{to}\mathrm{Equation}\left(16\right)\right)=$$

$$\left[\mathbf{\Psi }\left(t+r\right)-\mathbf{\Psi }\left(t+r+1\right)\right]\mathbf{Q}\left(t+r+1,t+\nu \right)+$$

$$+\mathbf{\Psi }\left(t+r+1\right)\mathbf{Q}\left(t+r+1,t+\nu \right).$$

(26)

Now, similarly, we obtain

$$\mathbf{\Psi }\left(t+r+1\right)\mathbf{Q}\left(t+r+1,t+\nu \right)=$$

$$\left[\mathbf{\Psi }\left(t+r+1\right)-\mathbf{\Psi }\left(t+r+2\right)\right]\mathbf{Q}\left(t+r+2,t+\nu \right)$$

$$+\mathbf{\Psi }\left(t+r+2\right)\mathbf{Q}\left(t+r+2,t+\nu \right),$$

(27)

and using this equation recursively, we obtain

$$\mathbf{\Psi }\left(t+r\right)\mathbf{Q}\left(t+r,t+\nu \right)=\sum _{j=t+r+1}^{t+\nu -1}\left[\mathbf{\Psi }\left(j-1\right)-\mathbf{\Psi }\left(j\right)\right]\mathbf{Q}\left(j,t+\nu \right)$$

$$+\mathbf{\Psi }\left(t+\nu -1\right)\mathbf{Q}\left(t+\nu -1,t+\nu \right).$$

(28)

From condition (16), we obtain that

$$\mathbf{\Psi }\left(t+\nu -1\right)=\mathbf{\Psi }\left(t+\nu -1\right)\mathbf{Q}\left(t+\nu -1\right),$$

(29)

and

$$\mathit{\psi }\left(t+\nu -1\right)=\mathit{\psi }\mathbf{\Psi }\left(t+\nu -1\right).$$

(30)

Therefore, from (28)–(30), as well as, Lemma 2, we have that

$$N_2\left(t,\nu \right)=∥\mathbf{\Psi }\left(t+r\right)\mathbf{Q}\left(t+r,t+\nu \right)-\mathbf{\Psi }\mathbf{\Psi }\left(t+\nu -1\right)∥\le$$

$$∥\sum _{j=t+r+1}^{t+\nu -1}\left[\mathbf{\Psi }\left(j-1\right)-\mathbf{\Psi }\left(j\right)\right]{\tau }_1\left(\mathbf{Q}\left(j,t+\nu \right)\right)∥\le$$

$$\sum _{j=t+r+1}^{t+\nu -1}∥\mathit{\psi }\left(j-1\right)-\mathit{\psi }\left(j\right)∥{\tau }_1\left(\mathbf{Q}\left(j,t+\nu \right)\right)\le$$

$$\begin{array}{ccc}& \le & ({\tau }_1\left(\mathbf{Q}\left(t+r,t+\nu \right)\right)<1\mathrm{due}\mathrm{to}\mathrm{weak}\mathrm{ergodicity}\mathrm{of}{\left\{Y\left(t\right)\right\}}_{t=0}^{\infty };\hfill \\ & & \mathrm{see}\mathrm{also}\mathrm{Remark}2).\hfill \end{array}$$

$$\le \sum _{j=t+r+1}^{t+\nu -1}∥\mathit{\psi }\left(j-1\right)-\mathit{\psi }\left(j\right)∥<\infty ,$$

(31)

due to condition (17). Since $N_2\left(t,\nu \right)<\infty$ for every r and $\nu$ and it is a sum of positive numbers, we have that its tail goes to zero. Hence, we could, for every $\epsilon >0$, fix $t+r$ such that $N_2\left(t,\nu \right)<\epsilon /3$ for $\nu -1\ge r$ and we can always take a $\nu$ large enough so that $N_1\left(t,\nu \right)<\epsilon /3$ and

$$N_3\left(t,\nu \right)=∥\mathbf{\Psi }\left(t+\nu -1\right)-\mathbf{\Psi }∥\le \frac{\epsilon }{3},$$

(32)

therefore, we obtain that

$$\underset{\nu \to \infty }{lim}∥\mathbf{Q}\left(t,t+\nu \right)-\mathbf{\Psi }∥=0.$$

(33)

Now, from (4), we have that

$$∥\mathbb{E}\left[\mathbf{q}\left(t,t+\nu \right)\right]-\mathit{\psi }∥=$$

$$∥\mathbf{q}\left(t\right)\frac{T\left(t\right)}{T\left(t+\nu \right)}\mathbf{Q}\left(t,t+\nu \right)+\frac{1}{T\left(t+\nu \right)}\sum _{\tau =t}^{t+\nu }\Delta T\left(\tau \right)\mathbf{Q}\left(\tau ,t+\nu \right)-\mathit{\psi }∥\le$$

$$\frac{1}{T\left(t+\nu \right)}∥\mathbf{q}\left(t\right)T\left(t\right)\mathbf{Q}\left(t,t+\nu \right)+\sum _{\tau =t}^{t+\nu }\Delta T\left(\tau \right)\mathbf{Q}\left(\tau ,t+\nu \right)-T\left(t+\nu \right)\mathit{\psi }∥\le$$

$$∥\mathbf{q}\left(t\right)T\left(t\right)\mathbf{Q}\left(t,t+\nu \right)-T\left(t\right)\mathit{\psi }∥+$$

$$∥\sum _{\tau =t}^{t+\nu }\Delta T\left(\tau \right){\mathbf{p}}_0\left(\tau \right)\mathbf{Q}\left(\tau ,t+\nu \right)-\left[T\left(t+\nu \right)-T\left(t\right)\right]\mathit{\psi }∥.$$

(34)

Now, we have

$$\mathcal{A}\left(t,t+\nu \right)=∥\mathbf{q}\left(t\right)T\left(t\right)\mathbf{Q}\left(t,t+\nu \right)-T\left(t\right)\mathit{\psi }∥\le$$

$$T\left(t\right)∥\mathbf{q}\left(t\right)∥∥\mathbf{Q}\left(t,t+\nu \right)-\mathit{\psi }∥.$$

(35)

By fixing $\epsilon >0$, we can always find ${\nu }_0$ such that

$$\mathcal{A}\left(t,t+\nu \right)\le \frac{\epsilon }{2}\mathrm{for}\nu \ge {\nu }_0.$$

(36)

$$\mathcal{B}\left(t,t+\nu \right)=∥\sum _{\tau =t}^{t+\nu }\Delta T\left(\tau \right){\mathbf{p}}_0\left(\tau \right)\mathbf{Q}\left(\tau ,t+\nu \right)-\left[T\left(t+\nu \right)-T\left(t\right)\right]\mathit{\psi }∥=$$

$$=\left(\sin \mathrm{ce}{\mathbf{p}}_0\left(\tau \right)\mathbf{\Psi }={\mathbf{p}}_0\mathbf{\Psi }=\mathit{\psi }\right)=$$

$$∥\sum _{\tau =t}^{t+\nu }\Delta T\left(\tau \right){\mathbf{p}}_0\left(\tau \right)\mathbf{Q}\left(\tau ,t+\nu \right)-\sum _{\tau =t}^{t+\nu }\Delta T\left(\tau \right){\mathbf{p}}_0\mathbf{\Psi }∥=$$

$$∥\sum _{\tau =t}^{t+\nu }\Delta T\left(\tau \right)\left[{\mathbf{p}}_0\left(\tau \right)\mathbf{Q}\left(\tau ,t+\nu \right)-{\mathbf{p}}_0\mathbf{\Psi }\right]∥\le$$

$$\sum _{\tau =t}^{t+\nu }\Delta T\left(\tau \right)∥{\mathbf{p}}_0\left(\tau \right)\mathbf{Q}\left(\tau ,t+\nu \right)-{\mathbf{p}}_0\mathbf{\Psi }∥\le$$

$$\sum _{\tau =t}^{t+\nu }\Delta T\left(\tau \right)∥{\mathbf{p}}_0\left(\tau \right)\left[\mathbf{Q}\left(\tau ,t+\nu \right)-\mathbf{\Psi }\right]+{\mathbf{p}}_0\left(\tau \right)\mathbf{\Psi }-{\mathbf{p}}_0\mathbf{\Psi }∥$$

$$\le \sum _{\tau =t}^{t+\nu }\Delta T\left(\tau \right)∥\mathbf{Q}\left(\tau ,t+\nu \right)-\mathbf{\Psi }∥$$

(37)

Now, for $\nu >{\nu }_0$ from (37), we have that

$$\mathcal{B}\left(t,t+\nu \right)\le \sum _{\tau =t}^{t+\nu -{\nu }_0}\Delta T\left(\tau \right)∥\mathbf{Q}\left(\tau ,t+\nu \right)-\mathbf{\Psi }∥+\sum _{\tau =t+\nu -{\nu }_0}^{t+\nu }\Delta T\left(\tau \right)∥\mathbf{Q}\left(\tau ,t+\nu \right)-\mathbf{\Psi }∥$$

$$\le \sum _{\tau =t}^{t+\nu -{\nu }_0}\Delta T\left(\tau \right)\frac{\epsilon }{2}+2\left|T\left(t+\nu \right)-T\left(t+\nu -{\nu }_0+1\right)\right|,$$

(38)

where the second part of (38) for $\nu >>{\nu }_0$ is less than $\epsilon /2$ due to condition (14). The first part is also for, $\nu >>{\nu }_0$, less than $\epsilon /2$ since $\Delta T\left(t\right){\to }_{t\to \infty }0$ due to condition (14). From (34) and (36), and a $\nu >{\nu }_0$ large enough $\left(\nu >>{\nu }_0\right)$, we easily see that

$$∥\mathbb{E}\left[\mathbf{q}\left(t,t+\nu \right)\right]-\mathit{\psi }∥\le \epsilon \mathrm{for}\mathrm{every}\nu >>{\nu }_0,$$

hence, the NHMS is strongly ergodic. □

4. Rate of Convergence in NHMS with Chronological Order

An important question in nonhomogeneous Markov chains and NHMS, due to its very large practical value, is the rate of convergence to their asymptotic structure. In fact, it is important to find the necessary conditions under which the rate of convergence is geometric because then the value of the asymptotic result is greater. The roots of the study of finding conditions under which the rate of convergence is geometric for nonhomogeneous Markov chains are in Huang, et al. [34,35] and Seneta [31]. The study of the geometric rate of convergence in NHMS started in Vassiliou and Tsaklidis [36] and Georgiou and Vassiliou [26], and an updated evolution of these theorems and their variants could be found in Vassiliou [12]. The importance of answering this problem for practical purposes is apparent in Bartholomew [37] for the homogeneous Markov system, which is a very special case of an NHMS. In the present section, we provide and prove a basic theorem for the rate of convergence to strong ergodicity in NHMS when the transition probabilities matrices of the inherent Markov chain ${\left\{\mathbf{Q}\left(t\right)\right\}}_{t=0}^{\infty }$ are given in chronological order. In Theorem 4, we relax the basic assumption present in all studies of the rate of convergence to its asymptotic behavior for an NHMS. That is, we will not assume that the inherent inhomogeneous Markov chain converges as time goes to infinity to a homogeneous Markov chain with a regular transition probability matrix geometrically fast. We start with some basic definitions and results which will be useful in what follows.

Definition 6.

We say that a sequence of matrices ${\left\{{\mathbf{A}}_n\right\}}_{n=0}^{\infty }$ converges with geometrical rate to a matrix $\mathbf{A}$ if there exists constants $c>0$ and $0<b<1$ such that

$$∥{\mathbf{A}}_n-\mathbf{A}∥\le c{b}^n\mathit{for}n=1,2,\dots .$$

(39)

Lemma 3

(Vassiliou and Tsaklidis [36]). The following statements are equivalent:

(i)

The sequence ${\left\{\Delta T\left(t\right)\right\}}_{t=0}^{\infty }$ converges to zero geometrically fast.

(ii)

The sequence ${\left\{T\left(t\right)\right\}}_{t=0}^{\infty }$ converges to T geometrically fast.

Definition 7.

$\left(i\right)$ A stochastic matrix $\mathbf{P}$ is called Markov if at least one column of $\mathbf{P}$ is entirely positive. Let $\mathcal{M}$ be the set of all Markov matrices. $\left(ii\right)$ We say that the stochastic matrix $\mathbf{P}\in {\mathcal{G}}_2$ if $\left(a\right)$ $\mathbf{P}\in {\mathcal{G}}_1$ the set of all regular matrices; $\left(b\right)$ $\mathbf{QP}\in {\mathcal{G}}_1$ for any $\mathbf{Q}\in {\mathcal{G}}_1$. $\left(iii\right)$ We say that the stochastic matrix $\mathbf{P}\in {\mathcal{G}}_3$ the set of all scrambling matrices if ${\tau }_{{∥.∥}_1}\left(\mathbf{P}\right)<1$.

Remark 3.

The distinction of the set ${\mathcal{G}}_2$ from all stochastic regular matrices is due to the fact that the product of two regular matrices is not always regular. In addition, the product of two nonregular stochastic matrices could be regular. A practical way to check if a stochastic matrix of small dimension is scrambling is the following: given any two rows $i,j$, there is at least one column k such that $p_{ik}>0$ and $p_{jk}>0$.

Definition 8.

The incidence matrix of a stochastic matrix $\mathbf{P}$ is a matrix where in the positions of positive elements we put the number 1. Therefore, two stochastic matrices $\mathbf{P}$ and $\mathbf{Q}$ of the same dimension have the same incidence matrix if they have the positive elements in the same positions. Then, we write $\mathbf{P}\sim \mathbf{Q}$.

We now state some known Lemmas and Theorems, the proofs of which can be found in Vassiliou ([12], p. 143).

Theorem 2.

For all stochastic matrices, we have $\mathcal{M}\subset {\mathcal{G}}_3\subset {\mathcal{G}}_2\subset {\mathcal{G}}_1$.

Lemma 4.

If $\mathbf{P}\left(t,t+\nu \right)\in {\mathcal{G}}_1$ with $t\ge 0,$$n\ge 1$, then $\mathbf{P}\left(t,t+\nu \right)\in \mathcal{M}$ for $t+\nu \ge \mu$ the number of distinct incidence matrices corresponding to ${\mathcal{G}}_1$ with the same dimension as $\mathbf{P}\left(t,t+\nu \right)$.

Theorem 3.

Let there be a complete probability space $\left(\Omega ,\mathcal{F},\mathbb{P}\right)$, and consider a nonhomogeneous Markov chain ${\left\{X_t\right\}}_{t=0}^{\infty }$ in discrete time and space with a sequence of transition probabilities matrices ${\left\{\mathbf{P}\left(t\right)\right\}}_{t=0}^{\infty }$. If $\mathbf{P}\left(t\right)\in {\mathcal{G}}_2$ for every $t=1,2,\dots$ and

$$\underset{i,j}{min}{\left(p_{ij}\left(t\right),0\right)}^{+}\ge \gamma >0,$$

(40)

uniformly for all $t\ge 1$, then weak ergodicity obtains at a uniform geometric rate.

We now define the geometrically strongly ergodic NHMS, which is a central concept in the present section.

Definition 9.

Let there be a complete probability space $\left(\Omega ,\mathcal{F},\mathbb{P}\right)$, and consider an NHMS in discrete time and space. We say that the NHMS is strongly ergodic if there exists a stochastic vector ψ and constants $c>0$ and $0<b<1$ such that

$$∥\mathbb{E}\left[\mathbf{q}\left(t,t+\nu \right)-\mathit{\psi }\right]∥\le c{b}^{\nu }\mathit{for}c>0\mathit{and}0<b<1.$$

We now state and prove the basic theorem of this section.

Theorem 4.

Let there be a complete probability space $\left(\Omega ,\mathcal{F},\mathbb{P}\right)$, and consider an NHMS in discrete time and space. We assume that the total number of memberships is increasing ($\Delta T\left(t\right)\ge 0$), and it converges geometrically fast to T. That is,

$$\underset{t\to \infty }{lim}T\left(t\right)=T\mathit{geometrically}\mathit{fast}\mathit{with}\Delta T\left(t\right)\ge 0.$$

(41)

Let ${\left\{Y\left(t\right)\right\}}_{t=0}^{\infty }$ be the inherent nonhomogeneous Markov chain of the movement of memberships. If ${\left\{Y\left(t\right)\right\}}_{t=0}^{\infty }$ is weakly ergodic with ${\left\{\mathbf{Q}\left(t\right)\right\}}_{t=0}^{\infty }$ the sequence of transition probabilities to be such that

$$\left(i\right)\mathbf{Q}\left(t\right)\in {\mathcal{G}}_2\mathit{for}\mathit{every}t=1,2,\dots ,$$

(42)

$$\left(ii\right)\underset{i,j}{min}{\left(q_{ij}\left(t\right),0\right)}^{+}\ge \gamma >0,$$

(43)

and if in addition there exists a stochastic vector

$$\mathit{\psi }\left(t\right)=\left[{\psi }_1\left(t\right),{\psi }_2\left(t\right),\dots ,{\psi }_k\left(t\right)\right],$$

(44)

which is the left eigenvector for $t=1,2,\dots$ of $\mathbf{Q}\left(t\right)$, that is,

$$\mathit{\psi }\left(t\right)=\mathit{\psi }\left(t\right)\mathbf{Q}\left(t\right)\mathit{for}\mathit{every}t,$$

(45)

and in addition

$$\underset{\nu \to \infty }{lim}∥\mathit{\psi }\left(t+\nu \right)-\mathit{\psi }\left(t+\nu -1\right)∥=0\mathit{geometrically}\mathit{fast},$$

(46)

then the NHMS is geometrically strongly ergodic.

Proof. 

In order to prove that the NHMS is geometrically strongly ergodic, we must show that the expected relative structure satisfies Definition 9.

From (46), we have that there exists a vector $\mathit{\psi }$ and constants $c_1>0$ and $0<b_1<1$ such that

$$∥\mathit{\psi }\left(t+\nu \right)-\mathit{\psi }∥\le c_1{b}_1^{t+\nu }.$$

(47)

We define $\mathbf{\Psi }\left(t\right)$ and $\mathbf{\Psi }$ as the stable matrices with rows $\mathit{\psi }\left(t\right)$ and $\mathit{\psi }$, respectively.

We now show that the inherent nonhomogeneous Markov chain ${\left\{Y\left(t\right)\right\}}_{t=0}^{\infty }$ is geometrically strongly ergodic. That is, we need to show that there exists constants $c_2>0$ and $0<b_2<1$ such that

$$∥\mathbf{Q}\left(t,t+\nu \right)-\mathbf{\Psi }∥\le c_2{b}_2^{\nu }\mathrm{for}\mathrm{every}t.$$

(48)

Let us denote by $\mu$ the number of distinct incidence matrices corresponding to ${\mathcal{G}}_1$ with the same dimension as $\mathbf{P}\left(t,t+\nu \right)$. Then, for $\nu >\mu$, we have that

$$∥\mathbf{Q}\left(t,t+\nu \right)-\mathbf{\Psi }∥\le ∥\mathbf{Q}\left(t,t+\mu \right)\mathbf{Q}\left(t+\mu ,t+\nu \right)-\mathbf{\Psi }\left(t+\mu \right)\mathbf{Q}\left(t+\mu ,t+\nu \right)∥$$

$$+∥\mathbf{\Psi }\left(t+\mu \right)\mathbf{Q}\left(t+\mu ,t+\nu \right)-\mathbf{\Psi }\left(t+\nu -1\right)∥+∥\mathbf{\Psi }\left(t+\nu -1\right)-\mathbf{\Psi }∥.$$

(49)

Now, we have that

$${\mathcal{D}}_1\left(t,\nu ,\mu \right)=∥\mathbf{Q}\left(t,t+\mu \right)\mathbf{Q}\left(t+\mu ,t+\nu \right)-\mathbf{\Psi }\left(t+\mu \right)\mathbf{Q}\left(t+\mu ,t+\nu \right)∥$$

$$\le ∥\left[\mathbf{Q}\left(t,t+\mu \right)-\mathbf{\Psi }\left(t+\mu \right)\right]\mathbf{Q}\left(t+\mu ,t+\nu \right)∥$$

$$\le ∥\mathbf{Q}\left(t,t+\mu \right)-\mathbf{\Psi }\left(t+\mu \right)∥{\tau }_1\left(\mathbf{Q}\left(t+\mu ,t+\nu \right)\right)$$

$$\le 2{\tau }_1\left(\mathbf{Q}\left(t+\mu ,t+\nu \right)\right).$$

(50)

For arbitrary but fixed t and $\nu$ with $m\mu \le \nu$ with m, the largest such integer from (50), we have that

$${\mathcal{D}}_1\left(t,\nu ,\mu \right)\le 2{\tau }_1\left(\mathbf{Q}\left(t+\mu ,t+2\mu \right)\right){\tau }_1\left(\mathbf{Q}\left(t+\mu ,t+2\mu \right)\right)\dots$$

$${\tau }_1\left(\mathbf{Q}\left(t+\left(m-1\right)\mu ,t+m\mu \right)\right){\tau }_1\left(\mathbf{Q}\left(t+m\mu ,t+\nu \right)\right).$$

(51)

Since $\mathbf{Q}\left(t\right)\in {\mathcal{G}}_2$ and by Lemma 4 we have that ${\tau }_1\left(\mathbf{Q}\left(t+\left(i-1\right)\mu ,t+i\mu \right)\right)\in \mathcal{M}$ and

$${\tau }_1\left(\mathbf{Q}\left(t+\left(i-1\right)\mu ,t+i\mu \right)\right)\le 1-{\gamma }^{\mu }\mathrm{with}0<\gamma <1\mathrm{for}i=2,3,\dots ,m.$$

(52)

From the weak ergodicity of the inherent nonhomogeneous Markov chain ${\left\{Y\left(t\right)\right\}}_{t=0}^{\infty }$, we have that

$${\tau }_1\left(\mathbf{Q}\left(t+m\mu ,t+\nu \right)\right)\le 1.$$

(53)

From (51)–(53), we arrive at

$${\mathcal{D}}_1\left(t,\nu ,\mu \right)\le {\left(1-{\gamma }^{\mu }\right)}^m,$$

(54)

which as $\nu \to \infty$ goes to zero at a uniform geometric rate.

Following the steps of arriving at relation (31), we straightforwardly obtain that

$${\mathcal{D}}_2\left(t,\nu ,\mu \right)=∥\mathbf{\Psi }\left(t+\mu \right)\mathbf{Q}\left(t+\mu ,t+\nu \right)-\mathbf{\Psi }\left(t+\nu -1\right)∥$$

$$\le ∥\mathbf{\Psi }\left(t+\mu \right)\mathbf{Q}\left(t+\mu ,t+\nu \right)-\mathbf{\Psi }\mathbf{\Psi }\left(t+\nu -1\right)∥$$

$$\le \sum _{j=t+\mu +1}^{t+\nu -1}∥\mathit{\psi }\left(j-1\right)-\mathit{\psi }\left(j\right)∥{\tau }_1\left(\mathbf{Q}\left(j,t+\nu \right)\right).$$

(55)

We now have the largest integer for m, such that $m\mu \le \nu$

$${\tau }_1\left(\mathbf{Q}\left(t+\mu ,t+\nu \right)\right)={\tau }_1(\mathbf{Q}\left(t+\mu ,t+2\mu \right)\mathbf{Q}\left(t+2\mu ,t+3\mu \right)\dots$$

$$\mathbf{Q}\left(t+\left(m-1\right)\mu ,t+m\mu \right)\mathbf{Q}\left(t+m\mu ,t+\nu \mu \right))$$

$$=\left(\mathrm{by}\mathrm{Lemma}1\right)$$

$$\le {\tau }_1\left(\mathbf{Q}\left(t+\mu ,t+2\mu \right)\right){\tau }_1\left(\mathbf{Q}\left(t+2\mu ,t+3\mu \right)\right)\dots$$

$${\tau }_1\left(t+\left(m-1\right)\mu ,t+m\mu \right){\tau }_1\left(t+m\mu ,t+\nu \right)\le$$

$$\le \left(\mathrm{using}\left(52\right)\right)$$

$$\le \left(1-{\gamma }^{\mu }\right)\left(1-{\gamma }^{\mu }\right)\dots \left(1-{\gamma }^{\mu }\right)={\left(1-{\gamma }^{\mu }\right)}^m.$$

(56)

From (56), we have that

$${\tau }_1\left(\mathbf{Q}\left(t+\mu ,t+\nu \right)\right){\to }_{\nu \to \infty }0\mathrm{uniformly}\mathrm{geometrically}\mathrm{fast}.$$

(57)

From (55), we have that

$${\mathcal{D}}_2\left(t,\nu ,\mu \right)\le ∥\mathit{\psi }\left(t+\mu -1\right)-\mathit{\psi }\left(t+\mu \right)∥{\left(1-{\gamma }^{\mu }\right)}^m+$$

$$∥\mathit{\psi }\left(t+\mu \right)-\mathit{\psi }\left(t+\mu +1\right)∥{\left(1-{\gamma }^{\mu }\right)}^{m-1}+∥\mathit{\psi }\left(t+\mu \right)-\mathit{\psi }\left(t+\mu +1\right)∥{\left(1-{\gamma }^{\mu }\right)}^{m-1}$$

$$+\dots +∥\mathit{\psi }\left(t+\mu \left(m-1\right)-1\right)-\mathit{\psi }\left(t+\mu \left(m-1\right)\right)∥\left(1-{\gamma }^{\mu }\right)$$

$$+∥\mathit{\psi }\left(t+\mu m-1\right)-\mathit{\psi }\left(t+\mu m\right)∥.$$

(58)

Now, we have that

$${\left(1-{\gamma }^{\mu }\right)}^m{\to }_{m\to \infty }0\mathrm{geometrically}\mathrm{fast},$$

since $\left(1-{\gamma }^{\mu }\right)<1$. Also, from condition (46), we have that

$$∥\mathit{\psi }\left(t+\mu m-1\right)-\mathit{\psi }\left(t+\mu m\right)∥{\to }_{m\to \infty }0\mathrm{geometrically}\mathrm{fast},$$

Therefore,

$${\mathcal{D}}_2\left(t,\nu ,\mu \right){\to }_{\nu \to \infty }0\mathrm{geometrically}\mathrm{fast}.$$

(59)

Now, from (49), (54) and (59), and condition (46), we easily have that

$$∥\mathbf{Q}\left(t,t+\nu \right)-\mathbf{\Psi }∥{\to }_{\nu \to \infty }0\mathrm{geometrically}\mathrm{fast};$$

(60)

hence, there exist constants $c>0$ and $0<b<1$ such that

$$∥\mathbf{Q}\left(t,t+\nu \right)-\mathbf{\Psi }∥\le c{b}^{\nu }\mathrm{for}\mathrm{every}t,\mathrm{and}c>0,0<b<1.$$

(61)

What remains, according to Definition 9, is to show that the expected relative population structure $\mathbb{E}\left[\mathbf{q}\left(t,t+\nu \right)\right]$ converges for every t to the vector $\mathit{\psi }$ as $\nu$ goes to infinity geometrically fast.

From (35), we have that

$$\mathcal{A}\left(t,t+\nu \right)=∥\mathbf{q}\left(t\right)T\left(t\right)\mathbf{Q}\left(t,t+\nu \right)-T\left(t\right)\mathit{\psi }∥\le$$

$$T\left(t\right)∥\mathbf{q}\left(t\right)∥∥\mathbf{Q}\left(t,t+\nu \right)-\mathit{\psi }∥.$$

(62)

From (61), we have that

$$\mathcal{A}\left(t,t+\nu \right)\le T\left(t\right)∥\mathbf{q}\left(t\right)∥∥\mathbf{Q}\left(t,t+\nu \right)-\mathit{\psi }∥\le c_1{b}^{\nu },$$

(63)

with $c_1>0$ and $0<b<1.$

From Lemma 3, we have that since $T\left(t\right)$ converges geometrically fast to T, then $\Delta T\left(t\right)$ converges geometrically fast to zero. Hence,

$$\Delta T\left(t\right)\le c_2{b}_2^t\mathrm{with}{c}_2>0\mathrm{and}0<b_2<1.$$

(64)

Now, from (37), we have that

$$\mathcal{B}\left(t,t+\nu \right)\le \sum _{\tau =t}^{t+\nu }\Delta T\left(\tau \right)∥\mathbf{Q}\left(\tau ,t+\nu \right)-\mathbf{\Psi }∥.$$

(65)

With no loss of generality, we may assume that $b>b_2$, and then, from (63) and (64), we have that

$$\mathcal{B}\left(t,t+\nu \right)\le c_1{c}_2\sum _{\tau =t}^{t+\nu }{\left(\frac{b_2}{b}\right)}^{\tau }{b}^{t+\nu }=c_3{b}^{t+\nu }\sum _{\tau =t}^{t+\nu }{\left(\frac{b_2}{b}\right)}^{\tau }$$

$$\le c_3{b}^{t+\nu }{\left(1-\frac{b_2}{b}\right)}^{\nu }{\left(1-\frac{b_2}{b}\right)}^{-1}\le c_4{b}_3^{\nu }.$$

(66)

From (63) and (66), we easily arrive at the proof of the theorem. □

5. An Illustrative Application

We will illustrate the results in the previous section with an example from a population of manpower. To possibly better visualize, the reader may have in mind a University system with three grades. That is, grade one is those with the level of Professors, grade two belongs to the Associate Professors, and finally, in grade three, there are the Assistant Professors. The University has a plan for funding $T\left(t\right)$ for $t=0,1,2,\dots$ memberships for the next few years. When a member of staff is leaving, their membership remains with the University, that is, the funding of their position is not lost but remains, and the University could go on to appoint someone at any grade. The external environment to which the leavers go and their membership is retained by the University and it is the population of members of academic staff from Universities almost all over the world, as practices have shown. Hence, it is from this external environment that the new members will obtain the memberships available from the organization of the University.

A fundamental question for the practitioner is the estimation of the transition probabilities from the historical records. The way to estimate the transition probabilities of the memberships is a small extension of the way it is performed in manpower planning, which is well documented in Bartholomew [38]. Another quite similar problem is that of the competing risk model in the medical literature, as it was presented by Kalbfleisch and Prentice ([39], Chapter 8), Lee [40], and Cox and Oakes [41]. Also, similar problems exist in the study of reliability models and various actuarial studies as discussed in Elandt–Johnson and Johnson ([42], Chapter 7).

Now define by

$N_{ij}\left(t\right)$: the number of memberships moving from grade i to grade j in the t-th interval.

$N_i\left(t-1\right)$: the number of memberships in state i at the beginning of the t-th interval.

Assume that the number of years available in the historical data of the University is n. Then, the maximum likelihood estimate of the probability $q_{ij}\left(t\right)$ is the following:

$${\widehat{q}}_{ij}\left(t\right)=\frac{N_{ij}\left(t\right)}{N_i\left(t-1\right)}\mathrm{for}\mathrm{any}t.$$

(67)

It is an apparent advantage that the probabilities ${\widehat{q}}_{ij}\left(t\right)$ are separately estimated for every $\left(i,j\right)$. In this way, the number of years of historical records necessary are significantly reduced. At this point, it is useful to test the hypothesis that the probabilities ${\widehat{q}}_{ij}\left(t\right)$ are indeed functions of time. That is,

$$H_0:{\widehat{q}}_{ij}\left(t\right)={\widehat{q}}_{ij}\mathrm{for}\mathrm{every}t.$$

(68)

Considering the flow of memberships which move from grade i to grade j as a multinomial random variable, then (see Andersen and Goodman [43]) hypothesis (68) is tested by the statistic:

$${\chi }^2\left(i,j\right)=\sum _{t=1}^n{N}_i\left(t-1\right)\frac{\left({\widehat{q}}_{ij}\left(t\right)-{\widehat{q}}_{ij}\right)}{{\widehat{q}}_{ij}},$$

(69)

where

$${\widehat{q}}_{ij}=\frac{{\sum }_{t=0}^n{N}_{ij}\left(t\right)}{{\sum }_{t=1}^n{N}_i\left(t-1\right)},$$

(70)

is the maximum likelihood estimate under the null hypothesis and is chi-square distributed with $n-1$ degrees of freedom.

Now, let that the ${\chi }^2\left(i,j\right)$ showed that the probabilities ${\widehat{q}}_{ij}\left(t\right)$ are functions of time. Then, there is a need to predict their values as functions of time. For a specific pair $\left(i,j\right)$, let that

$$x_{1ij}\left(t\right),x_{2ij}\left(t\right),\dots ,x_{mij}\left(t\right),$$

(71)

are probable covariates for the specific application. Then, logistic stepwise regression is an appropriate model for these probabilities. Let us define by

$$logit\left(x\right)=log\left(\frac{x}{1-x}\right),$$

(72)

then we obtain

$$logit\left({\widehat{q}}_{ij}\left(t\right)\right)=a_0+a_1{x}_{1ij}\left(t\right)+a_2{x}_{2ij}\left(t\right)+\dots +a_m{x}_{mij}\left(t\right).$$

(73)

Now, it is obvious that stepwise regression will show what are the important covariates to predict ${\widehat{q}}_{ij}\left(t\right)$ (see also Vassiliou [44]).

Let that following the above-described steps, we ended with the matrix:

$$\mathbf{Q}\left(t\right)=\left(\begin{array}{ccc}0.2+\frac{1}{4+t^3}& 0.8-\frac{1}{4+t^3}& 0\\ 0.3& 0.5-\frac{1}{8+t^4}& 0.2+\frac{1}{8+t^4}\\ 0& 0.2+\frac{2}{10+t^8}& 0.8-\frac{2}{10+t^8}\end{array}\right)\mathrm{for}\mathrm{t}=1,2,\dots .$$

(74)

The total population of memberships was planned according to the following sequence

$$T\left(0\right)=400,T\left(1\right)=430,T\left(2\right)=450,$$

$$T\left(3\right)=475,T\left(4\right)=500\mathrm{and}T\left(t\right)=500\mathrm{for}t=5,6,\dots ,$$

(75)

that is, the total number of memberships converge geometrically fast with $\Delta T\left(t\right)\ge 0$, satisfying the condition (41) in Theorem 4.

It is not difficult to check that the sequence (74) satisfies condition (42), that is, $\mathbf{Q}\left(t\right)\in {\mathcal{G}}_2$ for $t=1,2,\dots$. Also, it satisfies condition (43) since

$$\underset{i,j}{min}{\left(q_{ij}\left(t\right),0\right)}^{+}\ge \gamma =0.2>0.$$

(76)

For the sequence of transition probability matrices for the memberships (74), we find for the condition (45) that the vectors $\mathit{\psi }\left(t\right)$ that satisfy it were the following:

$$\mathrm{For}t=1\mathit{\psi }\left(1\right)=\left[0.216,0.432,0.352\right],$$

(77)

$$\mathrm{for}t=2\mathit{\psi }\left(2\right)=\left[0.166,0.390,0.444\right],$$

(78)

$$\mathrm{for}t=3\mathit{\psi }\left(3\right)=\left[0.164,0.416,0.420\right],$$

(79)

$$\mathrm{for}t=4\mathit{\psi }\left(4\right)=\left[0.164,0.424,0.412\right],$$

(80)

$$\mathrm{for}t=5\mathit{\psi }\left(5\right)=\left[0.163,0.426,0.411\right],$$

(81)

$$\mathrm{for}t=6\mathit{\psi }\left(6\right)=\left[0.163,0.427,0.410\right],$$

(82)

$$\mathrm{for}t=7\mathit{\psi }\left(7\right)=\left[0.163,0.427,0.410\right].$$

(83)

We observe that already for t = 6 in (82) and (83) we have convergence of $\mathit{\psi }\left(t\right)$ which satisfies condition (45) of Theorem 4, that is the convergence is geometrically fast. Hence, we conclude that

$$\mathit{\psi }=\left[0.16,0.43,0.41\right].$$

Calculating using the transition probability matrices given in (74) the matrix product $\mathbf{Q}\left(t,t+\nu \right)$ we find that for $\nu \ge 9$ it converges to $\mathit{\psi }=\left[0.16,0.43,0.41\right]$. Hence, we conclude that $\mathbf{Q}\left(t,t+\nu \right)$ as $\nu \to \infty$ converges geometrically fast as was expected from Theorem 4 relation (48).

Now, given the convergence of $\mathbf{Q}\left(t,t+\nu \right)$ as $\nu \to \infty$ it is straight forward to find from Equation (2) applying it recursively or equivalently from Theorem 1 that $\mathbb{E}\left[\mathbf{q}\left(s,t\right)\right]$ converges geometrically fast (in fact in 9 time steps) to

$$\mathit{\psi }=\left[0.16,0.43,0.41\right].$$

Hence, the NHMS is geometrically strongly ergodic.

6. Conclusions and Further Research

Two fundamental theorems have been founded that relax previous assumptions and provide conditions for ergodicity and for the convergence rate of the relative population structure in an NHMS. More specifically, in Theorem 1 the strong ergodicity of an NHMS is studied without assuming the convergence of the inherent inhomogeneous Markov chain to infinity to a homogeneous Markov chain with a regular transition probability matrix. In Theorem 4, it is proved under what conditions the rate of convergence of strong ergodicity in a NHMS is geometrically fast. This is done by departing from the basic assumption in which the inherent inhomogeneous Markov chain converges after a large time to a homogeneous Markov chain with a regular transition probability matrix. The proved theorems are expanding the understanding of the dynamics and behavior of an NHMS. The paper concludes with an illustrative application from the field of Manpower Planning, showcasing the vital practical relevance of the discussed concepts. Further research paths may include the relaxation of relative assumptions in the many variant models of the NHMS in diverse populations. Of particular interest may be the theorems of Laws of Large numbers in an NHMS.