A Fast-Pivoting Algorithm for Whittle’s Restless Bandit Index

Abstract

The Whittle index for restless bandits (two-action semi-Markov decision processes) provides an intuitively appealing optimal policy for controlling a single generic project that can be active (engaged) or passive (rested) at each decision epoch, and which can change state while passive. It further provides a practical heuristic priority-index policy for the computationally intractable multi-armed restless bandit problem, which has been widely applied over the last three decades in multifarious settings, yet mostly restricted to project models with a one-dimensional state. This is due in part to the difficulty of establishing indexability (existence of the index) and of computing the index for projects with large state spaces. This paper draws on the author’s prior results on sufficient indexability conditions and an adaptive-greedy algorithmic scheme for restless bandits to obtain a new fast-pivoting algorithm that computes the n Whittle index values of an n-state restless bandit by performing, after an initialization stage, n steps that entail $(2/3)n^3+O\left(n^2\right)$ arithmetic operations. This algorithm also draws on the parametric simplex method, and is based on elucidating the pattern of parametric simplex tableaux, which allows to exploit special structure to substantially simplify and reduce the complexity of simplex pivoting steps. A numerical study demonstrates substantial runtime speed-ups versus alternative algorithms.

1. Introduction

We consider a general two-action (1: engaged/active; 0: rested/passive) semi-Markov decision process (SMDP) restless bandit model (see, for example, ([1], Ch. 11) and [2]) of a dynamic and stochastic project, whose state $X\left(t\right)$ moves over continuous time $t\in [0,\infty )$ across the state space $\mathcal{N}$, which is assumed finite consisting of $n\triangleq \left|\mathcal{N}\right|$ states. At each of an increasing sequence of decision epochs ${\left\{t_k\right\}}_{k=0}^{\infty }$ starting at $t_0=0$ and with $t_k\nearrow \infty$ as $k\nearrow \infty$, the embedded state $X_k\triangleq X\left(t_k\right)$ is observed and then an action $A_k\triangleq A\left(t_k\right)\in \{0,1\}$ is chosen, which remains fixed over the subsequent stage$[t_k,t_{k+1})$, so $A\left(t\right)=A\left(t_k\right)$ for $t\in [t_k,t_{k+1})$. Note that the project state $X\left(t\right)$ can change within such stages, and that processes $X\left(t\right)$ and $A\left(t\right)$ are piecewise constant and right-continuous.

When the project is in state $X\left(t\right)=i$ under action $A\left(t\right)=a$, it accrues rewards and consumes a generic resource at rates $R_i^a$ and $Q_i^a$ per unit time, respectively, which are exponentially discounted with rate $\alpha >0$. Resource consumption rates are assumed to satisfy the natural requirement that $0<Q_i^1⩾Q_i^0⩾0$, so when the project is active, it consumes resources at a positive rate, which is not lower than when it is passive.

To select actions, a policy $\pi$ is adopted from the class $\Pi$ of history-dependent randomized policies, which base the choice of action at each decision epoch $t_k$ on the history ${\mathcal{H}}_k\triangleq \{(X_{k^{\prime }},A_{k^{\prime }}):0⩽k^{\prime }<k\}\cup \left\{X_k\right\}$ of embedded states and actions.

Suppose that the amount of resource consumed by the project must be paid for at the unit price $\lambda$, so, writing as ${\mathbb{E}}_i^{\pi }[·]$ the expectation starting from $X\left(0\right)=i$ under $\pi$,

$$V_i^{\pi }\left(\lambda \right)\triangleq {\mathbb{E}}_i^{\pi }\left[{\int }_0^{\infty }\left(R_{X\left(t\right)}^{A\left(t\right)}-\lambda Q_{X\left(t\right)}^{A\left(t\right)}\right)e^{-\alpha t}dt\right]$$

is the (expected total discounted) project value starting from i under $\pi$, and $V_i^{*}\left(\lambda \right)\triangleq {sup}_{\pi \in \Pi }{V}_i^{\pi }\left(\lambda \right)$ is the optimal project value starting from state i. Consider now the λ-price problem

$$\left(P_{\lambda }\right):\mathrm{find}{\pi }_{\lambda }^{*}\in \Pi \mathrm{such} \mathrm{that}{V}_i^{{\pi }_{\lambda }^{*}}\left(\lambda \right)=V_i^{*}\left(\lambda \right)\mathrm{for} \mathrm{every} \mathrm{state}i\in \mathcal{N},$$

(1)

which entails finding a policy that maximizes the project value for every initial state. We will call $P_{\lambda }$-optimal a policy ${\pi }_{\lambda }^{*}$ solving the $\lambda$-price problem $\left(P_{\lambda }\right)$ in (1).

Under mild assumptions, it is well-known from the theory of finite-state and -action SMDPs (see ([1], Ch. 11)) that, to solve problem (1), it suffices to consider the class ${\Pi }^{SD}$ of stationary deterministic policies, which prescribe a fixed action at each decision epoch based on the current project state. For any given resource price $\lambda$, a $P_{\lambda }$-optimal policy in ${\Pi }^{SD}$ can be computed by classic algorithms, such as value or policy iteration.

Instead, we will treat the resource price $\lambda$ as a parameter, and consider a solution approach for the parametric collection $\mathcal{P}$ of all$\lambda$-price problems $\left(P_{\lambda }\right)$ as $\lambda \in \mathbb{R}$. Thus, let us say that the project, or more precisely, the parametric problem collection $\mathcal{P}$, is indexable if, for every project state i, there exists a finite critical price ${\lambda }_i^{*}$ such that, for any problem $\left(P_{\lambda }\right)$ and at a decision epoch in which the project is in state i: (i) it is optimal to engage the project if, and only if, ${\lambda }_i^{*}⩾\lambda$; and (ii) it is optimal to rest the project if, and only if, ${\lambda }_i^{*}⩽\lambda$. Hence, both actions will be optimal in state i if, and only if, ${\lambda }_i^{*}=\lambda$.

We will refer to the mapping $i\mapsto {\lambda }_i^{*}$ as the project’s Whittle index, since it was Whittle who introduced such a concept in [2], in a Markovian setting with resource consumption $Q_i^a\triangleq a$. As for the extension to general resource consumption $Q_i^a$, it was introduced in [3]. In the case, $Q_i^a\triangleq a$, the Whittle index extends the Gittins index for classic (non-restless) bandit projects, which do not change state while passive. Thus, strictly speaking, the index considered herein for general $Q_i^a$ is a generalized Whittle index, which reduces to a generalized Gittins index in the non-restless case.

Considering a semi-Markov instead of a purely Markovian setting significantly expands the modeling power of the resulting restless bandits, as in some applications, the time during which a project remains engaged before the next decision can be made may follow a general distribution. As an example (see [4]), imagine that a “project” represents a queue of jobs with Poisson arrivals and generally distributed service times, where serving a job (active action) is non-preemptive, i.e., it cannot be interrupted once started. In [5], semi-Markov restless bandits are used to model dynamic job assignment for energy-efficient server farms. Semi-Markov restless bandits can also be used to model classic (non-restless) bandits where changing from engaging one project to another entails project-dependent switching times, as shown in [6].

Besides its intrinsic interest for solving the aforementioned single-project parametric problem collection $\mathcal{P}$, Whittle proposed in [2] to use the index ${\lambda }_i^{*}$ as the basis of a widely popular heuristic for the multi-armed restless bandit problem (MARBP), in which M out of $N>M$ restless bandit projects must be selected to be engaged at each time to maximize the value (under a discounted or long-run average criterion) earned from the N projects over an infinite horizon. For a sample of recent applications, see, for example, [5,7,8,9,10,11,12,13,14,15,16,17,18]. While the MARBP is computationally intractable (PSPACE-hard; see [19]), the Whittle index policy is an intuitively appealing heuristic where, at each time, M projects with the highest current indices are engaged, so the Whittle index plays the role of a priority index for a project to be engaged. This policy is convenient for practical implementation, as it avoids the curse of dimensionality, since each project has its own Whittle index. Furthermore, an extensive body of numerical evidence accumulated over the last three decades has shown that the policy is often nearly optimal. Though its exact analysis remains elusive, Whittle’s conjecture that his proposed policy enjoys a form of asymptotic optimality has been established under certain conditions. See [20,21,22,23].

Yet, unlike the Gittins index, which is well-defined for any classic bandit, the Whittle index exists only for a limited type of restless bandits, which are called indexable. Typically, researchers use ad hoc analyses to prove indexability and calculate the Whittle index in particularly models. In contrast, the author has introduced and developed in [3,24] a methodology to establish indexability and compute the Whittle index for general finite-state restless bandits, extended to the semi-Markov denumerable-state case in [4] and to the continuous-state case in [25]. The effectiveness of such an approach, based on verification of so-called PCL-indexability conditions—as they are grounded on satisfaction by project performance metrics of partial conservation laws (PCLs)—has been demonstrated in diverse models. See, for example, [14,26,27,28,29,30,31,32,33,34,35].

In the case of finite-state restless bandits, [3,24] introduced the concept of satisfaction of PCLs by project performance metrics, along with a related one-pass adaptive-greedy index algorithm, which calculates the n index values of an n-state PCL-indexable project in n steps. This has its early roots in Klimov’s algorithm in [36] for calculating the optimal priority indices for scheduling a multiclass queue with Bernoulli feedback, which was extended in [37] to a framework of stochastic scheduling systems satisfying so-called generalized conservation laws, such as the classic (non-restless) multi-armed bandit problem and branching bandits. See also [38,39]. Yet, the aforementioned work has not addressed the efficient computational implementation of such an algorithm, which is necessary for its actual deployment and widespread application.

This paper develops an efficient computational scheme for calculating the Whittle index of a general finite-state PCL-indexable restless bandit, by extending the approach in [40] from discrete-time classic bandits to semi-Markov restless bandits. The main contribution is a new fast-pivoting block implementation of the adaptive-greedy Whittle index algorithm in [3,24] that performs—after an initialization stage that entails solving a block $n\times n$ linear equation system—$(2/3)n^3+O\left(n^2\right)$ arithmetic operations. The complexity of the resulting algorithm can be further reduced in particular models, by exploiting the special structure of the underlying state space and transition matrices. However, it appears unlikely that such an operation count can be further reduced for general restless bandits, since as shown in [3,24] computing the Whittle index, even if the ordering of the project states was known in advance, entails the solution of an $n\times n$ linear equation system, which would be solved in $(2/3)n^3+O\left(n^2\right)$ operations by Gaussian elimination.

Note that this is the fastest operation count for a general Whittle index algorithm presented to date. In contrast, the Whittle index algorithm in [41], which applies only to the average criterion, has a complexity of $O\left(n^4{2}^n\right)$ operations, which reduces to $O\left(n^5\right)$ for projects with a one-dimensional state that are known beforehand to be both indexable and solved optimally by threshold policies.

Structure of the Paper

The rest of the paper proceeds as follows. Section 2 presents a review of the related literature. Section 3 reviews previous results on Whittle indexation for finite-state restless bandits via PCL-indexability. Section 4 lays the groundwork for an efficient implementation of the adaptive-greedy index algorithm, drawing on dynamic programming (DP) and linear programming (LP) methods. Section 5 applies such results to develop a fast-pivoting computational implementation of the adaptive-greedy index algorithm. Section 6 outlines how to extend the previous results to the average criterion. Section 7 presents the results of a numerical study testing the runtimes of several index algorithms and demonstrating that the proposed fast-pivoting algorithm has significantly lower runtimes than alternative algorithms. Section 8 discusses the results presented in the paper. Section 9 concludes the paper.

2. Review of Related Literature

In this section, we review related literature, focusing on relatively recent work. We refer the reader to the recent monograph [42] on multi-armed bandits, which discusses both the MABP and the MARBP and their widespread applications.

When projects are classic (non-restless), the Whittle index reduces to the Gittins index. The efficient computation of the Gittins index has been extensively investigated in the literature. See, for example, [43,44,45,46,47,48]. While some algorithms proposed in the aforementioned papers perform $O\left(n^3\right)$ arithmetic operations for a general n-state bandit, [40] presents a fast-pivoting implementation of the adaptive-greedy Gittins index algorithm in [37] that performs $(2/3)n^3+O\left(n^2\right)$ arithmetic operations, thus achieving lower complexity than alternative algorithms and matching that of Gaussian elimination for solving an $n\times n$ linear equation system. It is unlikely that such a complexity count can be improved, since it is shown in [37] that computing the Gittins index reduces precisely to solving an $n\times n$ linear equation system whose solution is subject to certain inequalities. The algorithm in [40] is based on an elucidation of the pattern of parametric simplex tableaux and exploitation of special structure to lower the computational effort of simplex pivoting steps with respect to standard pivoting.

In contrast, efficient computation of the Whittle index for restless bandits has received relatively scant research attention. Three approaches can be distinguished in the literature: deriving the Whittle index in closed form, iterative index approximation, and exact numerical computation. The first approach is to derive the Whittle index in closed form, as, for example,  in [5,13,15,16,17,18]. This is to be preferred whenever possible, as the resulting analytical expressions for the Whittle index, besides facilitating its numerical evaluation, may provide valuable insight on the index dependence on model parameters. However, obtaining closed-form Whittle index formulae is only possible in relative simple models, typically with a one-dimensional state.

In more complex models in which the Whittle index cannot be evaluated in closed form, the most widespread approach, which has its roots in the calibration method for the Gittins index in [49], is to apply an iterative procedure for approximately computing the index. This is done, for example, in [7,8,9,10,11,12]. Besides the drawback that the resulting index is only an approximation to the true Whittle index, this approach is typically computationally expensive.

As for the third approach, efficient exact numerical computation of the Whittle index, this has received the least research attention to date. In [3,24], the author introduced an adaptive-greedy algorithm for computing the Whittle index in general finite-state restless bandits, provided that they satisfy the PCL-indexability conditions also introduced in such work. The algorithmic computes the n Whittle index values of an n-state bandit in n steps, proceeding in a greedy fashion at each step. However, as given in [3,24], such a method provides only an algorithmic scheme, as it is not specified how certain metrics that arise in its description are to be computed in practice. The effectiveness of such an approach is demonstrated in [3] in the setting of a broad birth–death restless bandit model, both under the discounted and long-run average criteria, motivated by queueing admission control and routing problems, where a specific implementation of the adaptive-greedy algorithm for such a case is obtained that computes the first n Whittle indices in $O\left(n\right)$ operations, under mild conditions on model parameters. An alternative approach to long-run average birth–death restless bandits is developed in [50]. Yet, to prove indexability, Proposition 2 in that paper assumes both that threshold policies are optimal (which in the birth–death model in [3] is obtained as a byproduct of the PCL-indexability conditions) and that a certain function of steady-state probabilities is strictly increasing, which is nontrivial to verify. Further, the Reference [50] does not give an index-computing algorithm, but an expression for the Whittle index in terms of steady-state metrics, which also appears in [3].

A Whittle index computing algorithm for general continuous-time finite-state restless bandits has been recently proposed in [41], focusing on the average criterion, as it is stated there that the approach does not apply to the discounted criterion. As for the arithmetic-operation complexity of the Whittle index algorithm in [41], which also checks for indexability, Remark 3.5 states that it performs $O\left(n^4{2}^n\right)$ operations. Even in the case that indexability is known to hold and that threshold policies are optimal, it is stated in Remark 4.1 of that paper that the complexity of that index algorithm reduces to $O\left(n^5\right)$ operations. This is to be contrasted with the Whittle index algorithm presented herein, which has a cubic operation complexity in the number n of bandit states.

3. Review of Finite-State Restless Bandit Whittle Indexation via PCL-Indexability

This section reviews key results of the author’s approach to restless bandit indexation, as it applies to a single finite-state semi-Markov restless bandit project, which we will simply refer to as a “project” in the sequel.

3.1. SMDP Restless Bandits and Their Discrete-Stage Reformulation

Consider a SMDP restless bandit project, as outlined in Section 1. We next describe its standard discrete-stage reformulation (see, for example, ([1], Ch. 11)). If, at decision epoch $t_k$ the project lies in state $X_k=i$ and action $A_k=a$ is chosen, the joint distribution of the length $t_{k+1}-t_k$ of the following $(i,a)$-stage and embedded state $X_{k+1}$ is characterized by the transition distribution function

$$H_{ij}^a\left(t\right)\triangleq {\mathbb{P}}_i^a\left\{t_{k+1}-t_k⩽t,X_{k+1}=j\right\},t⩾0,$$

having Laplace–Stieltjes transform (LST)

$${\psi }_{ij}^a\left(\alpha \right)\triangleq {\int }_0^{\infty }{e}^{-\alpha t}d{H}_{ij}^a\left(t\right)={\mathbb{E}}_i^a\left[1_{\{X_{k+1}=j\}}{e}^{-\alpha (t_{k+1}-t_k)}\right],\alpha >0,$$

where ${\mathbb{P}}_i^a$ and ${\mathbb{E}}_i^a$ denote probability and expectation conditional on starting from state i with action a ($X_k=i,A_k=a$).

From $H_{ij}^a\left(t\right)$, we have that the distribution of the length of an $(i,a)$-stage is

$$H_i^a\left(t\right)\triangleq {\mathbb{P}}_i^a\{t_{k+1}-t_k⩽t\}=\sum _{j\in \mathcal{N}}{H}_{ij}^a\left(t\right),$$

having LST

$${\psi }_i^a\left(\alpha \right)\triangleq {\mathbb{E}}_i^a\left[e^{-\alpha (t_{k+1}-t_k)}\right]=\sum _{j\in \mathcal{N}}{\psi }_{ij}^a\left(\alpha \right),$$

(2)

and mean

$$m_i^a\triangleq {\mathbb{E}}_i^a[t_{k+1}-t_k\mid X_k=i,A_k=a]={\int }_0^{\infty }td{H}_i^a\left(t\right).$$

The one-stage transition probabilities for the embedded process are

$$p_{ij}^a\triangleq {\mathbb{P}}_i^a\{X_{k+1}=j\mid X_k=i,A_k=a\}=\underset{t\to \infty }{lim}{H}_{ij}^a\left(t\right)=\underset{\alpha \searrow 0}{lim}{\psi }_{ij}^a\left(\alpha \right).$$

Recall that the process $X\left(t\right)$ may change state between successive decision epochs. Its dynamics within an $(i,a)$-stage are characterized by

$${\tilde{p}}_{ij}^a\left(s\right)\triangleq {\mathbb{P}}_i^a\{X(t_k+s)=j\mid t_{k+1}-t_k>s\},s⩾0,$$

the conditional probability that, s time units after the start of an $(i,a)$-stage, and given that this is still ongoing, the project occupies state j. We can thus formulate the expected discounted amount of resource consumed and reward obtained in an $(i,a)$-stage, respectively, as 

$$q_i^a\triangleq {\mathbb{E}}_i^a\left[{\int }_{t_k}^{t_{k+1}}{Q}_{X\left(t\right)}^{A_k}{e}^{-\alpha (t-t_k)}dt\right]=\sum _{j\in \mathcal{N}}{Q}_j^a{\int }_0^{\infty }{\tilde{p}}_{ij}^a\left(s\right){\overline{H}}_i^a\left(s\right)e^{-\alpha s}ds$$

(3)

and

$$r_i^a\triangleq {\mathbb{E}}_i^a\left[{\int }_{t_k}^{t_{k+1}}{R}_{X\left(t\right)}^{A_k}{e}^{-\alpha (t-t_k)}dt\right]=\sum _{j\in \mathcal{N}}{R}_j^a{\int }_0^{\infty }{\tilde{p}}_{ij}^a\left(s\right){\overline{H}}_i^a\left(s\right)e^{-\alpha s}ds,$$

(4)

where ${\overline{H}}_i^a\left(s\right)\triangleq 1-H_i^a\left(s\right)$ is the tail distribution of the length of an $(i,a)$-stage.

Recall that we denote by $n\triangleq \left|\mathcal{N}\right|$ the number of project states.

3.2. Indexability, Whittle Index, and the Achievable Resource–Reward Performance Region

We start by considering the discounted criterion with rate $\alpha >0$, deferring discussion of the average criterion to Section 6. We consider the following project performance metrics to evaluate a policy $\pi \in \Pi$ starting from a state i: the reward metric

$$F_i^{\pi }\triangleq {\mathbb{E}}_i^{\pi }\left[{\int }_0^{\infty }{R}_{X\left(t\right)}^{A\left(t\right)}{e}^{-\alpha t}dt\right]={\mathbb{E}}_i^{\pi }\left[\sum _{k=0}^{\infty }{r}_{X_k}^{A_k}{e}^{-\alpha t_k}\right],$$

(5)

measuring the expected discounted value of rewards obtained, and the resource metric

$$G_i^{\pi }\triangleq {\mathbb{E}}_i^{\pi }\left[{\int }_0^{\infty }{Q}_{X\left(t\right)}^{A\left(t\right)}{e}^{-\alpha t}dt\right]={\mathbb{E}}_i^{\pi }\left[\sum _{k=0}^{\infty }{q}_{X_k}^{A_k}{e}^{-\alpha t_k}\right],$$

(6)

giving the expected discounted quantity of resource consumed. Note that the right-hand side expectations in (5) and (6) are formulated in terms of the aforementioned discrete-stage reformulation.

We will also refer to the metrics obtained by drawing the initial state from a probability mass vector $\mathbf{p}={\left(p_i\right)}_{i\in \mathcal{N}}$, given by

$$F_{\mathbf{p}}^{\pi }\triangleq \sum _{i\in \mathcal{N}}{p}_i{F}_i^{\pi }\mathrm{and}{G}_{\mathbf{p}}^{\pi }\triangleq \sum _{i\in \mathcal{N}}{p}_i{G}_i^{\pi }.$$

(7)

To calibrate the marginal value of engaging the project in each state, we introduce a parameter $\lambda \in \mathbb{R}$ representing the resource (unit) price, and define the (net) project value metric$V_{\mathbf{p}}^{\pi }\left(\lambda \right)\triangleq F_{\mathbf{p}}^{\pi }-\lambda G_{\mathbf{p}}^{\pi }$, so the optimal project value is $V_{\mathbf{p}}^{*}\left(\lambda \right)\triangleq {sup}_{\pi \in \Pi }{V}_{\mathbf{p}}^{\pi }\left(\lambda \right)$. We also write $V_i^{\pi }\left(\lambda \right)$ and $V_i^{*}\left(\lambda \right)$ as in Section 1. Consider the parametric family $\mathcal{P}$ of λ-price problems$\left(P_{\lambda }\right)$ defined in (1) for $\lambda \in \mathbb{R}$. Thus, for given $\lambda$, problem (1) is to find an admissible policy that maximizes the project value.

The dynamic programming (DP) optimality equations for $\lambda$-price problem $\left(P_{\lambda }\right)$ are

$$V_i^{*}\left(\lambda \right)=\underset{a\in \{0,1\}}{max}{r}_i^a-\lambda q_i^a+\sum _{j\in \mathcal{N}}{\psi }_{ij}^a{V}_j^{*}\left(\lambda \right),i\in \mathcal{N},$$

(8)

where we write ${\psi }_{ij}^a={\psi }_{ij}^a\left(\alpha \right)$. Classic results of the SMDP theory ensure existence of a $P_{\lambda }$-optimal policy ${\pi }_{\lambda }^{*}$ solving (1) that is stationary deterministic (${\pi }_{\lambda }^{*}\in {\Pi }^{SD}$).

We will also consider the optimal project value starting from state i with initial action a, given by

$$V_i^{\langle a,*\rangle }\left(\lambda \right)\triangleq r_i^a-\lambda q_i^a+\sum _{j\in \mathcal{N}}{\psi }_{ij}^a{V}_j^{*}\left(\lambda \right),$$

(9)

and say that action a is $P_{\lambda }$-optimal in state i if $V_i^{\langle a,*\rangle }\left(\lambda \right)⩾V_i^{\langle 1-a,*\rangle }\left(\lambda \right)$. It will be convenient to reformulate such a definition in terms of the sign of ${\Delta }_{a=0}^{a=1}{V}_i^{\langle a,*\rangle }\left(\lambda \right)\triangleq V_i^{\langle 1,*\rangle }\left(\lambda \right)-V_i^{\langle 0,*\rangle }\left(\lambda \right)$, the marginal value of engaging the project in state i. Thus, action $a=1$ is $P_{\lambda }$-optimal in i if ${\Delta }_{a=0}^{a=1}{V}_i^{\langle a,*\rangle }\left(\lambda \right)⩾0$; $a=0$ is $P_{\lambda }$-optimal in i if ${\Delta }_{a=0}^{a=1}{V}_i^{\langle a,*\rangle }\left(\lambda \right)⩽0$; and, hence, both actions are $P_{\lambda }$-optimal in i if, and only if,

$${\Delta }_{a=0}^{a=1}{V}_i^{\langle a,*\rangle }\left(\lambda \right)=0.$$

(10)

Since actions are binary, it is convenient to represent a stationary deterministic policy by its active set $S\subseteq \mathcal{N}$, which is the subset of states where, at a decision epoch, the policy chooses the active action. We will refer to the S-active policy and write $F_i^S$, $G_i^S$, and $V_i^S\left(\lambda \right)$.

Thus, we can reformulate $\lambda$-price problem (1) as the discrete optimization problem

$$\left(P_{\lambda }\right):\mathrm{find}{S}^{*}\subseteq \mathcal{N}\mathrm{such} \mathrm{that}{V}_i^{S^{*}}\left(\lambda \right)=V_i^{*}\left(\lambda \right)\mathrm{for} \mathrm{every}i\in \mathcal{N}.$$

(11)

For a given resource price $\lambda$, the $P_{\lambda }$-optimal active and passive sets are given by

$$S^{*,1}\left(\lambda \right)\triangleq \left\{i\in \mathcal{N}:{\Delta }_{a=0}^{a=1}{V}_i^{\langle a,*\rangle }\left(\lambda \right)⩾0\right\}\mathrm{and}{S}_{\lambda }^{*,0}\triangleq \left\{i\in \mathcal{N}:{\Delta }_{a=0}^{a=1}{V}_i^{\langle a,*\rangle }\left(\lambda \right)⩽0\right\}.$$

(12)

3.3. Indexability

We will address the parametric problem collection $\mathcal{P}$ in (1) through the concept of indexability, extended in [2] from classic to restless bandits with resource consumption $Q_i^a\triangleq a$, and further extended in [3] to general $Q_i^a$.

Under indexability, the $P_{\lambda }$-optimal active and passive sets $S^{*,1}\left(\lambda \right)$ and $S^{*,0}\left(\lambda \right)$ are characterized by an index attached to project states. Note that the definition below refers to the sign function $sgn:\mathbb{R}\to \{-1,0,1\}$, and follows the formulation of indexability in ([25], Definition 1).

Definition 1

(Indexability and Whittle index). The project is indexable if there exists a mapping $i\mapsto {\lambda }_i^{*}$ for $i\in \mathcal{N}$, such that

$$sgn{\Delta }_{a=0}^{a=1}{V}_i^{\langle a,*\rangle }\left(\lambda \right)=sgn({\lambda }_i^{*}-\lambda ),\mathrm{for} \mathrm{every} \mathrm{state}i\in \mathcal{N}\mathrm{and} \mathrm{price}\lambda \in \mathbb{R}.$$

(13)

We call ${\lambda }_i^{*}$ the project’s (generalized) Whittle index (Gittins index if the project is non-restless$).$

Thus, under indexability, the $P_{\lambda }$-optimal active and passive sets $S^{*,1}\left(\lambda \right)$ and $S^{*,0}\left(\lambda \right)$ are characterized as

$$S^{*,1}\left(\lambda \right)\triangleq \left\{i\in \mathcal{N}:{\lambda }_i^{*}⩾\lambda \right\}\mathrm{and}{S}_{\lambda }^{*,0}\triangleq \left\{i\in \mathcal{N}:{\lambda }_i^{*}⩽\lambda \right\}.$$

(14)

As for the economic interpretation of the Whittle index, it is shown in [3,4,27] that ${\lambda }_i^{*}$ measures the marginal productivity of the resource at state i.

3.4. PCL-Indexability and Adaptive-Greedy Algorithm

In applications of Whittle indexation to given restless bandit models, researchers are concerned with establishing analytically their indexability and efficiently computing the index. For that purpose, the author introduced the PCL-indexability conditions in [3,4,24] because they are grounded on the satisfaction of partial conservation laws (PCLs). See [38] for a survey on conservation laws in stochastic scheduling and multi-class queueing systems.

Such conditions can be used to establish indexability relative to a nonempty family $\mathcal{F}\subseteq 2^{\mathcal{N}}$ of active sets that one should posit a priori (based on insight on the model’s structure).

Definition 2

($\mathcal{F}$-indexability). We say that the project, or more precisely, the parametric collection $\mathcal{P}$ of λ-price problems $\left(P_{\lambda }\right)$ in (11), is $\mathcal{F}$-indexable if: (i) it is indexable; and (ii) for every price λ, there exists an optimal active set $S^{*}\left(\lambda \right)\in \mathcal{F}$, such that the $S^{*}\left(\lambda \right)$-active policy is $P_{\lambda }$-optimal.

Thus, for an $\mathcal{F}$-indexable project, problem (11) can be reduced to

$$\left(P_{\lambda }\right):\mathrm{find}{S}^{*}\in \mathcal{F}\mathrm{such} \mathrm{that}{V}_i^{S^{*}}\left(\lambda \right)=V_i^{*}\left(\lambda \right)\mathrm{for} \mathrm{every}i\in \mathcal{N},$$

(15)

since, in such a case $\mathcal{F}$-policies (those with active sets $S\in \mathcal{F}$) are optimal for (11).

Algorithmic considerations lead us to require that active-set family $\mathcal{F}$ satisfies the connectedness conditions stated next. We will refer to the inner boundary of S with respect to $\mathcal{F}$, given by

$${\partial }_{\mathcal{F}}^{\mathrm{in}}S\triangleq \{i\in S:S\backslash \left\{i\right\}\in \mathcal{F}\},$$

and to the outer boundary of S with respect to $\mathcal{F}$, given by

$${\partial }_{\mathcal{F}}^{\mathrm{out}}S\triangleq \{j\in S^c:S\cup \left\{j\right\}\in \mathcal{F}\}.$$

Assumption 1.

(i)

For every nonempty $S\in \mathcal{F}$, ${\partial }_{\mathcal{F}}^{\mathrm{in}}S$ is nonempty.

(ii)

For every $S\in \mathcal{F}\backslash \left\{\mathcal{N}\right\}$, ${\partial }_{\mathcal{F}}^{\mathrm{out}}S$ is nonempty.

Note that from the requirement that $\mathcal{F}$ be nonempty, along with Assumption 1, it follows that $\varnothing ,\mathcal{N}\in \mathcal{F}$.

To formulate the following result, we need to consider certain marginal metrics. For an action $a\in \{0,1\}$ and an active set $S\subseteq \mathcal{N}$, denote by $\langle a,S\rangle$ the policy taking initially action a, and thereafter following the S-active policy. For a state i and an active set S, define the marginal resource (consumption) metric by

$$g_i^S\triangleq G_i^{\langle 1,S\rangle }-G_i^{\langle 0,S\rangle },$$

(16)

that is, as the marginal increase in resource consumed resulting from taking first the active rather than the passive action at state i, provided that the S-active policy is followed thereafter.

Define also the marginal reward metric by

$$f_i^S\triangleq F_i^{\langle 1,S\rangle }-F_i^{\langle 0,S\rangle },$$

(17)

that is, as the marginal increase in the value of rewards gained. Finally, for $g_i^S\ne 0$, define the marginal productivity rate metric by

$${\lambda }_i^S\triangleq \frac{f_i^S}{g_i^S}.$$

(18)

The following definition refers to the adaptive-greedy index algorithm, which is given in Algorithms 1 and 2 in its top-down and bottom-up versions, respectively. In the former, index values are computed from largest to smallest, whereas in the latter they are computed in the opposite order.  

Algorithm 1: Adaptive-greedy algorithm: top-down version ${\mathrm{AG}}_{\mathcal{F}}^{TD}$.

Output: ${\{j_k,{\lambda }_{j_k}^{*}\}}_{k=1}^n$ S0 := ∅ $\mathbf{for} k:=1$tondo   pick  $j_k\in \arg \max \{{\lambda }_j^{S_{k-1}}:j\in {\partial }_{\mathcal{F}}^{\mathrm{out}}{S}_{k-1}\}$   ${\lambda }_{j_k}^{*}:={\lambda }_{j_k}^{S_{k-1}}$;   $S_k:=S_{k-1}\cup \left\{j_k\right\}$ end { for }

Algorithm 2: Adaptive-greedy algorithm: bottom-up version ${\mathrm{AG}}_{\mathcal{F}}^{\mathrm{BU}}$.

Output: ${\{i_k,{\lambda }_{i_k}^{*}\}}_{k=1}^n$ $S_0^{\prime }:=\mathcal{N}$ $\mathbf{for}k:=1$tondo   pick  $i_k\in \arg \min \{{\lambda }_i^{S_{k-1}^{\prime }}:i\in {\partial }_{\mathcal{F}}^{\mathrm{in}}{S}_{k-1}^{\prime }\}$   ${\lambda }_{i_k}^{*}:={\lambda }_{i_k}^{S_{k-1}^{\prime }}$;   $S_k^{\prime }:=S_{k-1}^{\prime }\backslash \left\{i_k\right\}$ end { for }

Definition 3

(PCL$\left(\mathcal{F}\right)$-indexability). We say that the project isPCL$\left(\mathcal{F}\right)$-indexable if:

(i) 

for every active set $S\in \mathcal{F}$, $g_i^S>0$ for each $i\in \mathcal{N}$; and

(ii) 

algorithm ${\mathrm{AG}}_{\mathcal{F}}^{\mathrm{TD}}$ computes a monotone non-increasing index sequence $({\lambda }_{j_1}^{*}⩾\cdots ⩾{\lambda }_{j_n}^{*})$; or algorithm ${\mathrm{AG}}_{\mathcal{F}}^{\mathrm{BU}}$ computes a monotone non-decreasing index sequence $({\lambda }_{i_1}^{*}⩽\cdots ⩽{\lambda }_{i_n}^{*})$.

Theorem 1.

Suppose that the project is PCL$\left(\mathcal{F}\right)$-indexable. Then, it is $\mathcal{F}$-indexable and the index ${\lambda }_i^{*}$ computed by either adaptive-greedy index algorithm is its Whittle index.

Theorem 1 was introduced and proven, in increasingly general settings, in ([24], Corollary 2), (Ref. [3], Theorem 6.3) and ([4], Theorem 4.1). The author’s work reviewed in [27] demonstrates the applicability of such a result to a number of relevant restless bandit models.

Regarding the interpretation of positive marginal resource condition (i) in Definition 3, it is shown in ([3], Proposition 6.2) and in ([4], Lemma 4.3) that it can be reformulated in terms of the resource metric as follows: for $\mathbf{p}>\mathbf{0}$,

$$G_{\mathbf{p}}^{S\backslash \left\{i\right\}}<G_{\mathbf{p}}^S<G_{\mathbf{p}}^{S\cup \left\{j\right\}},S\in \mathcal{F},i\in S,j\in S^c.$$

(19)

Thus, condition (i) represents a strong form of monotone increasingness of resource metric $G_{\mathbf{p}}^S$ in its active set S relative to inclusion. Additionally, recalling the definition of $G_{\mathbf{p}}^{\pi }$ in (7) in terms of metrics $G_i^{\pi }$, (19) can, in turn, be reformulated in terms of the latter metrics as follows: for $S\in \mathcal{F}$, $i\in S$ and $j\in S^c$,

$${\mathbf{G}}^{S\backslash \left\{i\right\}}⪇{\mathbf{G}}^S⪇{\mathbf{G}}^{S\cup \left\{j\right\}},S\in \mathcal{F},i\in S,j\in S^c$$

(20)

where ${\mathbf{G}}^{\pi }={\left(G_{i_0}^{\pi }\right)}_{i_0\in \mathcal{N}}$ and “⪇” means “less than or equal to componentwise, but not equal.”

Along with condition (i), condition (ii) in Definition 3 is interpreted in ([3], Section 6.4) in terms of satisfaction of the law of diminishing marginal returns (to resource usage) in economics, relative to $\mathcal{F}$-policies. Such an interpretation is based on the result in ([3], Proposition 6.4(a)) that the marginal productivity rates ${\lambda }_i^S$ in (18) for $S\in \mathcal{F}$ can be recast in terms of resource and reward metrics as

$${\lambda }_i^S=\left\{\begin{array}{cc}{\displaystyle \frac{F_{\mathbf{p}}^S-F_{\mathbf{p}}^{S\backslash \left\{i\right\}}}{G_{\mathbf{p}}^S-G_{\mathbf{p}}^{S\backslash \left\{i\right\}}},}\hfill & i\in S\hfill \\ \\ {\displaystyle \frac{F_{\mathbf{p}}^{S\cup \left\{i\right\}}-F_{\mathbf{p}}^S}{G_{\mathbf{p}}^{S\cup \left\{i\right\}}-G_{\mathbf{p}}^S},}\hfill & i\in S^c.\hfill \end{array}\right.$$

(21)

Such a reformulation allows us, in turn, to reformulate the adaptive-greedy algorithms above in a geometrically intuitive form in the resource–reward $(\gamma ,\varphi )$ plane, as shown in Algorithms 3 and 4. We thus see that the top-down algorithm ${\mathrm{AG}}_{\mathcal{F}}^{\mathrm{TD}}$ aims to traverse the upper boundary $\overline{\partial }{\mathcal{R}}_{\mathbf{p}}$ of the achievable resource–reward performance region ${\mathcal{R}}_{\mathbf{p}}$ from left to right using only active sets in $\mathcal{F}$, whereas the bottom-up version ${\mathrm{AG}}_{\mathcal{F}}^{\mathrm{BU}}$ seeks to traverse such an upper boundary from right to left, proceeding in a greedy fashion at each step. In such a setting, condition (ii) in Definition 3 means that the function obtained by linear interpolation on the points $(G_{\mathbf{p}}^S,F_{\mathbf{p}}^S)$ produced by either algorithm is concave. The remarkable result in Theorem 1 is that this, along with condition (i), suffices to ensure that such a function characterizes the upper boundary $\overline{\partial }{\mathcal{R}}_{\mathbf{p}}$ and the Whittle index.

In [4], such results are extended to restless projects on the denumerable state space of nonnegative integers, for which the resource metric is increasing along the nested family of active sets induced by the corresponding ordering. Further, in Section 3 of that paper the $\mathcal{F}$-indexability of such projects is characterized in terms of satisfaction of the law of diminishing marginal returns relative to $\mathcal{F}$-policies. In [25], such results are further extended to continuous-state projects.

Algorithm 3: Reformulated adaptive-greedy index algorithm: top-down version ${\mathrm{AG}}_{\mathcal{F}}^{\mathrm{TD}}$.

Output: ${\{j_k,{\lambda }_{j_k}^{*}\}}_{k=1}^n$ S0 := ∅ $\mathbf{for} k:=1$tondo   pick  ${\displaystyle j_k\in \arg \max \left\{\frac{F_{\mathbf{p}}^{S_{k-1}\cup \left\{j\right\}}-F_{\mathbf{p}}^{S_{k-1}}}{G_{\mathbf{p}}^{S_{k-1}\cup \left\{j\right\}}-G_{\mathbf{p}}^{S_{k-1}}}:j\in {\partial }_{\mathcal{F}}^{\mathrm{out}}{S}_{k-1}\right\}}$   ${\lambda }_{j_k}^{*}:={\lambda }_{j_k}^{S_{k-1}}$;   $S_k:=S_{k-1}\cup \left\{j_k\right\}$ end { for }

Algorithm 4: Reformulated adaptive-greedy index algorithm: bottom-up version ${\mathrm{AG}}_{\mathcal{F}}^{\mathrm{BU}}$.

Output: ${\{i_k,{\lambda }_{i_k}^{*}\}}_{k=1}^n$ $S_0^{\prime }:=\mathcal{N}$ $\mathbf{for}k:=1$tondo   pick  ${\displaystyle i_k\in \arg \min \left\{\frac{F_{\mathbf{p}}^{S_{k-1}^{\prime }}-F_{\mathbf{p}}^{S_{k-1}^{\prime }\backslash \left\{i\right\}}}{G_{\mathbf{p}}^{S_{k-1}^{\prime }}-G_{\mathbf{p}}^{S_{k-1}^{\prime }\backslash \left\{i\right\}}}:i\in {\partial }_{\mathcal{F}}^{\mathrm{in}}{S}_{k-1}^{\prime }\right\}}$   ${\lambda }_{i_k}^{*}:={\lambda }_{i_k}^{S_{k-1}^{\prime }}$;   $S_k^{\prime }:=S_{k-1}^{\prime }\backslash \left\{i_k\right\}$ end { for }

4. Laying the Groundwork for an Efficient Implementation of the Adaptive-Greedy Algorithm

This section lays the groundwork for developing an efficient implementation of the adaptive-greedy index algorithm. It draws on LP methods, based on formulating the $\lambda$-price problem (1) as a parametric LP problem, and elucidating the structure of its simplex tableaux.

4.1. Optimality Equations and Parametric LP Formulation

While the LP formulation below is well-known in SMDP theory (cf. [51]), for convenience we next outline it, starting from the optimality Equations (8) for (1). The primal LP formulation of such optimality equations is

$$\begin{array}{cc}\hfill V_{\mathbf{p}}^{*}\left(\lambda \right)=& min\sum _{j\in \mathcal{N}}{p}_j{v}_j\hfill \\ \hfill & \mathrm{subject}\mathrm{to}\hfill \\ \hfill x_i^a:& v_i-\sum _{j\in \mathcal{N}}{\psi }_{ij}^a{v}_j⩾r_i^a-\lambda q_i^a,(i,a)\in \mathcal{N}\times \{0,1\}.\hfill \end{array}$$

Our analyses will instead use the dual problem,

$$\begin{array}{cc}\hfill V_{\mathbf{p}}^{*}\left(\lambda \right)=& max\sum _{(j,a)\in \mathcal{N}\times \{0,1\}}(r_j^a-\lambda q_j^a)x_j^a\hfill \\ \hfill & \mathrm{subject}\mathrm{to}\hfill \\ \hfill v_j:& \sum _{a\in \{0,1\}}(x_j^a-\sum _{j\in \mathcal{N}}{\psi }_{ij}^a{x}_i^a)=p_j,j\in \mathcal{N}\hfill \\ \hfill & x_j^a⩾0,(j,a)\in \mathcal{N}\times \{0,1\}.\hfill \end{array}$$

It will be convenient to deal with the latter in matrix notation, as 

$$\begin{array}{cc}\hfill V_{\mathbf{p}}^{*}\left(\lambda \right)=& max({\mathbf{r}}^0-\lambda {\mathbf{c}}^0){\mathbf{x}}^0+({\mathbf{r}}^1-\lambda {\mathbf{c}}^1){\mathbf{x}}^1\hfill \\ \hfill & \mathrm{subject}\mathrm{to}\hfill \\ \hfill & \left[\begin{array}{cc}{(\mathbf{I}-{\mathsf{\Psi }}^0)}^{\mathsf{T}}& {(\mathbf{I}-{\mathsf{\Psi }}^1)}^{\mathsf{T}}\end{array}\right]\left(\begin{array}{c}{\mathbf{x}}^0\\ {\mathbf{x}}^1\end{array}\right)=\mathbf{p}\hfill \\ \hfill & {\mathbf{x}}^0,{\mathbf{x}}^1⩾\mathbf{0},\hfill \end{array}$$

(22)

where ${\mathbf{x}}^a=\left(x_j^a\right)$ is a column vector, ${\mathbf{r}}^a=\left(r_j^a\right)$ and ${\mathbf{c}}^a=\left(q_j^a\right)$ are row vectors, and ${}^{\mathsf{T}}$ is the transposition operator.

Dual variables $x_j^a$ correspond to the project’s discounted state-action occupation measures. For an initial state i, policy $\pi$, action a and state j, let

$$x_{ij}^{a,\pi }\triangleq {\mathbb{E}}_i^{\pi }\left[\sum _{k=0}^{\infty }{1}_{\{X\left(t_k\right)=j,A\left(t_k\right)=a\}}{e}^{-\alpha t_k}\right]$$

be the expected discounted number of $(j,a)$-stages under $\pi$ starting from i. If the initial state is drawn from $\mathbf{p}$, $x_j^a$ corresponds to occupation measure $x_{\mathbf{p}j}^{a,\pi }\triangleq {\sum }_i{p}_i{x}_{ij}^{a,\pi }$. Note that reward and resource metrics are linear functions of such occupation measures: writing ${\mathbf{x}}_{\mathbf{p}}^{a,\pi }={\left(x_{\mathbf{p}j}^{a,\pi }\right)}_{j\in \mathcal{N}}$,

$$F_{\mathbf{p}}^{\pi }=\sum _{(j,a)\in \{0,1\}\times \mathcal{N}}{r}_j^a{x}_{\mathbf{p}j}^{a,\pi }={\mathbf{r}}^0{\mathbf{x}}_{\mathbf{p}}^{0,\pi }+{\mathbf{r}}^1{\mathbf{x}}_{\mathbf{p}}^{1,\pi }\mathrm{and}{G}_{\mathbf{p}}^{\pi }=\sum _{(j,a)\in \{0,1\}\times \mathcal{N}}{q}_j^a{x}_{\mathbf{p}j}^{a,\pi }={\mathbf{c}}^0{\mathbf{x}}_{\mathbf{p}}^{0,\pi }+{\mathbf{c}}^1{\mathbf{x}}_{\mathbf{p}}^{1,\pi }.$$

(23)

4.2. Bases, Basic Feasible Solutions, and Reduced Costs

We next analyze the LP problem (22), starting by elucidating its basic feasible solutions (BFS). Each BFS is obtained from a basis corresponding to an active set $S\subseteq \mathcal{N}$, and hence we will refer to the S-active BFS. Yet, note that different S’s might yield the same BFS. For given S, we decompose the above vectors and matrices as

$${\mathbf{x}}^a=\left(\begin{array}{c}{\mathbf{x}}_S^a\\ {\mathbf{x}}_{S^c}^a\end{array}\right),\mathbf{p}=\left(\begin{array}{c}{\mathbf{p}}_S\\ {\mathbf{p}}_{S^c}\end{array}\right),{\mathsf{\Psi }}^a=\left(\begin{array}{cc}{\mathsf{\Psi }}_{SS}^a& {\mathsf{\Psi }}_{S{S}^c}^a\\ {\mathsf{\Psi }}_{S^{c}S}^a& {\mathsf{\Psi }}_{S^c{S}^c}^a\end{array}\right),\mathbf{I}=\left(\begin{array}{cc}{\mathbf{I}}_{SS}& {\mathbf{0}}_{S{S}^c}\\ {\mathbf{0}}_{S^{c}S}& {\mathbf{I}}_{S^c{S}^c}\end{array}\right),$$

and introduce the matrices

$$\begin{array}{cc}\hfill {\mathsf{\Psi }}^S& \triangleq \left(\begin{array}{cc}{\mathsf{\Psi }}_{SS}^1& {\mathsf{\Psi }}_{S{S}^c}^1\\ {\mathsf{\Psi }}_{S^{c}S}^0& {\mathsf{\Psi }}_{S^c{S}^c}^0\end{array}\right),{\mathsf{\Psi }}^{S^c}\triangleq \left(\begin{array}{cc}{\mathsf{\Psi }}_{SS}^0& {\mathsf{\Psi }}_{S{S}^c}^0\\ {\mathsf{\Psi }}_{S^{c}S}^1& {\mathsf{\Psi }}_{S^c{S}^c}^1\end{array}\right),\hfill \\ \hfill {\mathbf{B}}^S& \triangleq {(\mathbf{I}-{\mathsf{\Psi }}^S)}^{\mathsf{T}},{\mathbf{N}}^S\triangleq {(\mathbf{I}-{\mathsf{\Psi }}^{S^c})}^{\mathsf{T}},{\mathbf{H}}^S\triangleq {\left({\mathbf{B}}^S\right)}^{-1},{\mathbf{A}}^S\triangleq {\mathbf{H}}^S{\mathbf{N}}^S.\hfill \end{array}$$

(24)

Note that ${\mathsf{\Psi }}^S$ is the transition-transform matrix for the S-active policy. Additionally, ${\mathbf{B}}^S$ is the basis matrix in LP problem (22) for the S-active BFS, which has as basic variables

$$\left(\begin{array}{c}{\mathbf{x}}_S^1\\ {\mathbf{x}}_{S^c}^0\end{array}\right),$$

and ${\mathbf{N}}^S$ is the non-basic matrix of LP problem (22) corresponding to non-basic variables

$$\left(\begin{array}{c}{\mathbf{x}}_S^0\\ {\mathbf{x}}_{S^c}^1\end{array}\right).$$

We can thus rearrange the constraints in (22), decomposing them as

$${\mathbf{B}}^S\left(\begin{array}{c}{\mathbf{x}}_S^1\\ {\mathbf{x}}_{S^c}^0\end{array}\right)+{\mathbf{N}}^S\left(\begin{array}{c}{\mathbf{x}}_S^0\\ {\mathbf{x}}_{S^c}^1\end{array}\right)=\mathbf{p}.$$

We next evaluate performance metrics under the S-active policy. The notation $x_{\mathbf{p}j}^{a,S}$ below refers to occupation measure $x_{\mathbf{p}j}^{a,\pi }$ under such a policy. Further, ${\mathbf{F}}^S={\left(F_j^S\right)}_{j\in \mathcal{N}}$, ${\mathbf{G}}^S={\left(G_j^S\right)}_{j\in \mathcal{N}}$, ${\mathbf{f}}^S={\left(f_j^S\right)}_{j\in \mathcal{N}}$ and ${\mathbf{g}}^S={\left(g_j^S\right)}_{j\in \mathcal{N}}$ are row vectors.

Lemma 1.

(a)

$\left(\begin{array}{c}{\mathbf{x}}_{\mathbf{p}S}^{0,S}\\ {\mathbf{x}}_{\mathbf{p}{S}^c}^{1,S}\end{array}\right)=\mathbf{0}$ and $\left(\begin{array}{c}{\mathbf{x}}_{\mathbf{p}S}^{1,S}\\ {\mathbf{x}}_{\mathbf{p}{S}^c}^{0,S}\end{array}\right)={\mathbf{H}}^S\mathbf{p}$.

(b)

${\mathbf{G}}^S=\left(\begin{array}{cc}{\mathbf{c}}_S^1& {\mathbf{c}}_{S^c}^0\end{array}\right){\mathbf{H}}^S$.

(c)

${\mathbf{F}}^S=\left(\begin{array}{cc}{\mathbf{r}}_S^1& {\mathbf{r}}_{S^c}^0\end{array}\right){\mathbf{H}}^S$.

(d)

$\left(\begin{array}{cc}{\mathbf{g}}_S^S& -{\mathbf{g}}_{S^c}^S\end{array}\right)=\left(\begin{array}{cc}{\mathbf{c}}_S^1& {\mathbf{c}}_{S^c}^0\end{array}\right){\mathbf{A}}^S-\left(\begin{array}{cc}{\mathbf{c}}_S^0& {\mathbf{c}}_{S^c}^1\end{array}\right)$.

(e)

$\left(\begin{array}{cc}{\mathbf{f}}_S^S& -{\mathbf{f}}_{S^c}^S\end{array}\right)=\left(\begin{array}{cc}{\mathbf{r}}_S^1& {\mathbf{r}}_{S^c}^0\end{array}\right){\mathbf{A}}^S-\left(\begin{array}{cc}{\mathbf{r}}_S^0& {\mathbf{r}}_{S^c}^1\end{array}\right)$.

Proof. 

(a) Set to zero non-basic variables: ${\mathbf{x}}_{\mathbf{p}S}^{0,S}=\mathbf{0}$ and ${\mathbf{x}}_{\mathbf{p}{S}^c}^{1,S}=\mathbf{0}$. To calculate the values of basic variables, note that

$${\mathbf{B}}^S\left(\begin{array}{c}{\mathbf{x}}_{\mathbf{p}S}^1\\ {\mathbf{x}}_{\mathbf{p}{S}^c}^0\end{array}\right)=\mathbf{p},\mathrm{and} \mathrm{hence}\left(\begin{array}{c}{\mathbf{x}}_{\mathbf{p}S}^{1,S}\\ {\mathbf{x}}_{\mathbf{p}{S}^c}^{0,S}\end{array}\right)={\mathbf{H}}^S\mathbf{p}.$$

(b) Use part (a) with $\mathbf{p}={\mathbf{e}}_j$ (the jth unit coordinate vector) to formulate resource metrics as

$$G_j^S=\left(\begin{array}{cc}{\mathbf{c}}_S^1& {\mathbf{c}}_{S^c}^0\end{array}\right)\left(\begin{array}{c}{\mathbf{x}}_{jS}^{1,S}\\ {\mathbf{x}}_{j{S}^c}^{0,S}\end{array}\right)=\left(\begin{array}{cc}{\mathbf{c}}_S^1& {\mathbf{c}}_{S^c}^0\end{array}\right){\mathbf{H}}^S{\mathbf{e}}_j⟹{\mathbf{G}}^S=\left(\begin{array}{cc}{\mathbf{c}}_S^1& {\mathbf{c}}_{S^c}^0\end{array}\right){\mathbf{H}}^S.$$

(c) Proceed as in (b) to formulate reward metrics as

$$F_j^S=\left(\begin{array}{cc}{\mathbf{r}}_S^1& {\mathbf{r}}_{S^c}^0\end{array}\right)\left(\begin{array}{c}{\mathbf{x}}_{jS}^{1,S}\\ {\mathbf{x}}_{j{S}^c}^{0,S}\end{array}\right)=\left(\begin{array}{cc}{\mathbf{r}}_S^1& {\mathbf{r}}_{S^c}^0\end{array}\right){\mathbf{H}}^S{\mathbf{e}}_j⟹{\mathbf{F}}^S=\left(\begin{array}{cc}{\mathbf{r}}_S^1& {\mathbf{r}}_{S^c}^0\end{array}\right){\mathbf{H}}^S.$$

(d) Represent marginal resource metrics (cf. (16)) as

$${\mathbf{g}}_S^S={\mathbf{G}}_S^S-{\mathbf{c}}_S^0-{\mathbf{G}}^S{\left({\mathsf{\Psi }}_{S\mathcal{N}}^0\right)}^{\mathsf{T}}\mathrm{and}{\mathbf{g}}_{S^c}^S={\mathbf{c}}_{S^c}^1+{\mathbf{G}}^S{\left({\mathsf{\Psi }}_{S^{c}N}^1\right)}^{\mathsf{T}}-{\mathbf{G}}_{S^c}^S.$$

(25)

Recast now the equalities in (25), using (b), as 

$$\begin{array}{cc}\hfill \left(\begin{array}{cc}{\mathbf{g}}_S^S& -{\mathbf{g}}_{S^c}^S\end{array}\right)& ={\mathbf{G}}^S{\mathbf{N}}^S-\left(\begin{array}{cc}{\mathbf{c}}_S^0& {\mathbf{c}}_{S^c}^1\end{array}\right)=\left(\begin{array}{cc}{\mathbf{c}}_S^1& {\mathbf{c}}_{S^c}^0\end{array}\right){\mathbf{H}}^S{\mathbf{N}}^S-\left(\begin{array}{cc}{\mathbf{c}}_S^0& {\mathbf{c}}_{S^c}^1\end{array}\right)\hfill \\ \hfill & =\left(\begin{array}{cc}{\mathbf{c}}_S^1& {\mathbf{c}}_{S^c}^0\end{array}\right){\mathbf{A}}^S-\left(\begin{array}{cc}{\mathbf{c}}_S^0& {\mathbf{c}}_{S^c}^1\end{array}\right).\hfill \end{array}$$

(e) This part follows as part (d). □

The following result gives the reduced costs of the LP problem (22) in terms of the marginal resource and reward metrics in (16) and (17). It further uses such a result to represent such an LP problem’s objective in terms of reduced costs.

Lemma 2.

The reduced costs for non-basic variables corresponding to the S-active BFS for LP problem(22)are given by

$$\left(\begin{array}{cc}{\mathbf{f}}_S^S-\lambda {\mathbf{g}}_S^S& -{\mathbf{f}}_{S^c}^S+\lambda {\mathbf{g}}_{S^c}^S\end{array}\right),$$

(26)

and hence, the LP problem’s objective can be formulated as

$$\sum _{(j,a)\in \mathcal{N}\times \{0,1\}}(r_j^a-\lambda q_j^a)x_j^a=F_{\mathbf{p}}^S-\lambda G_{\mathbf{p}}^S-\sum _{j\in S}(f_j^S-\lambda g_j^S)x_j^0+\sum _{j\in S^c}(f_j^S-\lambda g_j^S)x_j^1.$$

(27)

Proof. 

The results follow from the well-known representation of reduced costs in LP theory, as given by Lemma 1 (d,e), along with the well-known representation of the LP problem’s objective in terms of the value of the current BFS and reduced costs. □

The following result, which follows from Lemma 2, represents metrics $G_{\mathbf{p}}^{\pi }$, $F_{\mathbf{p}}^{\pi }$ and objective $F_{\mathbf{p}}^{\pi }-\lambda G_{\mathbf{p}}^{\pi }$ in terms of their values under the S-active policy. These decomposition identities were first obtained in ([24], Theorem 3) and ([3], Proposition 6.1) via ad hoc arguments.

Lemma 3.

For any policy $\pi \in \Pi$:

(a)

${\displaystyle G_{\mathbf{p}}^{\pi }=G_{\mathbf{p}}^S-\sum _{j\in S}{g}_j^S{x}_{\mathbf{p}j}^{0,\pi }+\sum _{j\in S^c}{g}_j^S{x}_{\mathbf{p}j}^{1,\pi }}$.

(b)

${\displaystyle F_{\mathbf{p}}^{\pi }=F_{\mathbf{p}}^S-\sum _{j\in S}{f}_j^S{x}_{\mathbf{p}j}^{0,\pi }+\sum _{j\in S^c}{f}_j^S{x}_{\mathbf{p}j}^{1,\pi }}$.

(c)

${\displaystyle F_{\mathbf{p}}^{\pi }-\lambda G_{\mathbf{p}}^{\pi }=F_{\mathbf{p}}^S-\lambda G_{\mathbf{p}}^S-\sum _{j\in S}(f_j^S-\lambda g_j^S)x_{\mathbf{p}j}^{0,\pi }+\sum _{j\in S^c}(f_j^S-\lambda g_j^S)x_{\mathbf{p}j}^{1,\pi }}$.

The following result, first established in ([3], Corollary 6.1), elucidates the relationship between resource and reward metrics and the corresponding marginal metrics.

Lemma 4.

(a)

For $j\in S^c$, $G_{\mathbf{p}}^{S\cup \left\{j\right\}}=G_{\mathbf{p}}^S+g_j^S{x}_{\mathbf{p}j}^{1,S\cup \left\{j\right\}}$ and $F_{\mathbf{p}}^{S\cup \left\{j\right\}}=F_{\mathbf{p}}^S+f_j^S{x}_{\mathbf{p}j}^{1,S\cup \left\{j\right\}}$.

(b)

For $j\in S$, $G_{\mathbf{p}}^{S\backslash \left\{j\right\}}=G_{\mathbf{p}}^S-g_j^S{x}_{\mathbf{p}j}^{1,S\backslash \left\{j\right\}}$ and $F_{\mathbf{p}}^{S\backslash \left\{j\right\}}=F_{\mathbf{p}}^S-f_j^S{x}_{\mathbf{p}j}^{1,S\backslash \left\{j\right\}}$.

Proof. 

To obtain (a) use $\pi =S\cup \left\{j\right\}$ in Lemma 3 (a). Additionally, similarly with (b) with $\pi =S\backslash \left\{j\right\}$ and Lemma 3 (b). □

5. A Fast-Pivoting Index Algorithm for PCL$\left(\mathcal{F}\right)$-Indexable Projects

This section develops an efficient implementation of the adaptive-greedy index algorithm above, focusing on its top-down version, for a project that is PCL$\left(\mathcal{F}\right)$-indexable.

We start by noting that the index of such a project can be computed by deploying in the LP formulation (22) of the $\lambda$-price problem the classic parametric-objective simplex algorithm in [52]. In the present setting, the parametric simplex tableau for the S-active BFS is shown in

Table 1. Simplex tableau (parametric) for S-activeBFS, with pivot $a_{jj}^S$.

	${\left({\mathbf{x}}_S^0\right)}^{\mathrm{T}}$	$x_j^1$	${\left({\mathbf{x}}_{S^c\backslash \left\{j\right\}}^1\right)}^{\mathrm{T}}$
${\mathbf{x}}_S^1$	${\mathbf{A}}_{SS}^S$	${\mathbf{A}}_{Sj}^S$	${\mathbf{A}}_{S{S}^c\backslash \left\{j\right\}}^S$
$x_j^0$	${\mathbf{A}}_{jS}^S$	$\begin{array}{|c|}\hline a_{jj}^S\\ \hline\end{array}$	${\mathbf{A}}_{j{S}^c\backslash \left\{j\right\}}$
${\mathbf{x}}_{S^c\backslash \left\{j\right\}}^0$	${\mathbf{A}}_{S^c\backslash \left\{j\right\},S}^S$	${\mathbf{A}}_{S^c\backslash \left\{j\right\},j}^S$	${\mathbf{A}}_{S^c\backslash \left\{j\right\}{S}^c\backslash \left\{j\right\}}^S$
	${\mathbf{g}}_S^S$	$-g_j^S$	$-{\mathbf{g}}_{S^c\backslash \left\{j\right\}}^S$
	${\mathbf{f}}_S^S$	$-f_j^S$	$-{\mathbf{r}}_{S^c\backslash \left\{j\right\}}^S$

. This tableau has basic variables ${\mathbf{x}}_S^1$ and ${\mathbf{x}}_{S^c}^0$ in rows and non-basic variables ${\mathbf{x}}_S^0$ and ${\mathbf{x}}_{S^c}^1$ in columns, and further, has two rows of reduced-costs for non-basic variables. The tableau does not include right-hand sides or objectives, as they are not required in this context. The tableau is displayed just before pivoting on $a_{jj}^S$, where $j\in S^c$. That is, just before moving variable $x_j^0$ out of the basis, and  carrying $x_j^1$ into the basis, which corresponds to changing from the S- to the $S\cup \left\{j\right\}$-active BFS. After the pivot step, the updated tableau is shown in

Table 2. Tableau for $S\cup \left\{j\right\}$-active BFS, after pivoting on $a_{jj}^S$.

	${\left({\mathbf{x}}_S^0\right)}^{\mathrm{T}}$	$x_j^0$	${\left({\mathbf{x}}_{S^c\backslash \left\{j\right\}}^1\right)}^{\mathrm{T}}$
${\mathbf{x}}_S^1$	${\displaystyle {\mathbf{A}}_{SS}^S-{\left(a_{jj}^S\right)}^{-1}{\mathbf{A}}_{Sj}^S{\mathbf{A}}_{jS}^S}$	${\displaystyle -{\left(a_{jj}^S\right)}^{-1}{\mathbf{A}}_{Sj}^S}$	${\displaystyle {\mathbf{A}}_{S{S}^c\backslash \left\{j\right\}}^S-{\left(a_{jj}^S\right)}^{-1}{\mathbf{A}}_{Sj}^S{\mathbf{A}}_{j{S}^c\backslash \left\{j\right\}}^S}$
$x_j^1$	${\displaystyle {\left(a_{jj}^S\right)}^{-1}{\mathbf{A}}_{jS}^S}$	${\displaystyle {\left(a_{jj}^S\right)}^{-1}}$	${\displaystyle {\left(a_{jj}^S\right)}^{-1}{\mathbf{A}}_{j{S}^c\backslash \left\{j\right\}}^S}$
${\mathbf{x}}_{S^c\backslash \left\{j\right\}}^0$	${\displaystyle {\mathbf{A}}_{S^c\backslash \left\{j\right\},S}^S-{\left(a_{jj}^S\right)}^{-1}{\mathbf{A}}_{S^c\backslash \left\{j\right\},j}^S{\mathbf{A}}_{jS}^S}$	${\displaystyle -{\left(a_{jj}^S\right)}^{-1}{\mathbf{A}}_{S^c\backslash \left\{j\right\},j}^S}$	${\displaystyle {\mathbf{A}}_{S^c\backslash \left\{j\right\}{S}^c\backslash \left\{j\right\}}^S-{\left(a_{jj}^S\right)}^{-1}{\mathbf{A}}_{S^c\backslash \left\{j\right\},j}^S{\mathbf{A}}_{j{S}^c\backslash \left\{j\right\}}^S}$
	${\displaystyle {\mathbf{g}}_S^S+g_j^S{\left(a_{jj}^S\right)}^{-1}{\mathbf{A}}_{jS}^S}$	${\displaystyle g_j^S{\left(a_{jj}^S\right)}^{-1}}$	${\displaystyle -{\mathbf{g}}_{S^c\backslash \left\{j\right\}}^S+g_j^S{\left(a_{jj}^S\right)}^{-1}{\mathbf{A}}_{j{S}^c\backslash \left\{j\right\}}^S}$
	${\displaystyle {\mathbf{f}}_S^S+f_j^S{\left(a_{jj}^S\right)}^{-1}{\mathbf{A}}_{jS}^S}$	${\displaystyle f_j^S{\left(a_{jj}^S\right)}^{-1}}$	${\displaystyle -{\mathbf{f}}_{S^c\backslash \left\{j\right\}}^S+f_j^S{\left(a_{jj}^S\right)}^{-1}{\mathbf{A}}_{j{S}^c\backslash \left\{j\right\}}^S}$

.

One can readily use such tableaux to implement the top-down adaptive-greedy algorithm ${\mathrm{AG}}_{\mathcal{F}}^{\mathrm{TD}}$ in Algorithm 3, by first constructing the initial tableau for the ∅-active policy, and then carrying out $n=\left|\mathcal{N}\right|$ pivot steps, at each of which the former BFS active set is augmented by a state. Such a direct approach results in an implementation that we call the Conventional-Pivoting Index (CP) algorithm. An immediate counting argument shows that the n pivot steps of that algorithm perform $2n^3+O\left(n^2\right)$ arithmetic operations—without considering computation of the initial tableau, which will be addressed in Section 5.1 below.

Note that this algorithm can also be applied to a restless bandit instance to test numerically whether it is indexable. The project will be indexable if, and only if, the successive pivot steps, as the price parameter $\lambda$ decreases from ∞ to $-\infty$ in the parametric-objective simplex algorithm of [52], can be performed augmenting the current BFS by adding a state, thus producing a nested active-set family ${\mathcal{F}}_0=\{S_0,S_1,\dots ,S_n\}$ with $S_0=\varnothing \subset \cdots \subset S_n=\mathcal{N}$.

We now consider how to improve the efficiency of the CP algorithm, drawing on the observation that the tableau’s rows for basic variables ${\mathbf{x}}_S^1$ are not used to update the reduced costs. Thus, it is enough to update and store the reduced tableaux shown in

Table 3. Reduced tableau for S-active BFS, with pivot $a_{jj}^S$.

	${\left({\mathbf{x}}_S^0\right)}^{\mathrm{T}}$	$x_j^1$	${\left({\mathbf{x}}_{S^c\backslash \left\{j\right\}}^1\right)}^{\mathrm{T}}$
$x_j^0$	${\mathbf{A}}_{jS}^S$	$\begin{array}{|c|}\hline a_{jj}^S\\ \hline\end{array}$	${\mathbf{A}}_{j{S}^c\backslash \left\{j\right\}}^S$
${\mathbf{x}}_{S^c\backslash \left\{j\right\}}^0$	${\mathbf{A}}_{S^c\backslash \left\{j\right\},S}^S$	${\mathbf{A}}_{S^c\backslash \left\{j\right\},j}^S$	${\mathbf{A}}_{S^c\backslash \left\{j\right\}{S}^c\backslash \left\{j\right\}}^S$
	${\mathbf{g}}_S^S$	$-g_j^S$	$-{\mathbf{g}}_{S^c\backslash \left\{j\right\}}^S$
	${\mathbf{f}}_S^S$	$-f_j^S$	$-{\mathbf{f}}_{S^c\backslash \left\{j\right\}}^S$

. Table 2 demonstrates that a reduced tableau can be updated without using the rows that have been deleted. The resulting simplification of the CP algorithm yields the Reduced-Pivoting (RP) algorithm. By an elementary counting argument, it is readily seen that the RP algorithm carries out the n pivot steps in $n^3+O\left(n^2\right)$ operations, thus improving by a factor of 2 the arithmetic operation complexity of the CP algorithm.

We can exploit the assumption of PCL$\left(\mathcal{F}\right)$-indexability to reduce even further the operation count, by updating and storing only the minimal tableaux in

Table 4. Minimal tableau for S-active BFS.

	${\left({\mathbf{x}}_{S^c}^1\right)}^{\mathrm{T}}$
${\mathbf{x}}_{S^c}^0$	${\mathbf{A}}_{S^c{S}^c}^S$
	${\mathbf{g}}_{S^c}^S$
	${\mathbf{f}}_{S^c}^S$

. Such a tableau for the $S\cup \left\{j\right\}$-active BFS is computed from that for the S-active BFS in Table 4, as displayed in

Table 5. Minimal tableau for $S\cup \left\{j\right\}$-active BFS, obtained after pivoting on $a_{jj}^S$.

	${\left({\mathbf{x}}_{S^c\backslash \left\{j\right\}}^1\right)}^{\mathrm{T}}$
${\mathbf{x}}_{S^c\backslash \left\{j\right\}}^0$	${\displaystyle {\mathbf{A}}_{S^c\backslash \left\{j\right\}{S}^c\backslash \left\{j\right\}}^S-{\left(a_{jj}^S\right)}^{-1}{\mathbf{A}}_{S^c\backslash \left\{j\right\},j}^S{\mathbf{A}}_{j{S}^c\backslash \left\{j\right\}}^S}$
	${\displaystyle {\mathbf{g}}_{S^c\backslash \left\{j\right\}}^S-g_j^S{\left(a_{jj}^S\right)}^{-1}{\mathbf{A}}_{S^c\backslash \left\{j\right\},j}^S}$
	${\displaystyle {\mathbf{f}}_{S^c\backslash \left\{j\right\}}^S-f_j^S{\left(a_{jj}^S\right)}^{-1}{\mathbf{A}}_{S^c\backslash \left\{j\right\},j}^S}$

. This results in the fast-pivoting (FP) adaptive-greedy index algorithm ${\mathrm{FP}}_{\mathcal{F}}$ in Algorithm 5.

Algorithm 5: The fast-pivoting adaptive-greedy index algorithm ${\mathrm{FP}}_{\mathcal{F}}$.

Output: ${\{j_k,{\lambda }_{j_k}^{*}\}}_{k=1}^n$ solve ${\mathbf{A}}^{\left(0\right)}\left(\begin{array}{cc}{\mathbf{I}}_{\mathcal{N},\mathcal{N}\backslash \left\{j^{*}\right\}}-{\mathsf{\Psi }}_{\mathcal{N},\mathcal{N}\backslash \left\{j^{*}\right\}}^0& {\tilde{\mathbf{m}}}^0\end{array}\right)=\left(\begin{array}{cc}{\mathbf{I}}_{\mathcal{N},\mathcal{N}\backslash \left\{j^{*}\right\}}-{\mathsf{\Psi }}_{\mathcal{N},\mathcal{N}\backslash \left\{j^{*}\right\}}^1& {\tilde{\mathbf{m}}}^1\end{array}\right)$ $\left(\begin{array}{c}{\mathbf{g}}^{\left(0\right)}\\ {\mathbf{f}}^{\left(0\right)}\end{array}\right) :=\left(\begin{array}{c}{\mathbf{c}}^1\\ {\mathbf{r}}^1\end{array}\right)-\left(\begin{array}{c}{\mathbf{c}}^0\\ {\mathbf{r}}^0\end{array}\right){\mathbf{A}}^{\left(0\right)}$;   $S_0:=\varnothing$ $\mathbf{for}k:=1$tondo   ${\lambda }_j^{(k-1)} :=f_j^{(k-1)}/g_j^{(k-1)}, j\in {\partial }_{\mathcal{F}}^{\mathrm{out}}{S}_{k-1}$   pick $j_k\in \arg \max \left\{{\lambda }_j^{(k-1)}: j\in {\partial }_{\mathcal{F}}^{\mathrm{out}}{S}_{k-1}\right\}; {\lambda }_{j_k}^{*}:={\lambda }_{j_k}^{k-1}; S_k:=S_{k-1}\cup \left\{j_k\right\}$   if $k<n$ then    ${\mathbf{A}}_{S_k^c{j}_k}^{\left(k\right)}:=(1/a_{j_k{j}_k}^{\left(k-1\right)}){\mathbf{A}}_{S_k^c{j}_k}^{\left(k-1\right)}$;   ${\displaystyle {\mathbf{A}}_{S_k^c{S}_k^c}^{\left(k\right)}:={\mathbf{A}}_{S_k^c{S}_k^c}^{\left(k-1\right)}-{\mathbf{A}}_{S_k^c{j}_k}^{\left(k\right)}{\mathbf{A}}_{j_k{S}_k^c}^{\left(k-1\right)}}$   end { if }   ${\displaystyle {\mathbf{g}}_{S_k^c}^{\left(k\right)}:={\mathbf{g}}_{S_k^c}^{\left(k-1\right)}-g_{j_k}^{\left(k-1\right)}{\mathbf{A}}_{S_k^c{j}_k}^{\left(k\right)}}$;   ${\displaystyle {\mathbf{f}}_{S_k^c}^{\left(k\right)}:={\mathbf{f}}_{S_k^c}^{\left(k-1\right)}-f_{j_k}^{\left(k-1\right)}{\mathbf{A}}_{S_k^c{j}_k}^{\left(k\right)}}$ end { for }

The next result evaluates the operation count of algorithm ${\mathrm{FP}}_{\mathcal{F}}$, showing that it outperforms significantly that of algorithm RP. Note that the complexity of its loop—which performs the n pivot steps—matches that of Gaussian elimination for solving an $n\times n$ system of linear equations, and is hence unlikely that such complexity can be improved for general restless bandits.

Proposition 1.

The loop of algorithm ${\mathrm{FP}}_{\mathcal{F}}$ entails $(2/3)n^3+O\left(n^2\right)$ arithmetic operations.

Proof. 

The operation count in the loop is dominated by the update of matrix ${\mathbf{A}}_{S_k^c{S}_k^c}^{S_k}$ at each step k, taking $2{(n-k)}^2$ arithmetic operations. This yields the total operation count as stated. □

5.1. Computing the Initial Tableau

We next address computation of the initial tableau, which corresponds to the ∅-active BFS, in a form that is numerically stable and that applies both to the discounted and the average criterion to be addressed in Section 6. The tableaux for the average criterion arise as limits letting the discount rate $\alpha$ vanish in the discounted tableaux.

Note that (cf. (24))

$${\mathbf{B}}^{\varnothing }={(\mathbf{I}-{\mathsf{\Psi }}^0)}^{\mathsf{T}},{\mathbf{N}}^{\varnothing }={(\mathbf{I}-{\mathsf{\Psi }}^1)}^{\mathsf{T}},{\mathbf{H}}^{\varnothing }={\left({\mathbf{B}}^{\varnothing }\right)}^{-1},{\mathbf{A}}^{\varnothing }={\mathbf{H}}^{\varnothing }{\mathbf{N}}^{\varnothing }.$$

(28)

Thus, a straightforward approach to computing ${\mathbf{A}}^{\varnothing }$ is to solve the linear equation system

$${\left({\mathbf{A}}^{\varnothing }\right)}^{\mathsf{T}}(\mathbf{I}-{\mathsf{\Psi }}^0)=(\mathbf{I}-{\mathsf{\Psi }}^1).$$

(29)

However, this has a disadvantage: as $\alpha$ vanishes, the matrices $\mathbf{I}-{\mathsf{\Psi }}^a$ become increasingly ill-conditioned, as they are singular for $\alpha =0$—because they converge to $\mathbf{I}-{\mathbf{P}}^a$ where ${\mathbf{P}}^a\triangleq \left(p_{ij}^a\right)$.

To overcome such a drawback, we draw on the identity $(\mathbf{I}-{\mathsf{\Psi }}^a)\mathbf{1}=\mathbf{1}-{\mathsf{\Psi }}^a$, which is a consequence from (2). From this and (28), we have

$${\left({\mathbf{A}}^{\varnothing }\right)}^{\mathsf{T}}(\mathbf{1}-{\mathsf{\Psi }}^0)=\mathbf{1}-{\mathsf{\Psi }}^1.$$

The latter identity gives a useful counterpart as $\alpha \searrow 0$. Thus, writing as ${\xi }_i^a$ the length of an $(i,a)$-stage (cf. Section 3.1), and applying the MacLaurin series

$${\psi }_i^a=\mathbb{E}\left[e^{-\alpha {\xi }_i^a}\right]=1-\alpha m_i^a+O\left({\alpha }^2\right)\mathrm{as}\alpha \searrow 0,$$

one obtains in the limit

$${\left({\mathbf{A}}^{\varnothing }\right)}^{\mathsf{T}}{\mathbf{m}}^0={\mathbf{m}}^1,$$

with $m_i^a$ the mean length of an $(i,a)$-stage and $\mathbf{m}={\left(m_i^a\right)}_{i\in \mathcal{N}}$.

We thus obtain the following approach to computing the initial tableau, for $\alpha ⩾0$. Letting

$${\tilde{m}}_i^a\triangleq \left\{\begin{array}{cc}{m}_i^a\hfill & \mathrm{if}\alpha =0\hfill \\ (1-{\psi }_i^a)/\alpha \hfill & \mathrm{if}\alpha >0,\hfill \end{array}\right.$$

and ${\tilde{\mathbf{m}}}^a={\left({\tilde{m}}_i^a\right)}_{i\in \mathcal{N}}$, pick any state $j^{*}\in \mathcal{N}$ and solve the (block) linear equation system

$${\left(\begin{array}{cc}{\mathbf{I}}_{\mathcal{N},\mathcal{N}\backslash \left\{j^{*}\right\}}-{\mathsf{\Psi }}_{\mathcal{N},\mathcal{N}\backslash \left\{j^{*}\right\}}^0& {\tilde{\mathbf{m}}}^0\end{array}\right)}^{\mathsf{T}}{\mathbf{A}}^{\varnothing }={\left(\begin{array}{cc}{\mathbf{I}}_{\mathcal{N},\mathcal{N}\backslash \left\{j^{*}\right\}}-{\mathsf{\Psi }}_{\mathcal{N},\mathcal{N}\backslash \left\{j^{*}\right\}}^1& {\tilde{\mathbf{m}}}^1\end{array}\right)}^{\mathsf{T}}$$

(30)

to obtain ${\mathbf{A}}^{\varnothing }$. Then, calculate initial reduced costs using (28) and Lemma 1 (d,e) as

$$\begin{array}{cc}\hfill {\mathbf{g}}^{\varnothing }& ={\mathbf{c}}^1-{\mathbf{c}}^0{\mathbf{A}}^{\varnothing }\hfill \\ \hfill {\mathbf{f}}^{\varnothing }& ={\mathbf{r}}^1-{\mathbf{r}}^0{\mathbf{A}}^{\varnothing }.\hfill \end{array}$$

(31)

6. Extension to the Average Criterion

In applications where the (long-run) average criterion is employed, one must consider the version of $\lambda$-price problem (1) based on the average reward and resource metrics given by

$$F_i^{\pi }\triangleq \underset{T\nearrow \infty }{lim\; inf}\frac{1}{T}{\mathbb{E}}_i^{\pi }\left[{\int }_0^T{R}_{X\left(t\right)}^{A\left(t\right)}dt\right]=\underset{K\nearrow \infty }{lim\; inf}\frac{1}{K}{\mathbb{E}}_i^{\pi }\left[\sum _{k=0}^K{r}_{X_k}^{a_k}\right],$$

(32)

and

$$G_i^{\pi }\triangleq \underset{T\nearrow \infty }{lim\; sup}\frac{1}{T}{\mathbb{E}}_i^{\pi }\left[{\int }_0^T{Q}_{X\left(t\right)}^{A\left(t\right)}dt\right]=\underset{K\nearrow \infty }{lim\; sup}\frac{1}{K}{\mathbb{E}}_i^{\pi }\left[\sum _{k=0}^K{q}_{X_k}^{a_k}\right].$$

(33)

As in ([3], Section 6.5), we assume that the embedded process $X_k$ is communicating, so each state can be reached from every other state under some stationary policy. Under this assumption, the above metrics do not depend on the initial state under stationary deterministic policies, so one can write $F^S$ and $G^S$ for active sets $S\subseteq N$. This allows a straightforward extension of the above indexability theory to the average criterion.

Regarding the algorithms discussed above, they apply without change to the average criterion, as the results presented in Section 5.1 show that the corresponding tableaux are simply the limits of the discounted tableaux as the discount rate goes to zero, and also outline how to evaluate the initial tableau. To properly extend the results, one must further assume that the active-set family $\mathcal{F}$ satisfies that, for every $S\in \mathcal{F}$, the S-active policy is unichain, so it induces a single recurrent class on the embedded process $X_k$ plus a class of transient states, which may be empty.

7. Numerical Experiments

This section discusses results of numerical experiments, based on implementations by the author of the aforementioned algorithms.

Comparing Runtimes of Index Algorithms

The runtime performance of an algorithm depends not only on its arithmetic operation count, but also on its memory-access patterns, which can actually be the dominant factor. Hence, to compare the algorithms considered herein, a numerical study has been conducted, using MATLAB implementations developed by the author. The experiments were run on a PC with an Intel Core i7-8700 CPU at 3.2 GHz with 16 GB of RAM using MATLAB 2020b under Windows 10 Enterprise. For the state space sizes $n=1000,2000,\dots ,$ 15,000, a discrete-time restless bandit instance was randomly constructed. Transition matrices were generated from random matrices with Uniform$[0,1]$ entries, dividing each row by its sum. Active rewards were randomly generated with Uniform$[0,1]$ entries, and passive rewards were zero. The discount factor was $\beta =0.8$.

For each generated instance, the CP algorithm was first used to test for indexability and for PCL-indexability (by checking the signs of marginal resource metrics for the nested active-set family obtained). Since such tests were positive in each instance, the Whittle index was calculated using the CP, RP, and FP algorithms (taking $\mathcal{F}=2^{\mathcal{N}}$ in the latter).

Table 6. Runtimes (s) of index algorithms.

n	FP	RP	CP
1000	$1.8$	$2.6$	$2.2$
2000	$14.8$	$23.6$	$19.6$
3000	$53.3$	$77.1$	$67.8$
4000	$121.4$	$193.4$	$156.3$
5000	$227.6$	$342.8$	$295.9$
6000	$433.2$	$647.9$	$541.8$
7000	$699.6$	$1034.8$	$862.3$
8000	$1118.3$	$1531.5$	$1280.9$
9000	$1530.8$	$2173.6$	$1822.3$
10,000	$2100.7$	$2919.3$	$2500.9$
11,000	$2687.2$	$3580.0$	$3277.7$
12,000	$3575.9$	$5055.7$	$4300.6$
13,000	$4747.4$	$6610.4$	$5539.2$
14,000	$5629.5$	$7923.1$	$6829.9$
15,000	$7254.1$	$8871.0$	$8250.5$

records the runtimes for the loop of each algorithm, without counting the initialization stage of computing the initial tableau, while Figure 1 plots them, along with cubic least-squares fitting curves. These results highlight that the FP algorithm, whose loop operation count is of $(2/3)n^3+O\left(n^2\right)$, is indeed the fastest algorithm, followed by the CP and RP algorithms. Recall that $2n^3+O\left(n^2\right)$ and $n^3+O\left(n^2\right)$ are the loop operation counts for the CP and the RP algorithms.

The observed discrepancies between operation count complexity and actual runtime performance are explained by taking into account the memory-access patterns of the algorithms. Algorithm CP, which carries out conventional pivot steps, has efficient memory-access patterns, because the coefficient matrix $\mathbf{A}$ is always updated as a memory block of contiguous storage. Yet, both algorithms RP and FP achieve a reduction in operations by operating on submatrices of $\mathbf{A}$, with resulting noncontiguous memory-access patterns that are time-consuming. However, for the FP algorithm, as the decrease in arithmetic operations is substantial, it compensates such inefficiencies, and comes out in practice as the fastest algorithm.

8. Discussion

This paper has presented a new algorithm for computing the Whittle index of a general finite-state semi-Markov restless bandit, based on an efficient implementation of the adaptive-greedy algorithmic scheme introduced in [3,24] for restless bandits, in which it was not specified how to evaluate certain metrics arising in the algorithm description. The algorithm extends to restless bandits the fast-pivoting implementation developed in [40] for classic (non-restless) bandits, and results from a similar approach, exploiting the structure of parametric simplex tableaux to reduce the operation count down to $(2/3)n^3+O\left(n^2\right)$—apart from the computational effort to compute the initial tableau. It is unlikely that such a complexity count can be improved for general restless bandits, as $(2/3)n^3+O\left(n^2\right)$ is the complexity of Gaussian elimination for solving an $n\times n$ linear equation system, and it is shown in [3,24] that computing the Whittle index entails at least solving an equation system with such dimensions (albeit with an ordering of the states that generally is not known in advance).

The complexity of the proposed fast-pivoting algorithm is the best that has been reported in the literature. In contrast, the Whittle index algorithm recently presented in [41]—for the average criterion—has a complexity of $O\left(n^4{2}^n\right)$, which reduces to $O\left(n^5\right)$ for one-dimensional indexable bandits provided it is known that threshold policies are optimal for them.

The new algorithm presented herein, whose implementation is straightforward, will be most useful for computing the Whittle index in complex models with large-scale, multi-dimensional state spaces for which closed index formulae cannot be derived, and an efficient computational approach is needed. Given the explosion of research interest in restless bandit models in the last decade, the proposed algorithm thus has the potential of becoming a useful tool, allowing researchers to expand the scope of Whittle’s index policy to large-scale complex models.

Future research directions include developing efficient implementations with substantially lower complexity by exploiting the special structure of relevant model classes arising in applications, and testing the algorithm in large-scale real world models with real data. Another avenue of research is to develop software implementations that improve the efficiency of the computationally costly block matrix operations required by the fast-pivoting algorithm.

9. Conclusions

To conclude, the findings of this paper can be summarized as follows:

-

A new algorithm to compute the Whittle index of a general n-state semi-Markov restless bandit is presented, which can also be used to test numerically for indexability. After an initialization step, the algorithm computes the n index values in an n-step loop with a complexity of $(2/3)n^3+O\left(n^2\right)$ arithmetic operations.

-

The algorithm extends to Whittle’s index the fast-pivoting $(2/3)n^3+O\left(n^2\right)$ algorithm introduced by the author in [30] for the Gittins index of classic (non-restless) banditss, which also has the lowest complexity to date.

-

The proposed algorithm has substantially better complexity than alternative algorithms proposed in the literature.

-

The algorithm will be especially useful for computing the Whittle index in large-scale multi-dimensional models where the index cannot be derived in closed form and alternative algorithms will result in prohibitive computation times.