Generalized Quantum Convolution for Multidimensional Data

Abstract

The convolution operation plays a vital role in a wide range of critical algorithms across various domains, such as digital image processing, convolutional neural networks, and quantum machine learning. In existing implementations, particularly in quantum neural networks, convolution operations are usually approximated by the application of filters with data strides that are equal to the filter window sizes. One challenge with these implementations is preserving the spatial and temporal localities of the input features, specifically for data with higher dimensions. In addition, the deep circuits required to perform quantum convolution with a unity stride, especially for multidimensional data, increase the risk of violating decoherence constraints. In this work, we propose depth-optimized circuits for performing generalized multidimensional quantum convolution operations with unity stride targeting applications that process data with high dimensions, such as hyperspectral imagery and remote sensing. We experimentally evaluate and demonstrate the applicability of the proposed techniques by using real-world, high-resolution, multidimensional image data on a state-of-the-art quantum simulator from IBM Quantum.

1. Introduction

Convolution is a common operation that is leveraged in a wide variety of practical applications, such as signal processing [1], image processing [2], and most recently, convolutional neural networks [3]. However, leveraging the widespread utility of convolution operations in quantum algorithms is limited by the lack of a systematic, generalized implementation of quantum convolution. Specifically, contemporary quantum circuits for performing quantum convolution with a given filter are designed on a case-by-case basis [4,5,6,7,8]. In other words, implementing a novel convolution filter on a quantum computer is arduous and time consuming, requiring substantial human effort. Such a workflow is impractical for applications, such as quantum convolutional neural networks, which require a generalized, parameterized quantum circuit to iteratively test thousands of unique filters per training cycle.

In this work, we propose a generalizable algorithm for quantum convolution compatible with amplitude-encoded multidimensional data that is able to implement arbitrary multidimensional filters. Furthermore, our proposed technique implements unity stride, which is essential for capturing the totality of local features in input data. We experimentally verify our technique by applying multiple filters on high-resolution, multidimensional images and report the fidelity of the quantum results against the classically computed expectations. The quantum circuits are implemented on a state-of-the-art quantum simulator from IBM Quantum [9] in both noise-free (as a statevector) and noisy environments. Compared to classical CPU- and GPU-based implementations of convolution, we achieve an exponential improvement in time complexity with respect to data size. Additionally, when compared to existing quantum implementations, we achieve improved circuit depth complexity when factoring in the data encoding.

The work is structured as follows. In Section 2, we cover important background information and review the related work. In Section 3, we introduce the proposed quantum convolution circuits and provide analyses of the corresponding circuit depth. In Section 4, we present the experimental setup and results, while in Section 5, we provide discussions of the results and comparisons to related work. Finally, in Section 6, we present our conclusions and potential avenues for future explorations and extensions.

2. Background

In this section, we discuss related work pertinent to quantum convolution. Quantum operations that are relevant to convolution, such as quantum data encoding, and quantum shift operation, are also presented.

2.1. Related Work

Classically, the convolution operation is implemented either directly or by leveraging fast Fourier transform (FFT). On CPUs, the direct implementation has a time complexity of $\mathcal{O}\left(N^2\right)$ [10], where N is the data size, while FFT-based implementation has a time complexity of $\mathcal{O}(NlogN)$ [10]. On GPUs, the FFT-based implementation has a similar $\mathcal{O}(NlogN)$ complexity [11]. It is also common to take advantage of the parallelism offered by GPUs to implement convolution using general matrix multiplications (GEMMs) with $\mathcal{O}\left(N_{F}N\right)$ FLOPS [12,13], where $N_F$ is the filter size.

Techniques for performing quantum convolution have previously been reported [4,5,6,7,8]. However, these techniques only use fixed sizes of filter windows for specific filters, e.g., edge detection [4,5,6,7,8]. We will denote such methods as fixed-filter quantum convolution. Reportedly, these methods possess a quadratic circuit depth complexity of $\mathcal{O}\left(n^2\right)$ in terms of the number of qubits $n=\lceil {log}_{2}N\rceil$, where N is the size of the input data [4,5,6,7,8]. Because the shortest execution time of classical convolution is in the order of $\mathcal{O}\left(N\right)$ or $\mathcal{O}\left(2^n\right)$ [12,13] with respect to data size N, authors of the quantum counterparts often claim a quantum advantage [4]. However, the reported depth complexity of fixed-filter quantum convolution does not include the unavoidable overhead of data encoding. Furthermore, there does not exist, to the best of our knowledge, a method for performing generalized, multidimensional quantum convolution.

In reported related work [4,5,6,7,8], data encoding is performed with either the flexible representation of quantum images (FRQI) [14] or novel enhanced quantum representation (NEQR) [15] methods. In these encoding techniques, positional information is stored in the basis quantum states of n qubits, while color information is stored via angle encoding and basis encoding for FRQI and NEQR, respectively. FRQI and NEQR require a total of $n+1$ and $n+q$ qubits, respectively, where q is the number of qubits used to represent color values, e.g., $q=8$ for standard grayscale pixel representation. The reported circuit depth complexities of FRQI and NEQR are $\mathcal{O}\left(4^n\right)$ and $\mathcal{O}\left(qn{2}^n\right)$, respectively. When factoring in the depth complexities of either data-encoding technique, it is evident that the referenced fixed-filter quantum convolution techniques should be expected to perform worse than classical implementations.

In [16], the authors propose a method of edge detection based on amplitude encoding and the quantum wavelet transform (QWT), which they denote as quantum Hadamard edge detection (QHED). Although the work utilizes grayscale two-dimensional images, the QHED technique is highly customized for those data and does not easily scale or generalize to data of higher dimensions, such as colored and/or multispectral images. For example, the quantum discriminator operation in their technique is applied over all qubits in the circuit, without distinguishing between qubits representing each dimension, i.e., image rows or columns. Such a procedure not only forgoes parallelism and increases circuit depth but inhibits the algorithm’s ability to be generalized beyond capturing one-dimensional features.

In our proposed work, we achieve an exponential improvement in time complexity compared to classical implementations of convolution with respect to data size. Additionally, when compared to existing quantum convolution implementations, we achieve improved circuit depth complexity when factoring in the data encoding. The contribution of our work is analyzed, experimentally verified, and discussed in detail in Section 5.

2.2. Classical to Quantum (C2Q)

Our method of quantum convolution leverages amplitude encoding, which encodes N data values directly in the complex probability amplitudes $c_i\in \mathbb{C}$ of the positional basis state $|i\rangle$ for an n-qubit state $|\psi \rangle$, where $n=\lceil {log}_{2}N\rceil$ and $0\le i<n$, see (1):

$$|\psi \rangle =\sum _{i=0}^{2^n-1}{c}_i|i\rangle :c_i\in \mathbb{C}.$$

(1)

We use the classical-to-quantum (C2Q) [17] data-encoding technique to encode the amplitude encoded state $|{\psi }_0\rangle$ from the ground state $|0{\rangle }^{\otimes n}$, see Figure 1 and (2). The C2Q operation $U_{\mathrm{C}2\mathrm{Q}}$ has a circuit depth complexity of $\mathcal{O}\left(2^n\right)$, a quadratic and linear improvement over FRQI and NEQR, respectively:

$$\begin{array}{cc}\hfill U_{\mathrm{C}2\mathrm{Q}}|0{\rangle }^{\otimes n}& =|{\psi }_0\rangle \hfill \\ \hfill U_{\mathrm{C}2\mathrm{Q}}& =\left[\begin{array}{cc}|{\psi }_0\rangle & |\times \rangle \cdots |\times \rangle \end{array}\right],\mathrm{where}\hfill \\ \hfill |\times \rangle & =“\mathrm{don}’\mathrm{t}\mathrm{care}”\hfill \end{array}$$

(2)

2.3. Quantum Shift Operation

A fundamental operation for quantum convolution is the quantum shift operation, denoted in this work as $U_{\mathrm{shift}}^k$, which shifts the basis states of the state vector by k positions when applied to an m-qubit state $|\psi \rangle$, see (3). The quantum shift operation is critical for performing the cyclic rotations needed to prepare strided windows when performing convolution. It is also common for the operation to be described as a quantum incrementer when $k>0$, see Figure 2a, and a quantum decrementer when $k<0$ [16,18], see Figure 2b:

$$U_{\mathrm{shift}}^k|\psi \rangle =\sum _{i=0}^{2^m-1}{c}_i|j\rangle ,\mathrm{where}j=(i-k)mod{2}^m$$

(3)

3. Materials and Methods

In general, a convolution operation can be performed using a sequence of shift and multiply-and-accumulate operations. In our proposed methods, we implement the generalized convolution operations as follows:

• Shift: Auxiliary filter qubits and controlled quantum decrementers are used to create shifted (unity-strided) replicas of input data.
• Multiply-and-accumulate: Arbitrary state synthesis and classical-to-quantum (C2Q) encoding are applied to create generic multidimensional filters.
• Data rearrangement: Quantum permutation operations are applied to restructure the fragmented data into one contiguous output datum.

In Section 3.1, we present our quantum convolution technique in detail for one-dimensional data. In the following sections, we illustrate optimizations to improve circuit depth and generalize our method for multidimensional data. For evaluating our proposed methods, we used real-world, high-resolution, black-and-white (B/W) and RGB images, ranging in a resolution from $(8\times 8)$ pixels to $(512\times 512)$ pixels and $(8\times 8\times 3)$ pixels to $(512\times 512\times 3)$ pixels, respectively. We also performed experiments on 1-D real-world audio data and 3-D real-world hyperspectral data to demonstrate our method’s applicability to data and filters of any dimensionality. Further details about our experimental setup and dataset can be found in Section 4.

3.1. Quantum Convolution for One-Dimensional Data

The proposed structure of quantum convolution for one-dimensional (1-D) data is shown in Figure 3. The following sections show the details of the five steps of the convolution operation procedure to transform the initial encoded data $|{\psi }_0\rangle$ to the final state $|{\psi }_5\rangle$, see Figure 3.

3.1.1. Shift Operation

To perform convolution with unity stride with a filter of size $N_f$ terms, $N_f$ replicas of the input data must be made, strided for $0\le k<N_f$. To store these replicas, we add $n_f=\lceil {log}_2{N}_f\rceil$ auxiliary qubits, which we denote as “filter qubits”, to the most significant positions of the initial quantum state $|{\psi }_0\rangle$, see (4) and Figure 3:

$$|{\psi }_1\rangle =|0{\rangle }^{\otimes n_f}\otimes |{\psi }_0\rangle =\left[\begin{array}{c}|{\psi }_0\rangle \\ 0\\ ⋮\\ 0\end{array}\right]\begin{array}{c}\hfill \phantom{|{\psi }_0\rangle }↕2^n\\ \hfill \phantom{\begin{array}{c}0\\ ⋮\\ 0\end{array}}\end{array}↕2^{n+n_f}$$

(4)

Placing the filter qubits in superposition using Hadamard (H) gates creates $2^{n_f}$ identical replicas of the initial data $|{\psi }_0\rangle$, as shown in (5):

$$\begin{array}{c}\hfill |{\psi }_2\rangle =\left(H^{\otimes n_f}\otimes I^{\otimes n}\right)|{\psi }_1\rangle =\frac{1}{\sqrt{2^{n_f}}}\left[\begin{array}{c}|{\psi }_0\rangle \\ ⋮\\ |{\psi }_0\rangle \end{array}\right]\begin{array}{c}\hfill \phantom{|{\psi }_0\rangle }↕2^n\\ ⋮\\ \hfill \phantom{|{\psi }_0\rangle }↕2^n\end{array}↕2^{n+n_f}\end{array}$$

(5)

Finally, multiplexed quantum shift operations can be used to generate the strided replicas, see (6):

$$\begin{array}{cc}\hfill |{\psi }_3\rangle & =U_{\mathrm{mux}}|{\psi }_2\rangle =\frac{1}{\sqrt{2^{n_f}}}\left[\begin{array}{c}{U}_{\mathrm{shift}}^0|{\psi }_0\rangle \\ ⋮\\ U_{\mathrm{shift}}^{-(2^{n_f}-1)}|{\psi }_0\rangle \end{array}\right]\begin{array}{c}\hfill \phantom{U_{\mathrm{shift}}^0|{\psi }_0\rangle }↕2^n\\ ⋮\\ \hfill \phantom{U_{\mathrm{shift}}^{-(2^{n_f}-1)}|{\psi }_0\rangle }↕2^n\end{array}↕2^{n+n_f},\mathrm{where}\hfill \\ \hfill U_{\mathrm{mux}}& =\left[\begin{array}{ccc}{U}_{\mathrm{shift}}^0& & \\ & \ddots & \\ & & U_{\mathrm{shift}}^{-(2^{n_f}-1)}\end{array}\right]\hfill \end{array}$$

(6)

3.1.2. Multiply-and-Accumulate Operation

For the traditional convolution operation, applying a filter $F\in {\mathbb{R}}^{N_f}$ to an array of data $W\in {\mathbb{R}}^{N_f}$ produces a scalar output $x\in \mathbb{R}$, which can be expressed as $x=F^{T}W$. In the quantum domain, we can represent an array of data as the partial state $|\varphi \rangle$ and the normalized filter as $|F\rangle$. Accordingly, the output state can be expressed as shown in (7):

$$\begin{array}{cc}\hfill |{\psi }_{\mathrm{out}}\rangle & =\sum _{i=0}^{2^n-1}\langle F|{\varphi }_i\rangle ·|i\rangle ,\mathrm{where}\hfill \\ \hfill |{\varphi }_i\rangle & =\sum _{j=0}^{2^{n_f}-1}\langle k^{\prime }|{\psi }_3\rangle ·|k^{\prime }\rangle ,\mathrm{and}{k}^{\prime }=(2^{n_f}·i)+j\hfill \end{array}$$

(7)

To calculate $|{\psi }_4\rangle$ from $|{\psi }_3\rangle$, it is necessary to embed 〈F| into a unitary operation $U_F$ as shown in (8). Since 〈F| is a normalized row vector, we can define $U_F$ as a matrix such that its first row is 〈F| and the remaining rows are arbitrarily determined to preserve the unitariness of $U_F$ such that $U_F^{†}{U}_F=U_F{U}_F^{†}=I^{\otimes n_f}$. From (2), we can realize $U_F$ using an inverse C2Q operation, see (8):

$$\begin{array}{cc}\hfill |{\psi }_4\rangle =\left(I^{\otimes n}\otimes U_F\right)|{\psi }_3\rangle & =\left[\begin{array}{c}{U}_F|{\varphi }_0\rangle \\ ⋮\\ U_F|{\varphi }_{2^n-1}\rangle \end{array}\right]\begin{array}{c}\hfill \phantom{U_F|{\varphi }_0\rangle }↕2^{n_f}\\ ⋮\\ \hfill \phantom{U_F|{\varphi }_{2^n\rangle -1}}↕2^{n_f}\end{array}↕2^{n+n_f},\mathrm{where}\hfill \\ \hfill U_F& =\left[\begin{array}{c}\langle F|\\ \langle \times |\\ ⋮\\ \langle \times |\end{array}\right]↕2^{n_f}=U_{\mathrm{C}2\mathrm{Q}}^{†}\hfill \end{array}$$

(8)

3.1.3. Data Rearrangement

As of $|{\psi }_4\rangle$, the desired values of $|{\psi }_{\mathrm{out}}\rangle$ are fragmented among undesired/extraneous values, which we denote using the symbol “×”. We apply SWAP permutations to rearrange and coalesce our desired values to be contiguous in the final statevector $|{\psi }_5\rangle$, see (9) and Figure 3:

$$\begin{array}{cc}\hfill |{\psi }_5\rangle & =U_{\mathrm{SWAP}}^{1-D}|{\psi }_4\rangle \hfill \\ \hfill & =\left[\begin{array}{c}\langle F|{\varphi }_0\rangle \\ ⋮\\ \langle F|{\varphi }_{2^n-1}\rangle \\ \times \\ ⋮\\ \times \end{array}\right]\begin{array}{c}\hfill \phantom{\begin{array}{c}F{\varphi }_0\\ ⋮\\ F{\varphi }_{2^n-1}\end{array}}↕2^n\\ \hfill \phantom{\begin{array}{c}\times \\ ⋮\\ \times \end{array}}\end{array}↕2^{n+n_f}\hfill \\ \hfill & =\left[\begin{array}{c}|{\psi }_{out}\rangle \\ \times \\ ⋮\\ \times \end{array}\right]\begin{array}{c}\hfill \phantom{|{\psi }_{out}\rangle }↕2^n\\ \hfill \phantom{\begin{array}{c}\times \\ ⋮\\ \times \end{array}}\end{array}↕2^{n+n_f},\mathrm{where}\hfill \\ \hfill U_{\mathrm{SWAP}}^{1-D}& =\prod _{j=n_f-1}^0\left(I^{\otimes (n_f-1-j)}\otimes \mathrm{SWAP}\left(j,j+n\right)\otimes I^{\otimes j}\right)\hfill \end{array}$$

(9)

3.1.4. Circuit Depth Analysis of 1-D Quantum Convolution

When considering the circuit depth complexity of the proposed 1-D quantum convolution technique, it is evident from Figure 3 that the operations described by (5) and (9) are performed using parallel Hadamard and SWAP operations, respectively, and thus are of constant depth complexity, i.e., $\mathcal{O}\left(1\right)$. In contrast, the $U_{\mathrm{mux}}$ and $U_F$ operations incur the largest circuit depth, as they are both serial operations that scale with the data size and/or filter size, see Figure 3.

For the implementation of $U_{\mathrm{mux}}$ in Figure 3, there are a total of $2^{n_f}$ controlled quantum shift operations, where the i-th shift operation is a quantum decrementer $U_{\mathrm{shift}}^{-i}={\left(U_{\mathrm{shift}}^{-1}\right)}^i$. Since all the $U_{\mathrm{shift}}^{-1}$ operations are performed in series, the circuit depth of $U_{\mathrm{mux}}$ depends on the total number of unity quantum decrementers, $N_{U_{\mathrm{shift}}^{-1}}$, see (10):

$$\begin{array}{cc}\hfill N_{U_{\mathrm{shift}}^{-1}}\left(n_f\right)& =\sum _{i=0}^{2^{n_f}-1}i=\frac{2^{n_f}(2^{n_f}-1)}{2}=\frac{4^{n_f}}{2}-2^{n_f-1}\hfill \end{array}$$

(10)

As shown in Figure 2b, each quantum decrementer $U_{\mathrm{shift}}^{-1}$ acting on m qubits can be realized using m multi-controlled CNOT (MCX) gates, where the i-th MCX gate is controlled by i qubits and $0\le i<m$. Accordingly, for each quantum decrementer $U_{\mathrm{shift}}^{-1}$ that is controlled by c qubits, its i-th MCX gate is controlled by a total of $i+c$ qubits. Therefore, the depth of the quantum decrementer circuits can be expressed in terms of the MCX gate count as shown in (11):

$$\begin{array}{cc}\hfill {\mathcal{D}}_{U_{\mathrm{shift}}^{-1}}(m,c)& =\sum _{i=c}^{m+c-1}{\mathcal{D}}_{\mathrm{MCX}}\left(i\right)\hfill \end{array}$$

(11)

The depth of an MCX gate with a total of m qubits can be expressed with a linear function in terms of fundamental single-qubit rotation gates and CNOT gates [19] as shown in (12), where $\alpha$ represents the first-order coefficient and $\beta$ represents the constant bias term. Thus, the depth complexity of $U_{\mathrm{shift}}^{-1}$ can be expressed as shown in (13):

$$\begin{array}{cc}\hfill {\mathcal{D}}_{\mathrm{MCX}}\left(m\right)& =\alpha m+\beta :\alpha ,\beta \in \mathbb{R}\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill {\mathcal{D}}_{U_{\mathrm{shift}}^{-1}}(m,c)& =\sum _{i=0}^{m-1}\alpha (i+c)+\beta \hfill \\ \hfill & =\frac{\alpha }{2}{m}^2+\left(\alpha \left(c-\frac{1}{2}\right)+\beta \right)m\hfill \\ \hfill & =\mathcal{O}\left(m^2\right)\hfill \end{array}$$

(13)

To derive the circuit depth complexity of $U_{\mathrm{mux}}$, we leverage the definitions of $N_{U_{\mathrm{shift}}^{-1}}\left(n_f\right)$ and ${\mathcal{D}}_{U_{\mathrm{shift}}^{-1}}(m,c)$ as shown in (14), where $m=n$ and $c=n_f$:

$$\begin{array}{cc}\hfill {\mathcal{D}}_{U_{\mathrm{mux}}}(n,n_f)& =N_{U_{\mathrm{shift}}^{-1}}\left(n_f\right)·{\mathcal{D}}_{U_{\mathrm{shift}}^{-1}}(n,n_f)\hfill \\ \hfill & =\left(\frac{4^{n_f}}{2}-2^{n_f-1}\right)·\left(\frac{\alpha }{2}{n}^2+\left(\alpha \left(n_f-\frac{1}{2}\right)+\beta \right)n\right)\hfill \\ \hfill & =\left(4^{n_f-1}-2^{n_f-2}\right)·\left(\alpha n^2+2\alpha n_{f}n-(\alpha -2\beta )n\right)\hfill \\ \hfill & =\mathcal{O}\left(4^{n_f}{n}^2+4^{n_f}{n}_{f}n\right)\hfill \end{array}$$

(14)

As discussed in Section 3.1.2, we implement the filter operation $U_F$ by leveraging the C2Q arbitrary synthesis operation [17]. Although C2Q incurs a circuit depth of exponential complexity in terms of fundamental quantum gates, as shown in (15), $U_F$ is only applied to $n_f$ qubits, a small subset of the total number of qubits, which somewhat mitigates the circuit depth. Furthermore, in most practical scenarios, the dimensions of the filter are typically much smaller than the dimensions of the input data, i.e., $n_f\ll n$. As a result, $U_F$ should not incur overly large circuit depth relative to other circuit components, e.g., $U_{\mathrm{mux}}$. Altogether, the overall circuit depth complexity of the 1-D quantum convolution operation can be expressed according to (16):

$${\mathcal{D}}_{U_F}\left(n_f\right)=\mathcal{O}\left(2^{n_f}\right)$$

(15)

$${\mathcal{D}}_{1-\mathrm{D}\mathrm{conv}}(n,n_f)=\mathcal{O}\left(4^{n_f}{n}^2+4^{n_f}{n}_{f}n+2^{n_f}\right),\mathrm{where}n\gg n_f$$

(16)

3.2. Depth-Optimized 1-D Quantum Convolution

In Figure 4, we present an optimized implementation of $U_{\mathrm{mux}}$ that greatly reduces the circuit depth.

In Section 3.1, we implemented $U_{\mathrm{mux}}$ with $2^{n_f}$ controlled quantum decrementers $U_{\mathrm{shift}}^{-k}$, where $0\le k<2^{n_f}$. We can represent each k in binary notation, as shown in (17), to express $U_{\mathrm{shift}}^{-k}$ as a sequence of controlled shift operations by powers of 2. As shown in (18), we can denote such operations with the notation $U_{\mathrm{shift}}^{-b_j{2}^j}\left(n\right)$, where $0\le j<n_f$, and $\left(n\right)$ reflects that the shift operation is applied to an n-qubit state.

$$\begin{array}{cc}\hfill k& ={\left(b_{n_f-1}{b}_{n_f-2}\cdots b_j\cdots b_1{b}_0\right)}_2=\sum _{j=0}^{n_f-1}{b}_j{2}^j:b_j\in \{0,1\}\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill U_{\mathrm{shift}}^{-k}\left(n\right)& =U_{\mathrm{shift}}^{-{\sum }_{j=0}^{n_f-1}{b}_j{2}^j}\left(n\right)=\prod _{j=0}^{n_f-1}{U}_{\mathrm{shift}}^{-b_j{2}^j}\left(n\right)\hfill \end{array}$$

(18)

The binary decomposition of the uniformly controlled $U_{\mathrm{shift}}^{-k}$ operations is conducive to several circuit depth optimizations. As shown in (19), the value of $b_j$ is dependent only on the state of the j-th filter qubit $q_{n+j}$. In other words, each $U_{\mathrm{shift}}^{-2^j}\left(n\right)$ can only be controlled by one qubit $q_{n+j}$, independently from the other control qubits. Accordingly, it is possible to coalesce the multiplexed $U_{\mathrm{shift}}^{-2^j}\left(n\right)$ operations across the k control indices. The resultant implementation of $U_{\mathrm{mux}}$, therefore, becomes a sequence of $2^{n_f}$ single-controlled $U_{\mathrm{shift}}^{-2^j}\left(n\right)$ operations, where $0\le j<n_f$, which comparatively has a smaller circuit depth by a factor of $2^{n_f}$. Furthermore, each $U_{\mathrm{shift}}^{-2^j}\left(n\right)$ operation can be implemented using a single $U_{\mathrm{shift}}^{-1}(n-j)$ operation in lieu of sequential $U_{\mathrm{shift}}^{-1}\left(n\right)$ operations, see (20) and Figure 4, further reducing the depth by a factor of $2^j$ per operation:

$$b_j=\left\{\begin{array}{cc}1,\hfill & |q_{n+j}\rangle =|1\rangle \hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.,\forall k\in [0,2^{n_f}]$$

(19)

$$U_{\mathrm{shift}}^{-2^j}\left(n\right)\equiv \prod _{2^j}^{}{U}_{\mathrm{shift}}^{-1}\left(n\right)=U_{\mathrm{shift}}^{-1}(n-j)\otimes I^{\otimes j}$$

(20)

Circuit Depth Analysis of Optimized 1-D Quantum Convolution

With the aforementioned optimizations, the depth of the updated $U_{\mathrm{mux}}$ operation can be expressed in terms of ${\mathcal{D}}_{U_{\mathrm{shift}}^{-1}}(m,c)$ as described by (21), where $m=n-j$ and $c=1$ for all $0\le j<n_f$. In comparison with the depth complexity of the unoptimized $U_{\mathrm{mux}}$, see (14), the dominant term remains quadratic, i.e., $n^2$, in terms of the data qubits n. However, its coefficient is improved exponentially, from $4^{n_f}$ to $n_f$, see (14) and (21). Note that a cubic term $n_f^3$ in terms of the number of filter qubits is introduced in the optimized $U_{\mathrm{mux}}$ implementation, see (21). However, when considering the total depth for the optimized 1-D quantum convolution circuit ${\mathcal{D}}_{1-\mathrm{D}\mathrm{conv}}^{\mathrm{opt}}$, the $n_f^3$ term becomes negligible because of $U_F$, whose complexity $\mathcal{O}\left(2^{n_f}\right)$ is exponential in terms of the number of filter qubits, see (15), (21), and (22):

$$\begin{array}{cc}\hfill {\mathcal{D}}_{U_{\mathrm{mux}}}^{\mathrm{opt}}(n,n_f)& =\sum _{j=0}^{n_f-1}{\mathcal{D}}_{U_{\mathrm{shift}}^{-1}}(n-j,1)=\sum _{j=0}^{n_f-1}\left[\frac{\alpha }{2}{(n-j)}^2+\left(\frac{\alpha }{2}+\beta \right)(n-j)\right]\hfill \\ \hfill & =\frac{\alpha }{2}{n}_f{n}^2-\frac{\alpha }{2}{n}_f^{2}n+(\alpha +\beta )n_{f}n+\frac{\alpha }{6}{n}_f^3-\frac{\alpha +\beta }{2}{n}_f^2+\frac{2\alpha +3\beta }{6}{n}_f\hfill \\ \hfill & =\mathcal{O}\left(n_f{n}^2-n_f^{2}n+n_f^3\right),\mathrm{where}n\gg n_f\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill {\mathcal{D}}_{1-\mathrm{D}\mathrm{conv}}^{\mathrm{opt}}(n,n_f)& =\mathcal{O}\left(n_f{n}^2-n_f^{2}n+n_f^3+2^{n_f}\right)\hfill \\ \hfill & =\mathcal{O}\left(n_f{n}^2-n_f^{2}n+2^{n_f}\right),\mathrm{where}n\gg n_f\hfill \end{array}$$

(22)

3.3. Generalized Quantum Convolution for Multidimensional Data and Filters

In this section, we present the quantum circuit of our proposed quantum convolution technique generalized for multidimensional data and filters. Although quantum statevectors are one-dimensional, it is possible to map multidimensional data to a 1-D vector in either row- or column-major order. In this work, we represent multidimensional input data and convolutional filters in a quantum circuit in column-major order. In other words, for d-dimensional data of size $(N_0\times \cdots \times N_i\cdots \times N_{d-1})$ data values, the positional information of the i-th dimension is encoded in the ${\sum }_{j=0}^{i-1}{n}_j$ to $\left({\sum }_{j=0}^i{n}_j\right)-1$ qubits, where $n_i=\lceil {log}_2{N}_i\rceil$. Using this representation, the optimized 1-D quantum convolution circuit shown in Figure 4 can be generalized for d dimensions by “stacking” d 1-D circuits as shown in Figure 5.

The “stacked” quantum circuit in Figure 5 is based on the assumption that the overall (lumped) d-dimensional filter operator $U_F$ is separable and decomposable into d one-dimensional filters $U_{F_i}$ for $0\le i<d$. However, it would be more practically useful to generalize our multidimensional quantum convolution technique independently from the separability/decomposability of $U_F$. For this purpose, the identity in (23), which could be easily derived from either Figure 3 or Figure 4 for 1-D convolution, can be leveraged and generalized for multidimensional convolution circuits, see (24). The identity in (24) allows us to reverse the order of multiply-and-accumulate and data rearrangement steps and, therefore, generate one generic lumped $U_F$ that acts on the contiguous $n_f={\sum }_{i=0}^{d-1}\left(n_{f_i}\right)$ filter qubits, where $n_{f_i}$ is the number of qubits representing the filter dimension i for $0\le i<d$, see Figure 6. $U_F$ can be derived based on the given arbitrary multidimensional filter F using the method discussed in Section 3.1.2 when F is represented as a normalized 1-D vector $|F\rangle$ in a column major ordering:

$$U_{\mathrm{SWAP}}^{1-D}·\left(I^{\otimes n}\otimes U_F\right)=\left(U_F\otimes I^{\otimes n}\right)·U_{\mathrm{SWAP}}^{1-D}$$

(23)

$$\begin{array}{cc}\hfill U_{\mathrm{SWAP}}^{d-D}& ·\left(I^{\otimes n_f}\otimes \left(\underset{i=d-1}{\overset{0}{⨂}}\left[I^{\otimes (n_i-n_{f_i})}\otimes U_{F_i}\right]\right)\right)\begin{array}{cc}\hfill & \\ \hfill & =\left(\left(\underset{i=d-1}{\overset{0}{⨂}}\left[U_{F_i}\right]\right)\otimes I^{\otimes n}\right)·U_{\mathrm{SWAP}}^{d-D}\hfill \\ \hfill & =\left(U_F\otimes I^{\otimes n}\right)·U_{\mathrm{SWAP}}^{d-D},\mathrm{where}\hfill \end{array}\hfill \\ \hfill U_F& =\underset{i=d-1}{\overset{0}{⨂}}\left[U_{F_i}\right]\equiv U_{F_{d-1}}\otimes U_{F_{d-2}}\otimes \cdots \otimes U_{F_1}\otimes U_{F_0},\hfill \\ \hfill U_{\mathrm{SWAP}}^{d-D}& =\prod _{i=d-1}^0\prod _{j=n_{f_i}-1}^0\left(I^{\otimes \left(n_f-1-j-q_{f_i}\right)}\otimes \mathrm{SWAP}\left(j+q_i,j+n+q_{f_i}\right)\otimes I^{\otimes (j+q_i)}\right),\hfill \\ \hfill q_{f_i}& =\sum _{k=0}^{i-1}{n}_{f_k},q_i=\sum _{k=0}^{i-1}{n}_k,n_f=\sum _{k=0}^{d-1}{n}_{f_k},\mathrm{and}n=\sum _{k=0}^{d-1}{n}_k\hfill \end{array}$$

(24)

Circuit Depth Analysis of Generalized Multidimensional Quantum Convolution

As a result of the “stacked" structure, the data of all d dimensions could be concurrently processed in parallel. Consequently, the circuit depth of the multidimensional quantum circuit becomes dependent on the largest data dimension $N_{\max }=2^{n_{\max }}$, where $n_{\max }={max}_{i=0}^{d-1}\left(n_i\right)$, in lieu of the total data size N. The circuit component of the optimized 1-D circuit with the greatest depth contribution $U_{\mathrm{mux}}$ is performed in parallel on each dimension in the generic d-D circuit. Specifically, $U_{\mathrm{mux}}$ scales with the number of qubits used to represent the largest data dimension $n_{\max }=\lceil {log}_2{N}_{\max }\rceil$. Note that the parallelization across dimensions applies to the Hadamard and SWAP operations from (5) and (9), see Figure 6, and therefore these operations are of constant depth complexity, i.e., $\mathcal{O}\left(1\right)$. The depth complexity of the multidimensional $U_F$ operation is determined by the total number of elements in the filter $N_F$, and therefore the C2Q-based implementation of $U_F$ does not benefit from multidimensional stacking. Accordingly, the circuit depth of the generalized multidimensional quantum convolution operation could be derived from (22) and expressed in (25), where $n_{f_{\max }}={max}_{i=0}^{d-1}\left(n_{f_i}\right)$ is the number of qubits representing the maximum filter dimension $N_{f_{\max }}=2^{n_{f_{\max }}}$. It is worth mentioning that the generic multidimensional formula in (25) reduces to the 1-D formula in (22) when $n=n_{\max }$ and $n_f=n_{f_{\max }}$:

$$\begin{array}{cc}\hfill {\mathcal{D}}_{d\text{-}\mathrm{D}\mathrm{conv}}^{\mathrm{opt}}(n,n_f)& =\mathcal{O}\left(n_{f_{\max }}{n}_{\max }^2-n_{f_{\max }}^2{n}_{\max }+2^{n_f}\right),\mathrm{where}\hfill \\ \hfill n_{\max }& =\underset{i=0}{\overset{d-1}{max}}\left(n_i\right),n_{f_{\max }}=\underset{i=0}{\overset{d-1}{max}}\left(n_{f_i}\right),\mathrm{and}{n}_{\max }\gg n_{f_{\max }}\hfill \end{array}$$

(25)

4. Experimental Setup and Results

We experimentally demonstrate our proposed technique for generalized, multidimensional quantum convolution with unity stride on real-world, high-resolution, multidimensional image data, see Figure 7. By leveraging the Qiskit SDK (v0.39.4) from IBM Quantum [9], we simulate our quantum circuits in the following formats: (1) classically (to present the ideal/theoretical expectation), (2) noise-free (using statevector simulation), and (3) noisy (using 1,000,000 “shots” or circuit samples). Moreover, we present a quantitative comparison of the obtained results using fidelity [20] as a similarity metric between compared results $\rho$ and $\sigma$, see (26). Experiments were performed on a 16-core AMD EPYC 7302P CPU with frequencies up to 3.3 GHz, 128 MB of L3 cache, and access to 256 GB of DDR4 RAM. In our analysis, we evaluated the correctness of the proposed techniques by comparing classical results with noise-free results. We also evaluated the scalability of the proposed techniques for higher-resolution images by comparing the classical results with both the noise-free and noisy results. Finally, we plotted the circuit depth of our techniques with respect to the data size and filter size as shown in Figure 8.

$$\mathrm{Fidelity}(\rho ,\sigma )=\sqrt{\sqrt{\rho }·\sigma ·\sqrt{\rho }}$$

(26)

In our experiments, we evaluated our techniques using well-known $(3\times 3)$ and $(5\times 5)$ filters, i.e., Averaging $F_{\mathrm{avg}}$, Sobel edge-detection $F_{\mathrm{Sx}}$/$F_{\mathrm{Sy}}$, Gaussian blur $F_{\mathrm{blur}}$, and Laplacian of Gaussian blur (Laplacian) $F_{\mathcal{L}}$, see (27)–(30). We applied zero padding to maintain the size of the filter dimensions at a power of two for quantum implementation. In addition, we used wrapping to resolve the boundary conditions, and we restricted the magnitude of the output between $[0,255]$ to mitigate quantization errors in the classical domain:

$$\begin{array}{c}\hfill F_{\mathrm{avg}}^{3\times 3}=\frac{1}{9}\left[\begin{array}{ccc}1& 1& 1\\ 1& 1& 1\\ 1& 1& 1\end{array}\right],\end{array}\begin{array}{c}\hfill F_{\mathrm{avg}}^{5\times 5}=\frac{1}{25}\left[\begin{array}{ccccc}1& 1& 1& 1& 1\\ 1& 1& 1& 1& 1\\ 1& 1& 1& 1& 1\\ 1& 1& 1& 1& 1\\ 1& 1& 1& 1& 1\end{array}\right]\end{array}$$

(27)

$$\begin{array}{c}\hfill F_{\mathrm{blur}}^{3\times 3}=\frac{1}{16}\left[\begin{array}{ccc}1& 2& 1\\ 2& 4& 2\\ 1& 2& 1\end{array}\right],\end{array}\begin{array}{c}\hfill F_{\mathrm{blur}}^{5\times 5}=\frac{1}{273}\left[\begin{array}{ccccc}1& 4& 7& 4& 1\\ 4& 16& 26& 16& 4\\ 7& 26& 41& 26& 7\\ 4& 16& 26& 16& 4\\ 1& 4& 7& 4& 1\end{array}\right]\end{array}$$

(28)

$$\begin{array}{c}\hfill F_{\mathrm{Sx}}=\frac{1}{4}\left[\begin{array}{ccc}1& 0& -1\\ 2& 0& -2\\ 1& 0& -1\end{array}\right],\end{array}\begin{array}{c}\hfill F_{\mathrm{Sy}}=\frac{1}{4}\left[\begin{array}{ccc}1& 2& 1\\ 0& 0& 0\\ -1& -2& -1\end{array}\right]\end{array}$$

(29)

$$\begin{array}{c}\hfill F_{\mathcal{L}}^{3\times 3}=\frac{1}{6}\left[\begin{array}{ccc}1& 1& 1\\ 1& -8& 1\\ 1& 1& 1\end{array}\right],\end{array}\begin{array}{c}\hfill F_{\mathcal{L}}^{5\times 5}=\frac{1}{20}\left[\begin{array}{ccccc}1& 1& 1& 1& 1\\ 1& 1& 1& 1& 1\\ 1& 1& -24& 1& 1\\ 1& 1& 1& 1& 1\\ 1& 1& 1& 1& 1\end{array}\right]\end{array}$$

(30)

We applied 2-D convolution filters to black-and-white (B/W) and RGB images, see Figure 7, ranging in resolution from $(8\times 8)$ pixels to $(512\times 512)$ pixels and $(8\times 8\times 3)$ pixels to $(512\times 512\times 3)$ pixels, respectively. The number of filter qubits can be obtained by the size of filter dimensions, i.e., $n_f=\lceil {log}_{2}3\rceil +\lceil {log}_{2}3\rceil =4$ qubits for ($3\times 3$) filters and $n_f=\lceil {log}_{2}5\rceil +\lceil {log}_{2}5\rceil =6$ qubits for ($5\times 5$) filters. Therefore, our simulated quantum circuits ranged in size ($n+n_f$) from 10 qubits to 26 qubits. Figure 9 and Figure 10 present the reconstructed output images for classical, noise-free, and noisy experiments using $(128\times 128)$ and $(128\times 128\times 3)$-pixel input images, respectively.

To demonstrate our method’s applicability to data and filters of any dimensionality, we also performed experiments applying the 1-D and 3-D averaging filter to 1-D real-world audio data and 3-D real-world hyperspectral data, respectively. The audio files were sourced from the publicly available sound quality assessment material published by the European Broadcasting Union and modified to be single channel with data sizes ranging from $2^8$ values to $2^{20}$ values when sampled at 44.1 kHz [21]. Figure 11 and Figure 12 present the reconstructed output images and fidelity, respectively, from applying (3) and (5) averaging filters. The hyperspectral images were sourced from the Kennedy Space Center (KSC) dataset [22] and resized to range from ($8\times 8\times 8$) pixels to ($128\times 128\times 128$) pixels. Figure 13 and Figure 14 present the reconstructed output images and fidelity, respectively, from applying ($3\times 3\times 3$) and ($5\times 5\times 5$) averaging filters.

Comparison of the noise-free quantum results against the ideal classical results demonstrates a $100\%$ fidelity across all trials. Thus, in a noise-free (statevector) environment, our proposed quantum convolution technique correctly performs an identical operation to classical convolution given the same input parameters and boundary conditions.

When considering the behavior of noisy (sampled) environments, Figure 12, Figure 14 and Figure 15 plot the fidelity of the noisy quantum results against the ideal classical results. We observe a monotonic decrease in fidelity as the data size (image resolution) increases, consistent with previously reported behavior [23]. Such behavior derives from how the number of shots required to properly characterize a quantum state increases with the corresponding number of qubits in order to reduce the effects of statistical noise. Notably, the fidelity varies dramatically depending on the filter category and size, where the largest discrepancy occurs between the ($5\times 5$) Averaging and ($5\times 5$) Laplacian filters for a data size of $65,536$ values. Specifically, for a B/W image of ($256\times 256$) pixels, the Averaging filter had a fidelity of 94.84%, while the Laplacian filter had a fidelity of 8.82%—a difference of 86.02%, see Figure 15a. In general, we observed that the average and blur filters perform better than the edge detection methods (Sobel/Laplacian). Since the output data are reconstructed from only a portion of the final state $|{\psi }_5\rangle$, it is likely that sparse filters, represented in Figure 9 and Figure 10 as being mostly black, are significantly less likely to be recorded during sampled measurement, resulting in reduced fidelity. For practical applications, dimension reduction techniques, such as pooling, can be used to mitigate information loss [23].

5. Discussion

In the following section, we compare our proposed method of quantum convolution to the related work discussed in Section 2.1 in terms of filter generalization and circuit depth.

5.1. Arbitrary Multidimensional Filtering

Our generalizable and parameterized technique of quantum convolution with unity stride offers distinct workflow advantages over existing fixed-filter quantum convolution techniques in variational applications, such as quantum machine learning. Our technique does not require extensive development for each new filter design. For instance, current quantum convolutional filters are primarily two dimensional, only focusing on image processing. However, the development of even similar filters targeting audio and video processing, for example, would require extensive development and redesign. Our method offers a systematic and straightforward approach for generating practical quantum circuits given fundamental input variables.

5.2. Circuit Depth

Figure 8a,b show the circuit depth for our proposed technique of generalized quantum convolution with respect to the total number of data qubits $n={\sum }_{i=0}^{d-1}\left(n_i\right)$ and the total number of filter qubits $n_f={\sum }_{i=0}^{d-1}\left(n_{f_i}\right)$, respectively. The results were gathered using the QuantumCircuit.depth() method built into Qiskit for a QuantumCircuit transpiled to fundamental single-qubit and CNOT quantum gates. Figure 8a illustrates quadratic circuit depth complexity with respect to the data qubits n for a fixed filter size $N_F=2^{n_f}$, aligning with our theoretical expectation in (25). Note that $n={\sum }_{i=0}^{d-1}{n}_i$ and $n_{\max }={max}_{i=0}^{d-1}\left(n_i\right)$ for d-dimensional data. Similarly, Figure 8b (plotted on a log-scale) illustrates exponential circuit depth complexity with respect to $n_f$ for a fixed data size $N=2^n$, which also aligns with our theoretical expectation in (25).

The time complexity comparison of our proposed quantum convolution technique against related work is shown in

Table 1. Comparison of depth/time complexity of proposed generalized quantum convolution technique against related work.

a Depth complexity of quantum data encoding (I/O) techniques
FRQI [14]	NEQR [15]	C2Q [17]
${\mathcal{O}}_{\mathrm{I}/\mathrm{O}}\left(4^n\right)$	${\mathcal{O}}_{\mathrm{I}/\mathrm{O}}\left(qn{2}^n\right)$	${\mathcal{O}}_{\mathrm{I}/\mathrm{O}}\left(2^n\right)$
b Depth complexity of quantum convolution algorithms for a fixed filter
Proposed	Related Work [4,5,6,7,8]
${\mathcal{O}}_{\mathrm{alg}}\left(n_{\max }^2\right)$	${\mathcal{O}}_{\mathrm{alg}}\left(n^2\right)$
c Complexity of proposed technique compared to classical convolution
Proposed	Direct (CPU) [10]	FFT (CPU/GPU) [10,11]	GEMM (GPU) [12,13]
$\begin{array}{c}\hfill {\mathcal{O}}_{\mathrm{alg}}\left(\begin{array}{c}\hfill n_{f_{\max }}{n}_{\max }^2-n_{f_{\max }}^2{n}_{\max }\\ \hfill +2^{n_f}\end{array}\right)\\ \hfill {\mathcal{O}}_{\mathrm{alg}}+\mathrm{I}/\mathrm{O}\left(\begin{array}{c}\hfill n_{f_{\max }}{n}_{\max }^2-n_{f_{\max }}^2{n}_{\max }\\ \hfill +{\mathbf{2}}^{\mathit{n}}+2^{n_f}\end{array}\right)\end{array}$	$\begin{array}{c}{\mathcal{O}}_{\mathrm{alg}}\left(N^2\right)\equiv \hfill \\ {\mathcal{O}}_{\mathrm{alg}}\left(4^n\right)\hfill \end{array}$	$\begin{array}{c}{\mathcal{O}}_{\mathrm{alg}}\left(NlogN\right)\equiv \hfill \\ {\mathcal{O}}_{\mathrm{alg}}\left(n{2}^n\right)\hfill \end{array}$	$\begin{array}{c}{\mathcal{O}}_{\mathrm{alg}}\left(N_{F}N\right)\equiv \hfill \\ {\mathcal{O}}_{\mathrm{alg}}\left(2^{(n+n_f)}\right)\hfill \end{array}$

. Compared to classical direct implementations on CPUs, our proposed technique for generalized quantum convolution demonstrates an exponential improvement with respect to data size $N=2^n$, i.e., $\mathcal{O}\left(n^2\right)$ vs. $\mathcal{O}\left(N^2\right)$, see (25) and Table 1c. As discussed in Section 2.1, the fastest classical GEMM implementation of convolution on GPUs (excluding data I/O overhead) [12,13] has a complexity of $\mathcal{O}\left(N_{F}N\right)$, see Table 1c. Even when including quantum data encoding, which is equivalent to data I/O overhead, our method remains to demonstrate a linear improvement with respect to data size N by a factor of the filter size $N_F$, see (31), over classical GEMM GPUs:

$$\begin{array}{cc}\hfill {\mathcal{D}}_{\mathrm{proposed}}\left(n\right)& ={\mathcal{D}}_{\mathrm{C}2\mathrm{Q}}\left(n\right)+{\mathcal{D}}_{d\text{-}\mathrm{D}\mathrm{conv}}^{\mathrm{opt}}\left(n\right)\hfill \\ \hfill & =\mathcal{O}\left(2^n+n_{\max }^2\right)=\mathcal{O}\left(2^n\right)=\mathcal{O}\left(N\right),\mathrm{for}\mathrm{fixed}{n}_f\hfill \end{array}$$

(31)

Compared to fixed-filter quantum convolution techniques [4,5,6,7,8], our proposed arbitrary filter quantum convolution technique (for unity stride) demonstrates improved circuit depth complexity with respect to data size when factoring in the circuit depth contribution from data encoding. For a fixed filter, i.e., $n_f$ is constant, the depth of the proposed method scales quadratically with the largest data dimension $n_{\max }$, see (25) and (31). As described in Section 2.1, fixed-filter quantum convolution techniques similarly show quadratic depth scaling with respect to the number of qubits n, see Table 1b. For data encoding, the fixed-filter techniques use either the FRQI [14] or NEQR [15] algorithms, which have circuit depth complexities of $\mathcal{O}\left(4^n\right)$ or $\mathcal{O}\left(qn{2}^n\right)$, respectively. In contrast, our proposed technique uses C2Q data encoding [17], which has a depth complexity of $\mathcal{O}\left(2^n\right)$—a quadratic and linear improvement over FRQI and NEQR, respectively, see Table 1a.

6. Conclusions

In this work, we proposed and evaluated a method for generalizing the convolution operation with arbitrary filtering and unity stride for multidimensional data in the quantum domain. We presented the corresponding circuits and their performance analyses along with experimental results that were collected using the IBM Qiskit development environment. In our experimental work, we validated the correctness of our method by comparing classical results to noise-free quantum results. We also demonstrated the practicality of our method for various convolution filters by evaluating the noisy quantum results. Furthermore, we presented experimentally verified analyses that highlight our technique’s advantages in terms of time complexity and/or circuit depth complexity compared to existing classical and quantum methods, respectively. Future work will focus on adapting our proposed technique for arbitrary strides. In addition, we will investigate multidimensional quantum machine learning as a potential application of our proposed technique.