Lightweight Implicit Blur Kernel Estimation Network for Blind Image Super-Resolution

Abstract

Blind image super-resolution (Blind-SR) is the process of leveraging a low-resolution (LR) image, with unknown degradation, to generate its high-resolution (HR) version. Most of the existing blind SR techniques use a degradation estimator network to explicitly estimate the blur kernel to guide the SR network with the supervision of ground truth (GT) kernels. To solve this issue, it is necessary to design an implicit estimator network that can extract discriminative blur kernel representation without relying on the supervision of ground-truth blur kernels. We design a lightweight approach for blind super-resolution (Blind-SR) that estimates the blur kernel and restores the HR image based on a deep convolutional neural network (CNN) and a deep super-resolution residual convolutional generative adversarial network. Since the blur kernel for blind image SR is unknown, following the image formation model of blind super-resolution problem, we firstly introduce a neural network-based model to estimate the blur kernel. This is achieved by (i) a Super Resolver that, from a low-resolution input, generates the corresponding SR image; and (ii) an Estimator Network generating the blur kernel from the input datum. The output of both models is used in a novel loss formulation. The proposed network is end-to-end trainable. The methodology proposed is substantiated by both quantitative and qualitative experiments. Results on benchmarks demonstrate that our computationally efficient approach (12x fewer parameters than the state-of-the-art models) performs favorably with respect to existing approaches and can be used on devices with limited computational capabilities.

1. Introduction

The goal of single image super-resolution (SISR) is to generate a high-resolution (HR) image from a low-resolution (LR) one. It has several applications in visual inspection, satellite imaging, medical imaging [1], astronomy, microscope imaging, seismology, remote sensing [2,3], surveillance [4,5,6], biometrics [7], image compression, etc. Most existing literature (e.g., Refs. [8,9,10]) applies a bicubic downsampling kernel to an HR image to generate its LR counterpart such that the inverse process can be modeled. However, the downsampling process is not so unique in real-world LR images. As a result, modeling it may yield very different performances due to the mismatch with the real degradation settings. To overcome such limitations, blind super-resolution (Blind-SR) methods are introduced with the following degradation process:

$$\mathbf{Y}=(\mathbf{k}⊛\widehat{x}){\downarrow }_S+\eta$$

(1)

where $\widehat{x}$ and $\mathbf{Y}$ are the HR and LR images, ⊛ is the convolution operation, $\mathbf{k}$ is the blur kernel, and $\eta$ is the additive white Gaussian noise. ${\downarrow }_S$ is a down-sampling operator with scale factor S. In the real world, $\eta$ also includes factors that can alter the image acquisition process, including inherent sensor noise, stochastic noise, compression artifacts, and possible mismatches between the forward observation model and the camera device.

The approaches that assume a known blur kernel k are named non-blind image SR (Non-Blind-SR see Figure 1a) and have been extensively studied in literature [8,11,12,13,14]. Methods that assume the blur kernel k as unknown are named blind image SR (Blind-SR). Because there exist infinite pairs of SR images and blur kernels can produce the same LR image $\mathbf{Y}$, the SISR is an ill-posed problem. Thus, regularization is needed to choose the most likely one.

In recent years, deep neural network-based methods have achieved remarkable results in SISR [15,16]. For Non-Blind-SR, the blur kernel is the bicubic interpolation kernel [8,11,17,18,19]. This is used to synthesize a large scale dataset for model training. Real LR images might have a large disparity with the bicubic-generated ones since blur kernels are often more complex. This pushed the literature to focus on SR in the presence of unknown blur kernels. Several techniques have been proposed to address the Blind-SR problem by first estimating blur kernels using statistical priors (e.g., patch self-similarity [20]) or deep neural networks (e.g., Refs. [21,22]) and then applying traditional SR techniques assuming a known kernel (e.g., Refs. [23,24]).

These techniques work well to restore minor details but perform independent estimates of the blur kernels and HR images. So, if the blur kernel is incorrectly estimated, the subsequent restoration of the HR image will produce artifacts in the restored images (see Section 4 (Experiments) for a few examples). These introduced methods simultaneously estimate the blur kernel and latent HR image. Methods following such an approach [25,26,27] have focused on creating several efficient blur kernel estimation algorithms. After the blur kernels have been estimated, they are fed into deep models to be used as inputs to rectify the intermediate features used for HR restoration. However, it is unclear if such an exploitation process actually removes the blur.

Existing Blind-SR methods suffer from two main issues: (i) they explicitly estimate the blur kernel with the supervision of GT kernel (see Figure 1b) and require a large volume of training data to train deeper/wider (many model parameters) networks, and (ii) because of their numerous network parameters and large memory requirements, they are challenging to implement on devices with limited computational capabilities (e.g., embedded devices, smartphones, etc.).

We introduce a novel blind-SR technique to tackle such issues by means of a lightweight convolutional neural network (CNN) that can estimate the unknown blur kernel k and generate the super-resolved image ($\widehat{x}$) simultaneously (see Figure 1c). We propose two different modules: (i) the Estimator and (ii) the Super Resolver. In this method, the low-resolution input image LR$\left(\tilde{x}\right)$ is exploited by the former to predict a blur kernel k and by the latter to restore the SR image ($\widehat{x}$). The two outputs (i.e., k and $\widehat{x}$) are then combined in a joint loss function that encourages $\widehat{x}$ to be (i) as close as possible to the ground-truth HR datum (ii) while verifying that it is filtered (with $\widehat{x}$) and the down-scaled version matches the LR input. We introduce an end-to-end training methodology to achieve a high-quality image restoration result (see Figure 2).

The main contributions are:

• We introduce a Blind-SR approach that estimates the blur kernel k and the SR image ($\widehat{x}$) simultaneously. The blur kernel is implicitly estimated, hence not requiring the supervision of a ground truth kernel;
• Our proposed network compares favorably with respect to state-of-the-art approaches that have a similar number of learnable parameters;
• We provide an end-to-end architecture for the proposed algorithm and extensively analyze its performance via quantitative and qualitative evaluation on benchmark datasets.

2. Related Work

2.1. Non-Blind SR

Super Resolution is an ill-posed problem, for which creating image priors has proven to be a successful approach in recent years [28,29,30]. Even though these image prior-based algorithms have produced respectable results, they demand solutions to challenging optimization issues. The application of several DNNs has resulted in major advancements in recent years [8,11,13,18,23,31,32,33,34,35]. Most of these algorithms directly learn the mapping from LR to HR images, thus ignoring the difficult optimization process induced by sophisticated image priors. In Ref. [36], authors demonstrated that using feed-forward networks to predict the mapping of LR images to HR images is insufficient. Haris et al. [36] develop a deep back-projection network based on an error feedback mechanism to improve the LR to HR mapping. In Ref. [37], the image model’s formation process is incorporated into a deep CNN model [18], explicitly ensuring that the estimated high-resolution images satisfy the model of the image formation process.

These deep CNN-based algorithms significantly outperform image prior-based methods and reach state-of-the-art results on many benchmarks. However, such methods suppose that the degradation settings with known blur kernels, for example, the Bicubic interpolation [8,31], Gaussian blur [38], or generalized ones [36], which does not hold true in real circumstances because the blur kernels in the real degradation process are usually more sophisticated. As discussed in Ref. [39], exploiting known/predetermined blur kernels yields SR images with noisy artifacts. Differently, we follow a Blind-SR approach.

2.2. Blind-SR

Different Blind-SR methods [20,40,41] have been proposed to recover the HR image from an input LR with an unknown blur kernel. Conventional techniques typically require the estimate of the blur kernel and the latent HR restoration, which are bound by statistical image priors. Michaeli et al. [20] investigate the internal patch recurrence to estimate the blur kernel and latent HR images. Wang et al. [42] developed an effective probabilistic combination model for blind image SR based on a patch-based image synthesis constraint. In Ref. [43], the authors follow the real-world settings for degradation by using an adversarial learning procedure to train the model with pixel-by-pixel supervision from its LR counterpart in the HR domain. Ref. [44] introduces a deep network trained iteratively with a residual learning method that takes advantage of powerful image regularization and large-scale optimization techniques. Ref. [45] proposes a low-resolution to high-resolution domain translation approach for real-image super-resolution. In Ref. [46], a burst photography pipeline is used to restore the HR.

Despite having achieved reasonable results, these techniques frequently involve complex optimization techniques. To address the Blind-SR problem, multiple approaches develop deep CNNs instead of statistical image priors. In order to estimate blur kernels from LR images, Bell-Kligler et al. [21] develop an effective technique based on an image-specific Internal-GAN. This method provided respectable results and can be used with image SR techniques that rely on blur kernels for performance enhancement, such as those found in Ref. [24]. However, results with considerable artifacts will be produced as a result of the blur kernel errors. For accurate kernel estimation and SR refinement, Gu et al. [22] introduced a spatial feature transform (SFT) and an iterative kernel correction (IKC) technique. Luo et al. [47] estimate the reduced kernel and restore the HR image iteratively in an end-to-end fashion. Our approach has the same spirit as Refs. [22,47] with some relevant differences. In Refs. [22,47] the basic properties of the blind image SR problem are not adequately modeled by separately estimating the blur kernels and latent HR images, thus affecting the final latent HR image restoration. In addition, both such methods are extremely time-consuming and computationally very expensive. Differently, our approach employs a trainable end-to-end network with a limited number of parameters, thus opening to memory and computationally constrained devices.

3. Method

3.1. Problem Formulation

According to (1), estimating $\widehat{x}$ from $\mathbf{Y}$ is primarily based on the variational strategy for combining observation and prior knowledge. This requires solving the following minimization problem:

$$\widehat{E}\left(\widehat{x}\right)=arg\underset{\widehat{x}}{min}\frac{1}{2}{\parallel \mathbf{Y}-\mathbf{k}\widehat{x}\parallel }_2^2+\lambda R_W\left(\widehat{x}\right)$$

(2)

where $\frac{1}{2}{\parallel \mathbf{Y}-\mathbf{k}\widehat{x}\parallel }_2^2$ is the data fidelity term related to the model likelihood. It measures how closely the solution matches the observations. $R\left(\widehat{x}\right)$ is a regularization term related to image priors, and $\lambda$ is a trade-off parameter that controls how closely the solution resembles the observations. It is interesting to note that the variational technique directly relates to the Bayesian approach. The generated solutions can be categorized as either maximum a posteriori (MAP) estimates [48,49] or penalized maximum likelihood estimates. Due to strong prior capabilities, we adopt the generator network [43] for super-resolution learning and a simple CNN-based novel kernel estimator network, which predicts the kernel as closest to the original one. We trained both networks end-to-end by exploiting a GAN framework to minimize the energy-based objective function (2) with the discriminative and residual learning approaches, considering the estimated kernel.

3.2. Kernel Estimation

For training kernel estimation network ($N_{kest}$), we provided low-resolution input $\tilde{x}$, which is exploited by the network ($N_{kest}$) and predicts the kernel $\mathbf{k}$, which is further utilized for super resolver learning, as shown in Figure 2.

3.3. Super Resolver

The estimated blur kernel through the kernel estimator network is shown in Figure 2. In the training process, we use the same low-resolution input $\tilde{x}$ for super-resolver $G_{SR}$ and our proposed estimator $N_{kest}$. The super-resolved image $\widehat{x}$ obtained from super-resolver $G_{SR}$ is convolved with the predicted blur kernel $\mathbf{k}$ obtained from the $N_{kest}$ (estimator). The blur kernel $\mathbf{k}$ predicted by $N_{kest}$ (estimator)is used to calculate the novel loss ${\mathcal{L}}_k$ after the convolution with the super-resolved image. We calculate the ${\mathcal{L}}_1$ from the super-resolved image $\widehat{x}$ and ground truth. The loss ${\mathcal{L}}_k$ is added to the ${\mathcal{L}}_1$ and other network losses (i.e., ${\mathcal{L}}_{per},{\mathcal{L}}_{GAN},{\mathcal{L}}_{tv}$) to calculate the final loss ${\mathcal{L}}_{{\mathbf{G}}_{\mathbf{SR}}}$ as given in (9).

3.4. Network Architectures

The network architectures of the Generator (${\mathbf{G}}_{SR}$), Discriminator (${\mathbf{D}}_y$), and Kernel Estimator (${\mathbf{N}}_{kest}$) are depicted in Figure 3. The letters s, c, and k denoted stride size, number of filters, and kernel size, respectively.

3.4.1. Generator Network (${\mathbf{G}}_{SR}$):

As illustrated in Figure 3a, the Encoder and Decoder include 64 feature maps, $C\times H\times W$ tensors, a $5\times 5$ kernel, and C input channels. The LR input $\tilde{x}$ is upsampled using Bilinear kernel ${\mathbf{H}}^{\mathrm{T}}$. There are 5 residual blocks, 2 pre-activation convolutional layers with 64 feature maps each, and $3\times 3$ kernel. Pre-activations are parametrized rectified linear units (PReLU) that support 64 feature maps. The project layer (Proj) [50] in the decoder determines the proximal map using standard deviation ($\sigma$), which accounts for the prior terms and data fidelity term. The $\alpha$ parameter in Proj is fine-tuned with the back-propagation in training. In addition, the Resnet block located in-between the encoder and the decoder is where the noise is estimated. The input LR image is then subtracted from the estimated residual image provided by the Decoder. Finally, the clipping layer is responsible for complying with the valid intensity of the image ranging from 0 to 255. In training, reflection padding is utilized to lag the convolutional layers, which makes the consistent change in the input image.

3.4.2. Discriminator Network (${\mathbf{D}}_y$):

Figure 3b shows the discriminator network architecture. This aims to classify if the input is a generated SR image $\widehat{x}$ (i.e., fake) or a real HR image y. The discriminator comprises $3\times 3$ kernel, $4\times 4$ kernel, 64 feature map, 512 feature map, leaky ReLU, and Batch Normalization (BN) as suggested in SRGAN [23].

3.4.3. Estimator Network (${\mathbf{N}}_{kest}$):

Estimating the blur kernels from given LR images is difficult because blur and downsampling operations result in information loss. Existing methods typically require sophisticated priors and the solution of complex optimization problems [20]. Bell-Kligler et al. estimate the blur kernel from a single LR image using a generative adversarial network in Ref. [21]. Unlike [20,21], we develop a simple deep CNN model (${\mathbf{N}}_{kest}$) that ingests an LR image to estimate its blur kernel. This network, shown in Figure 3c, is trained using (8).

3.5. Loss Calculation

3.5.1. Texture Loss (${\mathcal{L}}_{GAN}$):

Although perceptual loss can improve the overall quality of a reconstructed image, it still introduces unwanted high-frequency components. That is why we consider including texture loss in the total loss function as follows.

$$\begin{array}{cc}\hfill {\mathcal{L}}_{GAN}=& {\mathcal{L}}_{RaGAN}-{\mathbb{E}}_y[log(1-{\mathbf{D}}_{\mathbf{y}})(\mathbf{y},{\mathbf{G}}_{\mathbf{SR}}(\widehat{\left(x\right)}))]\hfill \\ \hfill -& {\mathbb{E}}_{\widehat{y}}[log\left({\mathbf{D}}_{\mathbf{y}}({\mathbf{G}}_{SR}\left(\widehat{x}\right),y)\right)]\hfill \end{array}$$

(3)

where ${\mathbb{E}}_y$ and ${\mathbb{E}}_{\widehat{y}}$ are used for the average of real (y) and fake ($\widehat{y}$) data, respectively. We use a discriminator that gives the GAN score of a real image (HR) and a fake image (SR) as used in Ref. [10]. It is defined as follows.

$${\mathbf{D}}_y(y,\widehat{y})\left(C\right)=\sigma (C\left(y\right)-\mathbb{E}\left[C\left(\widehat{y}\right)\right]$$

(4)

The sigmoid function and the output of the raw discriminator have been represented by $\sigma$ and C, respectively, as shown in Figure 3b.

3.5.2. Perceptual Loss (${\mathcal{L}}_{per}$):

To generate images with more precise brightness and realistic textures, (${\mathcal{L}}_{per}$) based on the VGG network, it is configured to use information from the feature layer before the activation layer. On the activation layer of the pre-trained deep neural network, it is specified to minimize the Euclidean distance between two activation features.

$$\begin{array}{cc}\hfill {\mathcal{L}}_{per}=& \frac{1}{N}\sum _i^N{\mathcal{L}}_{VGG}\hfill \\ \hfill =& \frac{1}{N}\sum _i^N\parallel \varphi ({\mathbf{G}}_{SR}\left(\widehat{x_i}\right)-\varphi \left(y_i\right){\parallel }_1\hfill \end{array}$$

(5)

where $\varphi$ denotes extracted feature from VGG-19 pretrained network as specified in Ref. [10].

3.5.3. TV (Total-Variation) Loss (${\mathcal{L}}_{tv}$):

The absolute differences between adjacent pixel values in the input images are added to determine total variation loss. Total Variation loss measures the amount of noise in images. To remove the image’s rough texture and make the resultant image look smoother, we added the total variation loss to the total loss, as follows.

$$\begin{array}{cc}\hfill {\mathcal{L}}_{tv}=& \frac{1}{N}\sum _i^N(\parallel {▿}_h{\mathbf{G}}_{SR}\left({\widehat{x}}_i\right)-{▿}_h\left(y_i\right){\parallel }_1\hfill \\ \hfill +& \parallel {▿}_v{\mathbf{G}}_{SR}\left({\widehat{x}}_i\right)-{▿}_v\left(y_i\right){\parallel }_1)\hfill \end{array}$$

(6)

where ${▿}_v$ and ${▿}_h$ represent the vertical and horizontal gradients of the images.

3.5.4. Content Loss (${\mathcal{L}}_1$)

Mean Absolute Error (MAE) loss is used as the model’s content loss to ensure that low-frequency information is the same between the reconstructed and LR images. Its job is to minimize the difference between the pixels in the generated HR images and those in the real HR images. By making the distance between pixels smaller, the accuracy of the reconstructed image information can be evaluated more quickly and accurately, leading to a higher peak signal-to-noise ratio. We incorporated content loss to a total loss to generate a good quality reconstructed image, as follows.

$${\mathcal{L}}_1=\frac{1}{N}\sum _i^N{\parallel {\mathbf{G}}_{SR}\left(\widehat{x}\right)-y_i\parallel }_1$$

(7)

where N represents the batch size.

3.5.5. Estimator Loss(${\mathcal{L}}_k)$:

We computed the estimator loss in such a manner that we combined two outputs (i.e., k and $\widehat{x}$) such that $\widehat{x}$ should be as close as possible to the ground-truth HR image, and it is convolved (with $\widehat{x}$) and the down-scaled version matches the LR input.

$${\mathcal{L}}_k=\parallel \mathbf{k}⊛{\mathbf{G}}_{SR}\left(\widehat{x}\right){{\downarrow }_S-LR\left(\tilde{x}\right)\parallel }_2$$

(8)

where S is a downscaling factor, k is the estimated kernel, and ${\mathbf{G}}_{SR}\left(\widehat{x}\right)$ is the super-resolved imaged.

Total loss function (${\mathcal{L}}_{{\mathbf{G}}_{\mathbf{SR}}}$) formulation is defined as:

$${\mathcal{L}}_{{\mathbf{G}}_{\mathbf{SR}}}={\mathcal{L}}_{GAN}+{\mathcal{L}}_{per}+{\mathcal{L}}_{tv}+10·{\mathcal{L}}_1+{\mathcal{L}}_k$$

(9)

4. Experiments

4.1. Datasets

We followed a common protocol [22,43,45,47] and used 3450 high-resolution (HR) images from DIV2K [51] and Flickr2K [52] for model training. For a fair comparison with existing approaches, we followed [22,47] and trained/evaluated our approach with the two following degradation settings.

4.1.1. Setting 1

We follow the protocol in Ref. [22] and set the kernel size to 21. For scale factors 4 and 2, the kernel width is uniformly sampled during training in the ranges of [0.2, 4.0] and [0.2, 2.0]. Evaluation is conducted on popular benchmark HR datasets, such as Set5 [53], Set14 [54], Urban100 [55], BSD100 [56], and Manga109 [57]. For a fair comparison with existing methods, during testing, we adopted the same approach of Ref. [22] and uniformly selected 8 kernels from the ranges [1.8, 3.2] and [0.80, 1.60] for scale factors 4 and 2, respectively. The HR images are first blurred using the selected blur kernels, then downsampled to generate synthetic test images.

4.1.2. Setting 2

Following the training protocol of Ref. [47], we set the kernel size to 11 and generated Anisotropic Gaussian kernels with both axes’ lengths randomly taken in the range (0.6, 5). A random rotation in $[-\pi ,\pi ]$ is then applied. We added uniform multiplicative noise (up to 25 % of each kernel pixel value) and normalized it, to sum up to one. For a fair comparison, the evaluation is run on the DIV2KRK benchmark dataset (with no additional blur kernel applied to the images since these are already degraded).

4.2. Model Optimization

We trained our model with $32\times 32$ LR patches for 51,000 iterations. To minimize (9), we used 16 samples per batch with the Adam optimizer [58] having ${\beta }_1=0.9$, ${\beta }_2=0.999$, and $ϵ={10}^{-8}$ without weight decay for both generator and discriminator. We initially set the learning rate to ${10}^{-4}$, then reduce it by a factor of 2 after 5 K, 10 K, 20 K, and 30 K iterations. The projection layer parameter $\sigma$ (standard deviation) is estimated according to Ref. [59] from the input LR image. We initialize the projection layer parameter $\alpha$ on log-scale values from ${\alpha }_{max}$ = 2 to ${\alpha }_{min}=1$ and then further fine-tune during the training via a back-propagation. Using a GAN framework in Ref. [60] and the following loss functions, we fine-tune the SRResCGAN network to learn the super-resolution. (i.e., pre-trained ${\mathbf{G}}_{SR}$) [43].

Random vertical and horizontal flipping and $90deg$ rotations are used as data augmentation strategies.

4.3. Technical Details

We used Pytorch to implement our technique. The experiments use an i7-8700H processor, 32 GB of RAM, and a 24 GB NVIDIA GeForce RTX 3090 GPU on Ubuntu 20.04 LTS. It took approximately 18.5 h to train the model.

4.4. Evaluation Metrics

For a fair comparison, we evaluated the trained model under the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) [61] metrics. The PSNR and SSIM are distortion-based measures. The RGB color space is used to evaluate the quantitative SR results.

4.5. Experimental Results

4.5.1. Setting 1

We compare the proposed method with state-of-the-art Blind-SR approaches such as IKC [22], DAN [47], and KernelGAN [21] (which explicitly estimates the blur kernels), as well as with blind deblurring methods such as Pan et al [62]. We also compare our results with existing Non-Blind-SR approaches like RCAN [11], ZSSR [24], CARN [17], and MZSR [63].

Table 1. Comparing state-of-the-art Non-Blind (*) and Blind-SR techniques under Setting 1.

Methods	Scale	Set5 PSNR/SSIM	Set14 PSNR/SSIM	B100 PSNR/SSIM	Urban100 PSNR/SSIM	Manga109 PSNR/SSIM	#Params (M)
Bicubic		28.65/0.84	26.70/0.77	26.26/0.73	23.61/0.74	25.73/0.84	/
RCAN * [11]		29.73/0.86	27.65/0.79	27.07/0.77	24.74/0.78	27.64/0.87	15.59
ZSSR * [24]		29.74/0.86	27.57/0.79	26.96/0.76	24.34/0.77	27.10/0.87	0.22
MZSR * [63]		29.88/0.86	27.32/0.79	26.96/0.77	24.12/0.77	27.24/0.87	0.22
CARN * [17]	×2	30.99/0.87	28.10/0.78	26.78/0.72	25.77/0.76	26.86/0.86	1.592
Pan et al. [62] + CARN [17]		24.20/0.74	21.12/0.61	22.69/0.64	18.59/0.58	21.54/0.74	/
CARN [17] + Pan et al. [62]		31.27/0.89	29.03/0.82	28.72/0.80	25.62/0.79	29.58/0.91	/
KernelGAN [21] + ZSSR [24]		26.02/0.77	20.19/0.58	21.42/0.60	19.55/0.61	24.22/0.78	0.52
KernelGAN [21] + MZSR [63]		29.39/0.88	23.94/0.72	24.42/0.73	23.39/0.77	28.38/0.89	0.52
IKC [22]		33.62/0.91	29.14/0.85	28.46/0.82	26.59/0.84	30.51/0.91	9.05
DAN [47]		34.55/0.92	29.92/0.86	29.66/0.85	27.96/0.87	33.82/0.95	4.33
Ours		31.02/0.89	27.87/0.80	27.67/0.79	25.11/0.80	28.44/0.89	0.38
Bicubic		24.49/0.69	23.01/0.59	23.64/0.59	20.58/0.57	21.97/0.70	/
RCAN * [11]		24.95/0.71	23.33/0.61	23.65/0.62	20.73/0.61	23.30/0.76	15.59
ZSSR * [24]		24.77/0.70	23.32/0.60	23.72/0.61	20.74/0.59	22.75/0.74	0.22
MZSR * [63]		24.99/0.70	23.45/0.61	23.83/0.61	20.92/0.61	23.25/0.76	0.22
CARN * [17]	×4	26.57/0.74	24.62/0.62	24.79/0.59	22.17/0.58	21.85/0.68	1.592
Pan et al. [62]+ CARN [17]		18.10/0.48	16.59/0.39	18.46/0.44	15.47/0.38	16.78/0.53	/
CARN [17]+Pan et al. [62]		28.69/0.80	26.40/0.69	26.10/0.65	23.46/0.65	25.84/0.80	/
KernelGAN [21] + ZSSR [24]		17.59/0.42	19.20/0.49	17.14/0.40	16.95/0.47	19.40/0.61	0.52
KernelGAN [21] + MZSR [63]		23.08/0.66	22.24/0.61	21.51/0.56	19.37/0.58	22.05/0.70	0.52
IKC [22]		27.84/0.80	25.02/0.67	24.76/0.65	22.41/0.67	25.37/0.81	9.05
DAN [47]		27.64/0.80	25.46/0.69	25.35/0.67	23.21/0.71	27.04/0.85	4.33
Ours		25.94/0.73	23.83/0.62	24.12/0.60	21.15/0.59	22.15/0.71	0.38

shows the quantitative evaluation of the benchmark datasets on scale factors ×2 and ×4. The results show that the proposed method performed better than most of the considered methods. IKC [22] and DAN [47] have higher PSNRs and SSIMs but require a large number of parameters as compared to our method. Figure 4 and Figure 5 show visualized SR results of the evaluated methods when scaled by a factor of ×4, and the blur kernel in the degradation model is an isotropic Gaussian blur kernel. The non-blind SR approaches, such as Bicubic, RCAN [11], and MZSR [63], produce results with a considerable blur impact because they do not mimic the blur kernels while super-resolving LR images (Figure 4c–e). The KernelGAN [21] approach proposes an effective blur kernel estimation method and can use existing non-blind SR methods for blind SR, such as ZSSR [24]. However, because blur kernel estimate and HR image restoration are distinct processes, HR image restoration cannot rectify problems produced by inaccurate blur kernels. As a result, the restored image has artifacts, as illustrated in Figure 5f.

The IKC [22] proposed an effective iterative kernel correction method to explicitly estimate blur kernels and restore sharp images, as shown in Figure 4g, but has a large number of parameters.

Figure 4h shows that the state-of-the-art blind SR method DAN [47] also produce visually fine results.

On the other hand, our proposed method implicitly estimates blur kernels and HR images simultaneously using a lightweight (less number of parameters) end-to-end trainable network and the super resolver network to restore the HR images. Figure 4 and Figure 5 show that our generated SR images are much better than state-of-the-art methods (e.g., Figure 4d–f), however Figure 4g,h outperforms ours.

4.5.2. Setting 2

Setting 2 used irregular blur kernels (anisotropic), which is more generic but harder to solve.

Table 2. Comparing state-of-the-art Non-Blind (*) and Blind-SR techniques under Setting 2.

Scale	Bicubic	RCAN* [11]	ZSSR* [24]	KernelGAN [21] + ZSSR [24]	IKC [22]	DAN [47]	KOALAnet [27]	Ours
×2	27.00/0.77	27.52/0.79	27.47/0.79	27.62/0.79	29.24/0.84	31.09/0.88	30.48/0.86	27.53/0.79
×4	23.89/0.64	24.16/0.65	24.11/0.65	24.50/0.66	25.26/0.70	26.42/0.73	26.23/0.72	24.67/0.66

shows that our proposed method performs favorably against some state-of-the-art methods like RCAN [11], ZSSR [24], but IKC [22], DAN [47], and KOALAnet [27] have better results since they explicit estimate the blur kernels (hence assuming a GT kernel is available) and have a large number of parameters. Figure 6 shows the visual comparison with the state-of-the-art (SOTA) super-resolution (SR) method with setting 2.

4.6. Computational Cost

In

Table 3. Comparison of FLOPs.

Methods	FLOPs
IKC [22]	2178.72 G
DAN [47]	929.35 G
Ours	295.17 G

we report on the number of FLOPs required to run our method and compare it against [22] and [47]. We followed the same methodology described in Ref. [47] and computed the FLOPs and inference time considering an input size of $270\times 180$. This is done for 40 images synthesized by 8 blur kernels from the Set5 dataset. Results show that, with $3\times$ less required FLOPs, our model achieves a significantly faster inference time yielding 0.243 s/image while DAN and IKC have 0.312 and 1.735 s/image, respectively.

4.7. Ablation Study

In

Table 4. This table displays that the DIV2K validation set (100 images with unknown blur kernels) was used for our ablation investigation.

Methods	w/o (${\mathit{N}}_{\mathbf{kest}}$)	Ours
PSNR/SSIM	25.63/0.69	25.80/0.71

, we analyze the impact of the kernel estimation module. Results are computed on the DIV2K validation set with unknown blur kernels. It shows that without using (${\mathbf{N}}_{kest}$) it does not generate good SR images in terms of PSNR/SSIM, and with our proposed method, PSNR/SSIM increased by +0.17/+0.02, respectively. To study the effects of different blur kernels, we compute the results in

Table 5. Quantitative performance of proposed approach on Set5(×4) with different blur kernels width.

Kernel Width	1.0	2.5	3.0	3.5	4.0
Bicubic	25.20/0.72	24.38/0.69	23.70/0.66	23.18/0.63	22.80/0.62
ZSSR [24]	26.30/0.76	25.06/0.72	24.11/0.68	23.44/0.65	22.95/0.62
IKC [22]	28.12/0.82	28.32/0.82	28.29/0.81	27.90/0.80	24.26/0.69
Ours	26.31/0.76	26.05/0.73	25.09/0.70	24.17/0.66	23.38/0.63

. These indicate that our method can generalize better than a Non-Blind-SR method (i.e., Ref. [24]) while maintaining competitive performance with a Blind-SR method (i.e., Ref. [22]) requiring $7.5\times$ more learnable parameters.

5. Conclusions

In this paper, we have introduced an effective, lightweight, and implicit blur kernel estimation end-to-end approach for blind image super-resolution (blind-SR). Our proposed approach is based on a deep convolutional neural network (CNN) named estimator and a deep super-resolution residual convolutional generative adversarial network named super resolver. The Estimator module implicitly estimates the blur kernels from the LR input without the supervision of the ground truth kernel. The Super Resolver module restores the SR image by exploiting a GAN framework to minimize the energy-based objective function with the discriminative and residual learning approaches, considering the estimated kernel. The whole architecture is trained in an end-to-end fashion. Results on different benchmark datasets show that our approach achieves better performance than state-of-the-art methods having a similar number of learnable parameters, enabling it to work on devices with limited computational capacity.