Single Image Super-Resolution Reconstruction with Preservation of Structure and Texture Details

Abstract

In recent years, deep-learning-based single image super-resolution reconstruction has achieved good performance. However, most existing methods pursue a high peak signal-to-noise ratio (PSNR), while ignoring the quality of the structure and texture details, resulting in unsatisfactory performance of the reconstruction results in terms of human subjective perception. To solve this issue, this paper proposes a structure- and texture-preserving image super-resolution reconstruction method. Specifically, two different network branches are used to extract features for image structure and texture details. A dual-coordinate direction perception attention (DCDPA) mechanism is designed to highlight structure and texture features. The attention mechanism fully considers the complementarity and directionality of multi-scale image features and effectively avoids information loss and possible distortion of image structure and texture details during image reconstruction. Additionally, a cross-fusion mechanism is designed to comprehensively utilize structure and texture information for super-resolution image reconstruction, which effectively integrates the structure and texture details extracted by the two branch networks. Extensive experiments verify the effectiveness of the proposed method and its superiority over existing methods.

1. Introduction

Image super-resolution (SR) focuses on recovering the corresponding high-resolution images from degraded low-resolution images. Image SR techniques can be used in face recognition [1], medical imaging [2,3], satellite imagery [4] and so on. As an important image-processing method, image SR has received extensive attention from researchers. Existing SR methods can be roughly divided into four categories: interpolation-based methods [5,6,7,8], inverse reconstruction-based methods [9,10,11,12], traditional machine-learning-based methods [13,14,15,16,17,18,19,20,21,22], and deep-learning-based methods [23,24,25,26,27,28,29,30,31].

Interpolation-based methods use known adjacent pixel information to generate unknown pixels. This type of method is simple to implement, but its detail recovery ability is poor, and the reconstruction results are prone to blur and aliasing. Assuming that the low-resolution image is obtained by a series of degradations of the high-resolution image, inverse reconstruction-based methods use the reverse reconstruction algorithm to restore the high-resolution image. Based on the degradation model of the image, this kind of method is often developed in combination with prior knowledge of the image [9,11] and obtains a high-resolution image by optimizing the model. Although this kind of method preserves more image details, the reconstruction performance is susceptible to image degradation and regularization performance. Additionally, when the resolution is improved by more than four times, the reconstruction performance is usually not good. Traditional learning-based methods use the training data to learn the non-linear mapping relationship between a low-resolution image and its high-resolution version. Then, the learned mapping relationship is used to guide high-resolution image reconstruction. Freeman et al. [32] first proposed the learning of this relationship in Markov neural networks, but this method requires a lot of time to construct a training set and image reconstruction is also time-consuming. Chang et al. [33] assumed that both high- and low-resolution image patches have manifolds with the same geometric structure and proposed a super-resolution reconstruction method based on neighborhood embedding. Since dictionary learning has shown excellent performance in computer vision [14,34] and medical imaging [35,36], Yang et al. [37] proposed a super-resolution reconstruction method based on sparse representation. This method assumes that the high- and low-resolution images have the same coding coefficients under the respective dictionary, and reconstructs the high-resolution image by applying the coding coefficients of the low-resolution image under the low-resolution dictionary to the high-resolution dictionary. In view of the effectiveness of sparse representation and its superiority for image processing [38,39,40,41] and recognition [42,43,44], image super-resolution reconstruction based on sparse representation became a very popular method at that time. Although this type of method has achieved breakthroughs in performance, the optimization process of the model is too complicated, time-consuming and memory-intensive.

In 2014, Dong et al. [17] first applied deep learning to image super-resolution reconstruction and proposed super-resolution convolutional neural networks (SRCNN). Since then, deep-learning-based methods have become the mainstream methods for image super-resolution reconstruction [23,24,25,26,27,28,29,30,31,45,46]. This kind of method usually uses a single-branch network under the constraint of loss function so that the network can extract the texture details required to restore the low-resolution image to the high-resolution image, thereby effectively improving the visual effect of the reconstructed image. However, texture features are easily lost in the process of image convolution in real cases. This is extremely disadvantageous for recovering texture details in high-resolution images. In order to improve the visual performance of reconstructed images, Ledig et al. [13] proposed super-resolution generative adversarial networks (SRGAN), which use the generative adversarial loss to constrain the generator and discriminator. Hence, the generator obtains texture details that are closer to the real world. Since then, Wang et al. [47] further improved the detail recovery ability of SRGAN by introducing dense residual blocks. However, the discriminator may introduce instability in the optimization process during the adversarial process, causing structure inconsistency in the reconstructed image, resulting in unnatural artifacts and geometric distortions in the recovered super-resolution image. Therefore, Ma et al. [24] proposed a structure-preserving super-resolution (SPSR) method to alleviate the above problem, which uses the gradient information of the image to guide the low-resolution image restoration, while maintaining the advantages of GAN-based methods to generate results that achieve satisfactory performance in terms of human subjective perception.

In order to reconstruct the structure and texture details of images with high quality, this paper proposes a structure- and texture-preserving image super-resolution method. In the proposed method, the structure and texture details of the image are extracted by different network branches. In this process, a dual-coordinate direction perception attention (DCDPA) mechanism is designed to highlight structure and texture features. This mechanism not only makes full use of the complementarity of multi-scale image features, but also considers the directionality of image features in the horizontal and vertical coordinates. Therefore, both information loss and possible distortion of the image structure and texture details are effectively avoided during image reconstruction. To comprehensively utilize the structure and texture information to reconstruct high-resolution images, this paper proposes a cross-fusion mechanism to integrate structure and texture features.

In the network structure, the proposed network framework is mainly composed of a structure feature extraction module (SFEM), a texture detail feature extraction module (TDFEM), and feature highlight and aggregation modules (FHAM). The SFEM and TDFEM are responsible for the learning of image structure features and texture detail features, respectively. The FHAM enhances the structure and texture detail features in different directions in the image through the designed DCDPA mechanism and promotes information interaction between them through the cross fusion of the structure and texture detail features to improve the feature representation ability. The network is optimized by reconstruction loss, texture loss, and gradient loss. This paper makes three main contributions as follows:

• An image super-resolution reconstruction network for structure and texture restoration is designed. This network uses the dual-stream structure to extract the structure and texture details of the image, respectively, and enhances these two features layer-by-layer through the FHAM to improve the network’s ability to extract features.
• The FHAM is designed to highlight the structure and texture features in the image. In order to enable the network to capture the salient structure and texture features in the image, this paper proposes a DCDPA mechanism in this module. This mechanism makes full use of the directionality of image features to highlight the features of different directions in the image. Moreover, a cross-fusion block (CFB) is proposed to promote the interactive fusion of structure and texture features.
• A large number of experimental results show that the proposed method contains salient structure and texture details in its super-resolution reconstruction results, which outperform state-of-the-art methods in visual quality and performance evaluation.

2. Relate Work

2.1. Deep-Learning-Based Super-Resolution Reconstruction Methods

In recent years, deep learning has been widely used in single-image super-resolution reconstruction. Dong et al. [17] proposed the SRCNN. Compared with traditional methods, SRCNN achieves better super-resolution reconstruction results. Since simply stacking convolutional layers can lead to gradient explosion, Kim et al. [22] proposed an accurate image super-resolution using very deep convolutional networks (VDSR). The networks avoid the problem of gradient explosion and speed up the network training by adding jump connections. DRRN [48] and Memnet [49] proposed by Tai et al. introduced recursive blocks and memory blocks, respectively, to stack more convolutional layers. However, the above-mentioned networks all use predefined upsampling. Due to resolution changes using traditional methods, the network complexity increases. In addition, since the traditional upsampling methods do not involve network parameter training, the image cannot be reconstructed by convolution, activation function, etc. Therefore, the upsampling process magnifies the defects in low-resolution images. To solve this problem, Dong et al. [25] proposed a fast convolutional neural-network-based super-resolution reconstruction (FSRCNN) method, which performs feature extraction on low-resolution images, reduces the complexity of the network, and applies deconvolution to upsampling and image reconstruction. Shi et al. [26] proposed an efficient sub-pixel convolutional layer to achieve upsampling. In recent years, Zhu et al. [27] proposed a lightweight feature separation and fusion network, adopting a block-processing structural framework, which enables the network to fully utilize and enrich features at different levels. Liu et al. [28] proposed a multi-scale skip-connection network (MSN) to improve the visual quality of super-resolution images, utilizing convolution kernels of different sizes to capture multi-scale features of low-resolution images. Huang et al. [29] proposed an attention network with detail fidelity, which achieves super-resolution image reconstruction while obtaining detail fidelity. Mehri et al. [30] proposed the multi-path residual network (MPRNet), which adaptively learns the most valuable features to learn more high-frequency information about the image. Tan et al. [31] proposed an image super-resolution method via a self-calibrated feature fusion network (SCFFN) to achieve a better balance between network performance and applicability. However, the above methods all consider the texture feature information and structure feature information of the image as a whole. In reality, the texture features are easily lost during the image convolution process. This is extremely disadvantageous for reconstructing high-resolution images rich in texture feature information.

2.2. Visual-Perception-Based Super-Resolution Reconstruction Methods

To improve the perceptual quality of images, in recent years, a large number of super-resolution reconstruction models that benefit the perceptual performance of human eyes have been proposed [13,47,50]. These methods use a variety of information compensation methods to compensate the texture features of the super-resolution reconstruction results. While reconstructing the structure information, they prevent the loss of texture details during the super-resolution reconstruction process and improve the perceptual performance of images. A photo-realistic single image super-resolution using a generative adversarial network (SRGAN) proposed by Ledig et al. [13] uses the discriminator and the generator for adversarial learning and enables the generator to generate texture details that are close to the real world. In order to reduce the artifacts caused by generative confrontation in SRGAN, Wang et al. [47] proposed an enhanced super-resolution generative adversarial network (ESRGAN), which introduced dense residual blocks into the basic network framework and abandoned the batch normalization operation to generate more realistic and natural images. Wu et al. [51] proposed a perceptual generative adversarial network for single-image super-resolution (SRPGAN), in which a robust perceptual loss based on the model’s discriminator was proposed to recover texture details. Wang et al. [50] proposed an image super-resolution reconstruction method for texture recovery by deep spatial feature transform, which integrates a semantic prior into low-resolution images and uses a spatial feature transform (SFT) layer to further improve texture recovery. A new dense block proposed by Chen et al. [52] uses complex connections between each layer to build a more powerful generator and applies new feature maps to compute the perceptual loss to make the output image more realistic and natural. Ma et al. [24] proposed a structure-preserving image super-resolution method, which uses the gradient information of the image to guide image recovery. Fu et al. [53] proposed a method to guide the image super-resolution reconstruction by weak texture information. This network consists of a main network and two auxiliary prediction networks. The main network extracts and combines the main distinct depth features to assist the prediction network in extracting weak texture information. Cai et al. [54] proposed a texture and detail preservation network (TDPN), which not only focuses on local region feature recovery, but also texture and detail preservation. Meng et al. [55] proposed a gradient information distillation network that not only maintains the advantage of fast and lightweight, but also improves the model performance through information distillation. All of the above-mentioned methods have made great contributions to improving the visual perception of the reconstructed images. However, unnatural artefacts still appear in the recovered super-resolution images. Therefore, this paper proposes a super-resolution reconstruction network that fuses structure and texture information. By enhancing the structure and texture features in the image and cross-fusion, the extraction ability of the network for structure and texture features is improved, thereby reducing artefacts in the reconstructed image and improving the perceptual quality of the reconstructed image.

3. The Proposed Network Framework

The framework of the proposed image super-resolution reconstruction network for structure and texture detail recovery is shown in Figure 1. The model mainly comprises three modules: SFEM, TDFEM, and FHAM. The network structure of both SFEM and TDFEM is the same, but the parameters are not shared. They both consist of a $5\times 5$ convolutional layer and five sets of residual blocks (RBs). Convolutional layers and RBs extract shallow and deep features of the input image, respectively. The input of the SFEM is the original low-resolution image. Additionally, the texture map of the original low-resolution image is obtained by the local binary pattern (LBP) method [56]; the texture map and the original image are concatenated as the input of the TDFEM. The FHAM consists of five gated fusion blocks (GFBs). The GFB enhances the structure and texture information of the image, respectively, by introducing a DCDPA mechanism and fusing the enhanced structure and texture information through cross-fusion. Finally, the reconstructed super-resolution image is obtained by deconvolution.

3.1. Structural Feature Extraction Module and Texture Detail Feature Extraction Module

The network structure of both SFEM and TDFEM is the same, including shallow-feature extraction and deep-feature extraction. The network adopts a $5\times 5$ convolution layer and five RBs to extract the shallow and deep features of the image, respectively. Assuming that $\mathit{I}$ represents the input of the $5\times 5$ convolutional layer, the shallow feature ${\mathit{F}}^s$ obtained by convolution is:

$${\mathit{F}}^s=sconv(\mathit{I},k=5)$$

(1)

where $sconv$ represents equal-sized convolution operations, and k represents the size of the convolution kernel.

Let ${\mathit{I}}_{LR}$ denote the input original low-resolution image. For the SFEM $\mathit{I}$ = ${\mathit{I}}_{LR}$, the obtained shallow structure feature is ${\mathit{F}}_s^s$. In order to prevent the subjective image quality degradation caused by loss of texture detail information during the super-resolution reconstruction process, the LBP method is used to obtain the texture map ${\mathit{I}}_t$ of the original image ${\mathit{I}}_{LR}$ and to concatenate them with the input of the TDFEM. Therefore, for the TDFEM, $\mathit{I}=concat( {\mathit{I}}_{LR},{\mathit{I}}_t)$, where $concat$ denotes the concatenation operation. The obtained shallow texture detail feature is ${\mathit{F}}_t^s$. The RB is used to extract the deep features of the image. The relationship between the input and output of the i-th ($i=1,2,\dots ,5$) RB can be expressed as:

$${\mathit{F}}_r^i=scon{v}_g\left({\mathit{F}}_r^{i-1}\right)+{\mathit{F}}_r^{i-1}$$

(2)

where $scon{v}_g$ represents a convolution group, which consists of six $3\times 3$ convolutional layers. ${\mathit{F}}_r^{i-1}$ and ${\mathit{F}}_r^i$ represent the input and output of the i-th RB, respectively.

3.2. Feature Highlight and Aggregation Module and Feature Reconstruction

To ensure that the network can realize the fusion of image structure and texture features and prevent the loss of structure and texture details, this paper proposes the FHAM. The module consists of five GFBs, as shown in Figure 1. The GFB is composed of DCDPA and CFB.

3.2.1. Dual-Coordinate Direction Perception Attention

To enable the network to capture structure and texture details in different directions from the image, a DCDPA mechanism is used, which includes direction perception coordinate attention (DPCA) and global perception coordinate attention (GPCA), as shown in Figure 2.

The input ${\mathit{F}}_{Gin}$ of the DCDPA module contains the texture detail and structure features of the image. Specifically, for the first GFB, its input is obtained by concatenating the shallow structure feature ${\mathit{F}}_s^s$ and the shallow texture detail feature ${\mathit{F}}_t^s$. For other GFBs, the corresponding input is the output of the previous GFB. ${\mathit{F}}_{Gin}$ passes through $3\times 3$ and $5\times 5$ convolutions, respectively, to obtain the features ${\mathit{F}}_l$ ($l=3,5$) of different scales. ${\mathit{F}}_l$ obtains the horizontal feature ${\mathit{F}}_l^h$ and the vertical feature ${\mathit{F}}_l^v$ through global average pooling (GAP) in the horizontal and vertical directions, respectively:

$$\begin{array}{c}\hfill {\mathit{F}}_l^h=P_h\left({\mathit{F}}_l\right)\\ \hfill {\mathit{F}}_l^v=P_v\left({\mathit{F}}_l\right)\end{array}$$

(3)

where $l=3,5$, and $P_h$ and $P_v$ represent the GAP in the horizontal and vertical directions, respectively. Set ${\mathit{F}}_l\in {\mathbb{R}}^{3\times H\times W}$, then ${\mathit{F}}_l^v\in {\mathbb{R}}^{3\times 1\times W}$, ${\mathit{F}}_l^h\in {\mathbb{R}}^{3\times H\times 1}$. To concatenate features in different directions, it is necessary to reshape ${\mathit{F}}_l^h$ into $3\times 1\times H$ dimensions. The features in different directions are concatenated along the horizontal direction to obtain four sets of joint features as follows:

$$\begin{array}{c}\hfill {\mathit{F}}_{3,3}^{hv}=concat({\mathit{F}}_3^h,{\mathit{F}}_3^v)\\ \hfill {\mathit{F}}_{3,5}^{hv}=concat({\mathit{F}}_3^h,{\mathit{F}}_5^v)\\ \hfill {\mathit{F}}_{5,3}^{hv}=concat({\mathit{F}}_5^h,{\mathit{F}}_3^v)\\ \hfill {\mathit{F}}_{5,5}^{hv}=concat({\mathit{F}}_5^h,{\mathit{F}}_5^v)\end{array}$$

(4)

where $concat$ represents the concatenation operation. To obtain the perceptual coordinate attention weights in different directions, $1\times 1$ convolution and $sigmoid$ are applied to process the joint features to obtain attention maps in different directions. Taking ${\mathit{F}}_{3,3}^{hv}$ as an example, the attention maps in the horizontal and vertical directions obtained after ${\mathit{F}}_{3,3}^{hv}$ is processed are ${\mathit{M}}_{3,3}^h\in {\mathbb{R}}^{3\times 1\times W}$ and ${\mathit{M}}_{3,3}^v\in {\mathbb{R}}^{3\times 1\times W}$:

$$\left[{\mathit{M}}_{3,3}^h,{\mathit{M}}_{3,3}^v\right]=split(sigmoid(BN(sconv({\mathit{F}}_{3,3}^{hv},k=1))))$$

(5)

where $split$ represents the segmentation operation, $sigmoid$ represents the sigmoid activation function, and $BN$ represents the batch normalization layer. After applying $1\times 1$ convolution and dimension expansion in the horizontal and vertical directions to ${\mathit{M}}_{3,3}^h$ and ${\mathit{M}}_{3,3}^v$, the perceptual coordinate attention maps ${\tilde{\mathit{M}}}_{3,3}^h\in {\mathbb{R}}^{3\times W\times H}$ and ${\tilde{\mathit{M}}}_{3,3}^v\in {\mathbb{R}}^{3\times W\times H}$ in different directions are obtained, respectively. Similarly, other attention maps $\left[{\tilde{\mathit{M}}}_{3,5}^h,{\tilde{\mathit{M}}}_{3,5}^v\right]$, $\left[{\tilde{\mathit{M}}}_{5,3}^h,{\tilde{\mathit{M}}}_{5,3}^v\right]$, and $\left[{\tilde{\mathit{M}}}_{5,5}^h,{\tilde{\mathit{M}}}_{5,5}^v\right]$, can be obtained.

The features in different directions in an image are enhanced using DPCA in both the vertical and horizontal directions. ${\mathit{F}}_3$ is enhanced with attention maps $\left[{\tilde{\mathit{M}}}_{3,3}^h,{\tilde{\mathit{M}}}_{3,3}^v\right]$ and $\left[{\tilde{\mathit{M}}}_{3,5}^h,{\tilde{\mathit{M}}}_{3,5}^v\right]$, respectively, and the enhanced features are ${\tilde{\mathit{F}}}_{3,3}$ and ${\tilde{\mathit{F}}}_{3,5}$:

$$\begin{array}{c}\hfill {\tilde{\mathit{F}}}_{3,3}={\mathit{F}}_3\odot {\tilde{\mathit{M}}}_{3,3}^h\odot {\tilde{\mathit{M}}}_{3,3}^v\\ \hfill {\tilde{\mathit{F}}}_{3,5}={\mathit{F}}_3\odot {\tilde{\mathit{M}}}_{3,5}^h\odot {\tilde{\mathit{M}}}_{3,5}^v\end{array}$$

(6)

where ⊙ denotes point-wise multiplication.

Similarly, ${\mathit{F}}_5$ is enhanced with attention maps $\left[{\tilde{\mathit{M}}}_{5,3}^h,{\tilde{\mathit{M}}}_{5,3}^v\right]$ and $\left[{\tilde{\mathit{M}}}_{5,5}^h,{\tilde{\mathit{M}}}_{5,5}^v\right]$, respectively, and the enhanced features are ${\tilde{\mathit{F}}}_{5,3}$ and ${\tilde{\mathit{F}}}_{5,5}$, respectively. The GPCA map ${\mathit{M}}_{att}$ is generated through the concatenation, convolution and sigmoid activation function of the features ${\tilde{\mathit{F}}}_{3,3}$, ${\tilde{\mathit{F}}}_{3,5}$, ${\tilde{\mathit{F}}}_{5,3}$ and ${\tilde{\mathit{F}}}_{5,5}$:

$${\mathit{M}}_{att}=sigmoid(sconv(concat({\tilde{\mathit{F}}}_{3,3},{\tilde{\mathit{F}}}_{3,5},{\tilde{\mathit{F}}}_{5,3},{\tilde{\mathit{F}}}_{5,5})))$$

(7)

The output ${\mathit{F}}_s^i$ of the i-th RB of the structure feature extraction module is enhanced by the GPCA map ${\mathit{M}}_{att}^i$ ($i=1,2,\dots ,5$) in the i-th GFB, and the output ${\mathit{F}}_t^i$ of the i-th RB of the TDFEM is enhanced by the complementary attention map $1-{\mathit{M}}_{att}^i$. The enhanced structure feature and texture feature are ${\mathit{F}}_{se}^i$ and ${\mathit{F}}_{te}^i$, respectively, as follows:

$$\begin{array}{cc}\hfill & {\mathit{F}}_{se}^i={\mathit{F}}_s^i\odot {\mathit{M}}_{att}^i\hfill \\ \hfill & {\mathit{F}}_{te}^i={\mathit{F}}_t^i\odot (1-{\mathit{M}}_{att}^i)\hfill \end{array}$$

(8)

3.2.2. Cross-Fusion Block

To make full use of the structure and texture detail information enhanced by the DCDPA mechanism, a cross-fusion block (CFB) is proposed, as shown in Figure 3. Specifically, the structural feature ${\mathit{F}}_{se}$ and texture feature ${\mathit{F}}_{te}$ generate the spatial attention map ${\mathit{M}}_s$ through concatenation, convolution and the sigmoid:

$${\mathit{M}}_s=sigmoid\left(sconv(concat({\mathit{F}}_{se},{\mathit{F}}_{te}),k=3)\right)$$

(9)

Using ${\mathit{M}}_s$ to further enhance the structure features, ${\mathit{F}}_{st}$ is obtained by the cross-fusion of the enhanced structure features and texture features. Similarly, ${\mathit{F}}_{ts}$ is obtained by the cross-fusion of the structure features and the enhanced texture results by ${\mathit{M}}_s$:

$$\begin{array}{c}\hfill {\mathit{F}}_{st}={\mathit{F}}_{se}\odot {\mathit{M}}_s+{\mathit{F}}_{te}\\ \hfill {\mathit{F}}_{ts}={\mathit{F}}_{te}\odot {\mathit{M}}_s+{\mathit{F}}_{se}\end{array}$$

(10)

The final fusion result is obtained by the concatenation of ${\mathit{F}}_{st}$ and ${\mathit{F}}_{ts}$:

$${\mathit{F}}_{Gout}=concat({\mathit{F}}_{st},{\mathit{F}}_{ts})$$

(11)

The fusion result ${\mathit{F}}_{Gout}^i$ of the i-th GFB is used as the input of the $i+1$-th GFB, i.e., ${\mathit{F}}_{Gin}^{i+1}={\mathit{F}}_{Gout}^i$.

3.2.3. Feature Reconstruction

To obtain the reconstructed image, the output ${\mathit{F}}_{Gout}^d$ of the last layer in the FHAM and the output ${\mathit{F}}_t^d$ of the last layer in the TDFEM are upsampled by deconvolution to obtain ${\mathit{F}}^h$ and ${\mathit{F}}_t^h$, respectively. The texture detail feature map ${\mathit{F}}_t^h$ is concatenated with ${\mathit{F}}^h$ to obtain features with further enhanced details; the high-resolution image ${\mathit{I}}_{SR}$ is finally reconstructed by $3\times 3$ convolution:

$$\begin{array}{cc}\hfill & {\mathit{F}}^h=Deconv\left({\mathit{F}}_{Gout}^d\right)\hfill \\ \hfill & {\mathit{F}}_t^h=Deconv\left({\mathit{F}}_t^d\right)\hfill \\ \hfill & {\mathit{I}}_{SR}=sconv(concat({\mathit{F}}^h,{\mathit{F}}_t^h),k=3)\hfill \end{array}$$

(12)

where $Deconv$ stands for deconvolution.

3.3. Loss Function

The model is trained by a joint loss including reconstruction loss, gradient loss, and texture loss to achieve good visual performance.

Reconstruction loss: In this paper, the labeled high-resolution image, and the high-resolution image after super-resolution reconstruction, are constrained by the reconstruction loss as follows:

$$L_{re}={\parallel {\mathit{I}}_{HR}-{\mathit{I}}_{SR}\parallel }_1$$

(13)

where ${\mathit{I}}_{HR}$ and ${\mathit{I}}_{SR}$ are the labeled high-resolution image and reconstructed super-resolution image, respectively.

Gradient loss: To ensure that the reconstruction result has the same gradient information as the labeled image, the following gradient loss function to further optimize the reconstructed image is used.

$$L_{grad}=\frac{1}{H\times W}{\parallel \nabla {\mathit{I}}_{HR}-\nabla {\mathit{I}}_{SR}\parallel }_1$$

(14)

where ∇ is the gradient operator and H and W represent the height and width of the image, respectively.

Texture loss: To make the texture features in the reconstructed image and the labeled image consistent, a texture loss is introduced as follows:

$$L_{tex}={\parallel {\mathit{I}}_{HR}^t-{\mathit{I}}_{SR}^t\parallel }_1$$

(15)

where ${\mathit{I}}_{HR}^t$ and ${\mathit{I}}_{SR}^t$ are the texture image extracted from the labeled high-resolution image by the LBP operator and the texture image reconstructed by the texture detail feature extraction module, respectively. So, the joint loss is summarized as follows:

$$L_{joint}=L_{re}+{\lambda }_{grad}{L}_{grad}+{\lambda }_{tex}{L}_{tex}$$

(16)

where ${\lambda }_{grad}$ and ${\lambda }_{tex}$ represent the weight parameters.

4. Experiment and Result Analysis

4.1. Experimental Setup

The Pytorch 1.7 framework and NVIDIA GeForce RTX 2080ti 12 GB GPU were used in the experiments. The training images were from the DIV2K dataset [57], which includes 800 training images, 100 validation images, and 100 test images. Four benchmark datasets, Set5 [58], Set14 [59], B100 [60], Urban100 [61], were used to evaluate the performance of the different methods. The Set5, Set14, and B100 datasets contained 5, 10, and 100 natural images, respectively. There were 100 urban building images in the Urban100 dataset. The high-resolution testing images in the testing set were downsampled by $2\times$, $3\times$ and $4\times$ bicubic interpolation to obtain the corresponding low-resolution images for super-resolution reconstruction testing. When training the network model with a $2\times$ upsampling factor, to reduce memory usage and save running time, the size of the high-resolution image was $256\times 256$ and the size of the low-resolution image was $128\times 128$. When training the network model with a $3\times$ upsampling factor, the size of the high-resolution image was $192\times 192$, and the size of the low-resolution image was $64\times 64$. When training the network model with a $4\times$ upsampling factor, the size of the high-resolution image was $180\times 180$ and the size of the low-resolution image was $45\times 45$. In the training process, the Adam optimizer [62] was selected to train all modules. Following [54], the learning rate of the model training was set to ${10}^{-4}$. The convergence curves of the training loss during the training for scale 2 magnification on the DIV2K dataset are shown in Figure 4. It can be seen from Figure 4 that the training loss gradually decreased as the training epochs increased. When it reached 1800 epochs, the loss curve converged. Allowing for some margin, the epoch for the model training was set to 2000. The parameters of the entire network were trained through back propagation.

Two objective evaluation indicators, the peak signal-to-noise ratio (PSNR) and a structural similarity index measure (SSIM) were used to evaluate the quality of the super-resolution reconstruction results. The PSNR was used to measure the mean-square error between the reconstructed image and the original image; its unit is dB. The larger the value of PSNR, the less the image distortion, that is, the higher the image reconstruction quality. The SSIM was used to measure the structural similarity between the reconstructed image and the original image. A higher value of SSIM indicates that the reconstructed image is closer to the original image, that is, it shows better image reconstruction performance.

Given two images, one being the ground truth (GT) $\mathit{X}$, the other the super-resolution reconstruction result $\mathit{Y}$, the PSNR [63] can be defined as:

$$PSNR=10{log}_{10}(\frac{{(2^n-1)}^2}{MSE})$$

(17)

where $MSE$ is the mean square error of the GT image $\mathit{X}$ and the reconstructed image $\mathit{Y}$. $MSE=\frac{1}{H\times W}{\sum }_{i=1}^H{\sum }_{j=1}^W{(\mathit{X}(i,j)-\mathit{Y}(i,j))}^2$. H and W are the height and width of the image, respectively. n is the bits of a pixel which is generally set to 8 (i.e., a gray level of 256). The unit of the PSNR is the decibel (dB).

The SSIM [64] is defined as:

$$SSIM(x,y)=\frac{(2{\mu }_x{\mu }_y+c_1)(2{\sigma }_{xy}+c_2)}{({\mu }_x^2+{\mu }_y^2+c_1)({\sigma }_x^2+{\sigma }_y^2+c_2)}$$

(18)

where ${\mu }_x$ and ${\mu }_y$ are the mean values of $\mathit{X}$ and $\mathit{Y}$, respectively. ${\sigma }_x$ and ${\sigma }_y$ are the standard deviations of $\mathit{X}$ and $\mathit{Y}$, respectively. ${\sigma }_{xy}$ is the covariance of $\mathit{X}$ and $\mathit{Y}$. $c_1$ and $c_2$ are set to constants to avoid system error caused by a zero denominator. The value of the SSIM is between 0 and 1.

4.2. Quantitative Comparison

To verify the effectiveness of the proposed method, it was compared with state-of-the-art super-resolution methods, such as SRCNN [65], FSRCNN [25], VDSR [22], LapSRN [66], CDC [67], and SREFBN [68]. SRCNN and FSRCNN were trained on the 91-image dataset [17]. VDSR, LapSRN, CDC and SREFBN were trained on the DIV2K dataset. The above methods were all validated on the Set5, Set14, B100, and Urban100 datasets. The quantitative evaluation results are shown in

Table 1. Quantitative comparison of $2\times$ super-resolution results obtained by different methods.

Method	Scale	Set5		Set14		B100		Urban100
		PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
SRCNN	2	36.66	0.9542	32.45	0.9067	31.36	0.8879	29.50	0.8946
FSRCNN	2	37.05	0.9560	32.66	0.9090	31.53	0.8920	29.88	0.9020
VDSR	2	37.53	0.9590	33.05	0.9130	31.90	0.8960	30.77	0.9140
LapSRN	2	37.52	0.9591	33.08	0.9130	31.08	0.8950	30.41	0.9101
CDC	2	32.35	0.8766	29.20	0.7932	28.58	0.7856	26.06	0.7766
SREFBN	2	37.92	0.9593	33.59	0.9175	32.17	0.8991	32.13	0.9279
Proposed	2	38.07	0.9713	33.60	0.9442	32.20	0.9036	32.19	0.8549

,

Table 2. Quantitative comparison of $3\times$ super-resolution results obtained by different methods.

Method	Scale	Set5		Set14		B100		Urban100
		PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
SRCNN	3	32.75	0.9090	29.30	0.8215	28.41	0.7863	26.24	0.7989
FSRCNN	3	33.18	0.9140	29.37	0.8240	28.53	0.7910	26.43	0.8080
VDSR	3	33.67	0.9210	29.78	0.8320	28.83	0.7990	27.14	0.8290
LapSRN	3	33.82	0.9227	29.87	0.8320	28.82	0.7980	27.07	0.8280
CDC	3	25.29	0.2966	23.97	0.6224	24.74	0.6165	21.96	0.6087
SREFBN	3	34.33	0.9257	30.28	0.8407	29.06	0.8038	28.03	0.8494
Proposed	3	35.12	0.9432	30.37	0.8557	29.13	0.7941	28.27	0.7249

and

Table 3. Quantitative comparison of $4\times$ super-resolution results obtained by different methods.

Method	Scale	Set5		Set14		B100		Urban100
		PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
SRCNN	4	30.48	0.8628	27.50	0.7513	26.90	0.7101	24.52	0.7221
FSRCNN	4	30.72	0.8660	27.61	0.7550	26.98	0.7150	24.62	0.7280
VDSR	4	31.35	0.8830	28.02	0.7680	27.29	0.7026	25.18	0.7540
LapSRN	4	31.54	0.8850	28.19	0.7720	27.32	0.7270	25.21	0.7560
CDC	4	25.41	0.7177	24.01	0.6165	24.19	0.6085	21.50	0.5610
SREFBN	4	32.01	0.8917	28.52	0.7790	27.50	0.7332	25.87	0.7787
Proposed	4	32.27	0.9039	28.59	0.7624	27.60	0.7075	26.13	0.7893

with the best results marked in bold. According to Table 1, Table 2 and Table 3, the proposed method achieved most of the best evaluation results.

4.3. Qualitative Comparison

To compare the visual quality of the super-resolution reconstruction results obtained by different methods, images from the Set5, Set14, BSD100, and Urban100 datasets were selected for the visual display of the $2\times$, $3\times$, and $4\times$ super-resolution reconstruction results and ground truth (GT), as shown in Figure 5, Figure 6 and Figure 7. In terms of visual performance, the detailed textures of the reconstructed images obtained by the SRCNN, FSRCNN, VDSR, and CDC methods were relatively blurred, while the reconstruction results obtained by the LapSRN, CDC, and SREFBN methods contained detailed features that did not match the actual texture details. The proposed method could fit real natural images well, enhancing the brightness of reconstructed high-resolution images, while recovering textures, so that the reconstructed images had rich texture details. In summary, the proposed method has certain advantages in terms of visual performance and quantitative evaluation.

4.4. Ablation Study

To verify the effectiveness of the proposed TDFEM, DCDPA, and CFB methods on the quality of the reconstructed super-resolution images, ablation experiments were performed. Taking the network without DCDPA, CFB and TDFEM as the benchmark (Base), the Base network plus TDFEM was marked as Base + TDFEM, the Base + TDFEM network plus DCDPA was marked as Base + TDFEM + DCDPA, and the Base + TDFEM + DCDPA network plus CFB was marked as Base+TDFEM+DCDPA+CFB. Ablation experiments were performed on the images from the Urban100 dataset at $2\times$ magnification. The quantitative evaluation results of the ablation experiments are shown in

Table 4. Ablation experiment performance.

Method	Base	TDFEM	DCDPA	CFB	PSNR/SSIM
Base	√				31.38/0.8333
Base + TDFEM	√	√			31.51/0.8456
Base + TDFEM + DCDPA	√	√	√		31.85/0.8509
Base + TDFEM + DCDPA + CFB	√	√	√	√	32.19/0.8549

and the corresponding visual performance is shown in Figure 8.

4.4.1. The Effectiveness of TDFEM

TDFEM was used to extract texture information separately to prevent the loss of texture detail information in the process of super-resolution reconstruction, which led to the deterioration in subjective image quality. So, TDFEM was introduced into the proposed model. In the TDFEM, the LBP operator is used to extract the texture details of the original image and the obtained texture detail map is concatenated with the original image, which enhances the network’s ability to extract texture detail information. This helps to improve the perceptual quality of reconstructed super-resolution images. To demonstrate its effectiveness, the reconstruction results of the ‘Base’ model were compared with the reconstruction results of the ‘Base + TDFEM’ model. As shown in Figure 8, the ‘Base + TDFEM’ model achieved good performance in texture detail extraction during the reconstruction process. Additionally, as shown in Table 4, the utilization of ‘TDFEM’ also led to a significant improvement in the objective evaluation results, which confirmed the effectiveness of ‘TDFEM’ in the proposed model.

4.4.2. The Effectiveness of DCDPA

In the proposed model, DCDPA is used to capture the structure and texture feature regions of interest and to prevent the loss of feature information. This attention mechanism makes full use of the complementarity and directionality of multi-scale image features and effectively avoids information loss and the possible distortion of image structure and texture details during image reconstruction. This is conducive to improving the visual quality of the reconstructed image. To verify the effectiveness of ‘DCDPA’, the reconstruction results of the ‘Base + TDFEM + DCDPA’ model were compared with those of the ‘Base + TDFEM’ model. In Figure 8, the structure and texture details of the reconstructed images obtained by the ‘Base + TDFEM + DCDPA’ are highlighted. Moreover, according to the values of the objective evaluation metrics shown in Table 4, compared with ‘Base + TDFEM’, the performance of ‘Base + TDFEM + DCDPA’ was improved. So, the effectiveness of DCDPA was demonstrated.

4.4.3. The Effectiveness of CFB

To prevent the loss of texture details and ensure that the texture detail information enhanced by the DCDPA was fully utilized by the later layers, a CFB was proposed. This module facilitates the information interaction between structure and texture detail features by cross-fusion of them to enhance the network’s ability to mine structure and texture features. This contributes to improving the quality of the reconstructed image. To verify the effectiveness of ‘CFB’, the reconstruction results of ‘Base + TDFEM + DCDPA + CFB’ were compared with those of ‘Base + TDFEM + DCDPA’. As shown in Figure 8, the ‘Base + TDFEM + DCDPA + CFB’ model more fully integrated structure and texture features in the reconstruction process and the corresponding evaluation results presented in Table 4 were also enhanced.

4.5. Parameter Analysis

4.5.1. Influence of the Number of GFBs

This section discusses the influence of the number of GFBs on the performance of the proposed model. The proposed model with different numbers of GFBs was tested on the DIV2K dataset.

Table 5. Performance of the model with different numbers of GFBs.

GFB Modules	n = 3	n = 5	n = 7	n = 9
PSNR	37.10	38.07	37.40	37.24
SSIM	0.9674	0.9713	0.9688	0.9676
Params (M)	18.4	21.1	26.1	30.2

shows the performance of the model with different numbers of GFBs (denoted by n). ’Params’ indicates the number of parameters. As shown in Table 5, PSNR and SSIM achieved the best performance at $n=5$. When n was 7 or 9, the number of model parameters increased and the model’s performance decreased. Therefore, n was set to 5.

4.5.2. Hyper-Parameter Selection

In Equation (16), two hyper-parameters ${\lambda }_{grad}$ and ${\lambda }_{tex}$ need to be set. The impact of one hyper-parameter is analyzed by fixing another hyper-parameter. For the DIV2K dataset, the impact of ${\lambda }_{grad}$ and ${\lambda }_{tex}$ on the super-resolution reconstruction results was analyzed by PSNR and SSIM, respectively; the corresponding experimental results are shown in Figure 9. As shown in Figure 9, when ${\lambda }_{grad}=0.01$ and ${\lambda }_{tex}=0.001$, both PSNR and SSIM achieved the optimal values. Therefore, ${\lambda }_{grad}=0.01$ and ${\lambda }_{tex}=0.001$ were set for model training.

5. Conclusions

A super-resolution reconstruction network that preserves structure and texture details was designed. The proposed network uses different network branches for feature extraction of the image structure and texture details, respectively, and enhances these two features layer-by-layer through the FHAM to improve the network’s ability to extract features. To fully extract the information of different directions in the image, a dual-coordinate direction perception attention mechanism is proposed. The proposed attention mechanism fully considers the complementarity and directionality of the multi-scale image features, which can avoid information loss and possible distortion of the image structure and texture details. Additionally, a cross-fusion mechanism to effectively integrate the structure and texture features extracted by the network was designed, which can enhance the network’s ability to mine structure and texture features. This is helpful to improve the quality of the reconstructed image. The experimental results show that the super-resolution reconstruction results of the proposed method produced good structure and texture details. The proposed method outperformed state-of-the-art methods in both visual quality and performance evaluation. However, the proposed method uses synthetic datasets to train networks in a supervised way. In the synthetic datasets, a low-resolution image is generated by bicubic downsampling its high-resolution counterpart. In practical scenarios, the high-resolution images are unavailable and image degradation models are unknown. Therefore, how to improve the quality of super-resolution reconstructed images in real scenes will be the direction of our future research.