No-Reference Image Quality Assessment with Convolutional Neural Networks and Decision Fusion

Abstract

No-reference image quality assessment (NR-IQA) has always been a difficult research problem because digital images may suffer very diverse types of distortions and their contents are extremely various. Moreover, IQA is also a very hot topic in the research community since the number and role of digital images in everyday life is continuously growing. Recently, a huge amount of effort has been devoted to exploiting convolutional neural networks and other deep learning techniques for no-reference image quality assessment. Since deep learning relies on a massive amount of labeled data, utilizing pretrained networks has become very popular in the literature. In this study, we introduce a novel, deep learning-based NR-IQA architecture that relies on the decision fusion of multiple image quality scores coming from different types of convolutional neural networks. The main idea behind this scheme is that a diverse set of different types of networks is able to better characterize authentic image distortions than a single network. The experimental results show that our method can effectively estimate perceptual image quality on four large IQA benchmark databases containing either authentic or artificial distortions. These results are also confirmed in significance and cross database tests.

1. Introduction

The aim of image quality assessment (IQA) is predicting digital images’ perceived quality that is consistent with the human visual system’s (HVS) perception [1]. The most obvious way to evaluate the quality of digital images is a subjective user study where quality scores are obtained from human observers in a laboratory environment [2] or a crowdsourcing experiment [3]. However, such experiments are expensive, laborious, and time consuming. Thus, they cannot be applied in a real-time system. On the other hand, they result in publicly available benchmark IQA databases, such as KonIQ-10k [4] or SPAQ [5], which consist of images with their quality scores. In most cases, quality scores are given as mean opinion scores (MOS), which are arithmetic averages of all quality opinions given by individual observers. In contrast, the aim of objective IQA is constructing computational, mathematical models that can estimate digital images’ human perceptual quality.

Traditionally, objective IQA is divided into three branches in the literature based on the availability of the distortion-free reference images. The first branch is no-reference image quality assessment (NR-IQA) where no information is available about the reference images. Contrarily, full-reference image quality assessment (FR-IQA) methods have full access to the reference images, while reduced-reference image quality assessment (RR-IQA) models have only partial information about the reference images.

Recently, deep learning has dominated the field of image processing and computer vision. The field of objective IQA has not been an exception to this trend. The goal of this study is to introduce a novel deep learning-based NR-IQA method for authentic distortions that relies on decision fusion of multiple quality scores coming from different types of convolutional neural network architectures. The proposed decision fusion-based model allows us to better characterize authentic image distortions due to the diverse set of different types of networks.

The rest of this section is organized as follows. First, related and previous work is reviewed in Section 1.1. Second, the main contributions of this study are enumerated in Section 1.2 in light of previous work. Third, the structure of the paper is defined in Section 1.3.

1.1. Related Work

In recent decades, a lot of NR-IQA methods have been published in the literature. NR-IQA models can be divided into two categories, such as distortion-specific and general methods. Specifically, distortion-specific methods assume a certain distortion type (e.g., blur [6] or JPEG compression noise [7]) present in the image, while general algorithms do not make any assumptions about the noise or distortion type present in the image. Early NR-IQA methods mainly fell into the distortion-specific category. On the other hand, general NR-IQA algorithms can be divided into three groups, i.e., natural scene statistics (NSS)-based, HVS-based, and learning-based ones.

1.1.1. NSS-Based Methods

The main assumption behind NSS-based methods is that natural images of high quality have certain statistical features which lose strength in the presence of image noise and degradation. A typical NSS-based algorithm has three distinct stages, i.e., feature extraction, NSS modeling and regression [8]. A representative example of NSS-based methods was proposed by Moorthy and Bovik [9]. Namely, the statistical properties of wavelet coefficients were utilized as quality-aware features using a Gaussian scale mixture model in the authors’ work. Another well-known example is the blind image quality index (BIQI) [10] which works with wavelet transform to decompose an input image into three scales and orientations. Next, generalized Gaussian distributions (GGD) were fitted onto the wavelet coefficients and the mean, the variance and the shape parameters of the GGDs were considered as quality-aware features. These were mapped onto perceptual quality scores with a trained support vector regressor (SVR). NSS-based methods have been proposed in other transform domains, i.e., curvelet [11], contourlet [12], quaternion wavelet transform [13] and discrete cosine transform [14] and in the spatial domain [15], as well.

1.1.2. HVS-Based Methods

In contrast to NSS-based methods, HVS-based approaches try to utilize one or more computational aspects of the HVS to devise an NR-IQA algorithm. For example, Zhai et al. [16] employed the free-energy principle [17] and considered the visual quality assessment of an image as an active inference process. Similarly, Gu et al. [18] applied the free-energy principle but it was improved by structural and gradient information. Li et al. [19] utilized the observation of previous studies [20,21] that the HVS is conformed to extract structural information from images. Thus, the authors introduced a special variant of local binary pattern [22] operators on the luminance map to describe the structural information. In [23], the authors took the same assumption, but first- and second-order image structures were extracted to characterize the structural information of an image. Namely, Prewitt filters [24] were applied for first-order image structure extraction, while local contrast normalization [25] was used for second-order image structure extraction. On the other hand, Saha and Wu [26] built on redundancy and scale invariance properties of natural images [27].

1.1.3. Learning-Based Methods

Learning-based methods heavily rely on data-driven techniques, such as deep learning. First, Kang et al. [28] applied convolutional neural network (CNN) for NR-IQA. Namely, the authors trained a CNN of five layers from scratch on locally normalized $32\times 32$ image patches. The overall quality estimation was obtained by averaging the image patches’ quality scores. Similarly, Bosse et al. [29] trained a patch-based CNN, but a patch weighting scheme was also developed to reflect the importance of individual patches in quality estimation. Jia and Zhang [30] assigned weights to image patches using image saliency. In [31], the predicted quality scores of image patches were directly weighted by local image saliency. In contrast, Chetouani and Li [32] proposed a specific scanpath predictor, which mimics the movements of the human eye [33], for image patch selection. Fan et al. [34] developed multi-expert CNNs for evaluating the perceptual quality of artificially distorted images. Namely, the authors trained a CNN for each possible distortion type (JPEG, JPEG2000, Gaussian, etc.) of an IQA benchmark database and a distortion identification network. The outputs of the previously mentioned networks were combined to obtain an input image’s perceptual quality. Another line of works deals with deep feature extraction for NR-IQA [35,36,37,38,39,40]. Namely, the first layers of a CNN usually capture elemental image features, such as blobs, edges, boundaries and color patterns, which are quality-aware features [41]. First, Bianco et al. [35] applied deep features extracted from pretrained CNNs for perceptual image quality estimation. Namely, random patches were sampled from an input image and deep features were derived from each patch. Subsequently, the extracted deep feature vectors were pooled together and mapped onto quality scores with a trained SVR. In [36], the inception modules [42] of a CNN and global average pooling layers were used to compile deep features which were mapped onto perceptual quality with an SVR. In contrast, Chetouani et al. [39] combined viewing distance information with deep features to obtain a better performance. Su and Korhonen [43] introduced weighted spatial pooling to aggregate arbitrarily sized spatial feature maps into a feature vector which is mapped onto quality scores with the help of a fully connected layer. In [40,44], a multi-scale representation was constructed from deep features to get an effective image representation for NR-IQA. In contrast, Korhonen et al. [45] utilized a recurrent neural network for spatial pooling of deep features. Ying et al. [46] applied the pooling of regions’ of interest features and the training across multiple IQA benchmark databases.

1.2. Contributions

The main contributions of this study are as follows: (i) decision fusion of deep convolutional architectures for NR-IQA. (ii) The usage of a diverse set of pretrained convolutional architectures to better characterize natural images’ authentic distortions. (iii) Extensive evaluation of different deep architectures on large IQA benchmark databases.

1.3. Structure

The remainder of this study is organized as follows. Section 2 can be divided into two main parts. First, the tools applied in our experiments are introduced. Next, the proposed method is presented. Section 3 gives the experimental results and performance analysis of the proposed approach. Moreover, it presents a comparison to other state-of-the-art methods. Finally, a conclusion is drawn in Section 4 with possible directions of future research.

2. Materials and Methods

In this section, the tools applied in our experiments and the proposed method are described. Specifically, Section 2.1 reports on the applied publicly available databases, evaluation protocol and implementation details. In Section 2.2, the proposed method is introduced in detail.

2.1. Materials

2.1.1. Databases

As already mentioned, a number of benchmark IQA databases have been made publicly available in the recent decades [8,47,48,49]. In this study, we deal with the quality assessment of natural images containing authentic distortions. To evaluate and compare our proposed method to the state of the art, three large databases—LIVE In the Wild (CLIVE) [50], KonIQ-10k [4] and SPAQ [5]—with authentic distortions have been chosen. Moreover, one database—TID2013 [51]—with artificial distortions was also utilized in our experiments. In the following, a brief overview is given about the previously mentioned IQA benchmark databases.

Ghadiyaram et al. [50] collected 1162 images with unique scenes captured by a variety of mobile device cameras. Subsequently, these images were evaluated in a crowdsourcing experiment to obtain MOS values (ranging from 0 to 100). Similarly to CLIVE, the images of KonIQ-10k [4] were evaluated in a crowdsourcing experiment. An important difference between CLIVE [50] and KonIQ-10k [4] is that the images of KonIQ-10k were collected from the YFCC-100m [52] multimedia database using a two-stage filtering process. First, those images were discarded which did not have a creative commons license. Second, such a filtering process was applied that enforces the relative uniform distribution of eight image attributes, i.e., bitrate, resolution, JPEG quality, brightness, colorfulness, contrast, sharpness and deep feature. The MOS range in KonIQ-10k [4] spans from 1 to 5. Fang et al. [5] collected 11,125 high-resolution digital images captured by 66 smartphones. Subsequently, the collected images were evaluated in a laboratory environment involving 600 subjects. Moreover, the MOS values of each image have been determined based on at least 15 quality scores. In contrast to the previously mentioned databases, TID2013 [51] contains images with artificial distortions. Specifically, it contains a set of 25 pristine reference images whose perceptual quality are considered perfect. From these reference images, 3000 distorted images are derived using 24 different types of distortions at five distortion levels.

The empirical MOS distributions of CLIVE [50], KonIQ-10k [4], SPAQ [5] and TID2013 [51] are depicted in Figure 1. The main characteristics of the applied IQA databases are summarized in

Table 1. Overview about the applied publicly available IQA benchmark databases. In this table, we denoted by ‘-’ if the data was not available.

Attribute	CLIVE [50]	KonIQ-10k [4]	SPAQ [5]	TID2013 [51]
Year	2015	2018	2020	2013
Number of images	1169	10,073	11,125	3000
Number of scenes	1169	10,073	11,125	25
Distortion type	authentic	authentic	authentic	artificial
Subjective framework	crowd-sourcing	crowd-sourcing	laboratory	laboratory
Number of annotators	8000	350,000	600	540
Number of annotations	1400	1,200,000	186,400	524,340
Resolution	$500\times 500$	$1024\times 768$	∼$4000\times 4000$	$512\times 384$
MOS range	0–100	1–5	0–100	0–9

. Figure 2 and Figure 3 depict several sample images from KonIQ-10k and TID2013 databases, respectively.

2.1.2. Evaluation Protocol and Implementation Details

The evaluation of NR-IQA algorithms is based on the correlation strength measured between predicted and ground-truth scores. To characterize the correlation strength, the Pearson linear correlation coefficient (PLCC), Spearman rank order correlation coefficient (SROCC) and Kendall rank order correlation coefficient (KROCC) are widely applied in the literature [8]. Moreover, before the calculation of PLCC, a non-linear mapping is carried out between the predicted and ground-truth scores using a logistic function with 5 parameters, as recommended by Sheikh et al. [53]. Formally, this non-linear mapping can be given as

$$Q={\beta }_1\left(\frac{1}{2}-\frac{1}{e^{-{\beta }_2(Q_p-{\beta }_3)}}\right)+{\beta }_4{Q}_p+{\beta }_5,$$

(1)

where ${\beta }_i$’s $(i=1,\dots ,5)$ are the fitting parameters. Moreover, $Q_p$ and Q denote the predicted and mapped quality scores, respectively. Between vectors $\mathbf{x}$ and $\mathbf{y}$ of the same length N, PLCC is defined as

$$PLCC(\mathbf{x},\mathbf{y})=\frac{{\sum }_{i=1}^N\left((x_i-\overline{x})(y_i-\overline{y})\right)}{\sqrt{{\sum }_{i=1}^N{(x_i-\overline{x})}^2{\sum }_{i=1}^N{(y_i-\overline{y})}^2}},$$

(2)

where $x_i$, $y_i$ denote the ith element of $\mathbf{x}$, $\mathbf{y}$, respectively. Moreover, $\overline{x}=\frac{1}{N}{\sum }_{i=1}^N{x}_i$ and $\overline{y}=\frac{1}{N}{\sum }_{i=1}^N{y}_i$. Between these vectors, SROCC is defined as

$$SROCC(\mathbf{x},\mathbf{y})=1-\frac{6{\sum }_{i=1}^N{d}_i^2}{N(N^2-1)},$$

(3)

where $d_i$ stands for the difference between the two ranks of each $x_i$ and $y_i$. Moreover, the following formula is used to determine KROCC

$$KROCC(\mathbf{x},\mathbf{y})=\frac{C-D}{\left(\genfrac{}{}{0pt}{}{N}{2}\right)},$$

(4)

where C is the number of concordant pairs between $\mathbf{x}$ and $\mathbf{y}$, while D is the number of discordant pairs.

As is common in the literature [8], we report on median PLCC, SROCC and KROCC values measured over 100 random train-validation-test splits. Since the median is more robust to outliers, it is preferred in the literature to arithmetic mean. More precisely, an IQA benchmark database containing authentic distortions (CLIVE [50], KonIQ-10k [4] and SPAQ [5]) was randomly divided into a training set with approximately 70% of images, a validation set with 10% of images, and a test set with the remaining 20%. Since TID2013 [51] consists of images with artificial distortions, it was divided with respect to the reference images to avoid any semantic content overlap between the train, validation and test sets.

The proposed method was implemented in Python 3.6.1 programming language using the Keras deep learning library [54] and the Keras utilities (Kuti) library. Moreover, the proposed method was trained on two NVidia Titan Xp GPUs.

2.2. Methods

The high-level overview of the proposed end-to-end method is depicted in Figure 4. An input image to be evaluated is passed through six in-parallel state-of-the-art CNN bodies pretrained on ImageNet [55] database for image classification, such as VGG16 [56], ResNet50 [57], InceptionV3 [58], InceptionResNetV2 [59], DenseNet201 [60] and NASNetMobile [61], succeeded by global average pooling layers and regression heads. The main characteristics of the applied CNNs are summarized in

Table 2. On ImageNet [62] database pretrained CNNs applied in this study. In this table, Top-1 and Top-5 accuracy refers to the model’s performance on the ImageNet [55] validation dataset.

CNN	Size (MByte)	Top-1 Accuracy	Top-5 Accuracy	Parameters	Depth
VGG16 [56]	528	0.713	0.901	138,357,544	23
ResNet50 [57]	98	0.749	0.921	25,636,712	-
InceptionV3 [58]	92	0.779	0.937	23,851,784	159
InceptionResNetV2 [59]	215	0.803	0.953	55,873,736	572
DenseNet201 [60]	80	0.773	0.936	20,242,984	201
NASNetMobile [61]	23	0.774	0.919	5,326,716	-

. It is important to note that the fully connected output layers of these models were not utilized in this work, allowing us to attach new layers to them and train them for NR-IQA. Moreover, the pretrained CNN body term is reserved for a deep model without the fully connected output layers in our terminology.

The main idea behind global average pooling (illustrated in Figure 5) is to enforce correspondences between feature maps and the number of predicted semantic categories by taking the average of each feature map and compiling a feature vector. Thus, arbitrary sized images can be put to the input of a CNN. Other advantages are that overfitting can be avoided with this layer and robustness can be increased against spatial translations of the input [63].

The outputs of the global average pooling layers are connected to six regression heads containing four separately fully connected layers. The inner structure of the regression heads is depicted in Figure 6. Specifically, the first three layers consists of 2048, 1024 and 256 nodes, respectively. To avoid overfitting, these layers possess dropout rates [64] of 0.25, 0.25 and 0.5, respectively. To ensure better gradient propagation, rectified linear units (ReLU) were applied as activation functions in the first three layers [62]. Since the last layer predicts the quality, it contains only one neuron and applies a simple linear activation function. Let $(\mathbf{I},q)$ denote the training dataset, where $\mathbf{I}$ is an RGB image of arbitrary size and q is the ground-truth quality score belonging to $\mathbf{I}$. Moreover, we denote by $\widehat{q}$ the quality score predicted by one regression head. Since image quality prediction is formulated as a regression task here, mean squared error (MSE) loss is applied, which—as the name suggests—takes the mean of squared differences between the ground-truth and predicted values. Formally, this can be written as: $L_{MSE}={(q-\widehat{q})}^2$.

The six branches of the proposed fused architecture are trained using the same training and validation datasets but independently from each other. Specifically, a pretrained CNN body plus regression head structure was trained using the Adam [65] optimizer with the default parameters of the Keras [54] deep learning library $({\beta }_1=0.9,{\beta }_2=0.999)$ and a batch size of 16. First, a model is trained through 40 epochs with a learning rate of $1\times {10}^{-4}$. During training, the PLCC computed between the predicted and ground-truth quality scores on the validation set is monitored and after 40 epochs the best performing option is restored. Second, this restored model is trained through 20 epochs with a learning rate of $5\times {10}^{-5}$. Similarly to the previous step, PLCC on the validation set is constantly monitored during the training process, and after 20 epochs, the best performing one is reloaded.

The quality predictions of the six branches are fused together to obtain an overall quality score for the input image. Specifically, average and median pooling were applied in this work. A parameter study was conducted to ascertain which is the better option (presented in Section 3.1).

3. Experimental Results and Analysis

In this section, detailed experimental results and analysis are presented. More specifically, Section 3.1 introduces a parameter study regarding the proposed method so that the justification of the proposed method’s design choices is possible. Next, a comparison to the state of the art is shown on the CLIVE [50] and KonIQ-10k [4] databases in Section 3.2.

3.1. Parameter Study

First, we examine the performance of the individual models which consist of a pretrained CNN body, global average pooling layer and a regression head. Second, we compare the effects of two different fusion of quality subscores, i.e., average and median pooling. The evaluation protocol is exactly the same as described in Section 2.1. The results obtained on KonIQ-10k [4] are summarized in

Table 3. Base CNN and decision fusion comparison on KonIQ-10k [4]. Median SROCC values were measured over 100 random train–validation–test splits.

Architecture	SROCC
VGG16	0.861
ResNet50	0.860
InceptionV3	0.909
InceptionResNetV2	0.918
DenseNet201	0.914
NASNetMobile	0.899
Average pooling of subscores	0.927
Median pooling of subscores	0.931

. From these results, it can be concluded that using an optional pretrained CNN body results in a rather strong correlation between the predicted and ground-truth quality scores. On the other hand, the performance of VGG16 [56] and ResNet50 [57] lags behind those of the other pretrained networks. Further, regarding the decision fusion of different network types, both average and median pooling are able to increase the prediction performance substantially. Since median pooling outperforms average pooling, as one can see in Table 3, this was used in our NR-IQA method for a comparison to other state-of-the-art algorithms.

3.2. Comparison to the State of the Art

To compare the proposed method to the state of the art, several NR-IQA methods have been collected, such as BLIINDER [66], DeepRN [67], BLIINDS-II [14], BMPRI [68], BRISQUE [15], CurveletQA [11], DIIVINE [9], ENIQA [69], GRAD-LOG-CP [70], MultiGAP- NRIQA [36], MSDF-IQA [40], NBIQA [71], PIQE [72], OG-IQA [73], SSEQ [74] and UNIQUE [75], whose original source codes are publicly available online. Excluding PIQE [72], which is an opinion-unaware method without a training step, these methods were evaluated using the same principle as the proposed method’s on the applied IQA benchmark databases (CLIVE [50], KonIQ-10k [4], SPAQ [5] and TID2013 [51]). As already mentioned, the median PLCC, SROCC and KROCC values are reported, which were measured over 100 random train–validation–test splits. Specifically, approximately 80% of images were in the training set, while the remaining 20% were in the test set, if a validation set is not required for an NR-IQA method. If a validation set is needed, then 70% of the images were in the training set, 10% in the validation set and 20% in the test set. Since TID2013 [51] contains images with artificial distortions, it was divided with respect to the reference images. Moreover, the performance results of several state-of-the-art deep learning-based methods, such as CONTRIQUE [76], DB-CNN [77], DeepFL-IQA [78], KonCept512 [79], MLSP [78,80], PaQ-2-PiQ [46], PQR [81] and WSP [43], were copied from the authors’ corresponding papers and added to our comparison. It is important to note that our proposed method was trained on down-sampled versions of KonIQ-10k [4] ($512\times 384$ resolution instead of $1024\times 768$) and SPAQ [5] ($2000\times 2000$ resolution instead of ∼$4000\times 4000$) due to graphics processing unit (GPU) memory limitations and to accelerate the training process. Further, our proposed method is codenamed as DF-CNN-IQA throughout the comparative tables published in this paper.

The experimental results obtained on CLIVE [50] and KonIQ-10k [4] are summarized in

Table 4. Comparison of DF-CNN-IQA to the state of the art on CLIVE [50] and KonIQ-10k [4]. Median PLCC, SROCC and KROCC values were measured over 100 random train–validation–test splits. The best results are typed in bold, and the second-best results are underlined. We denoted by ‘-’ if the data was not available.

	CLIVE [50]			KonIQ-10k [4]
Method	PLCC	SROCC	KROCC	PLCC	SROCC	KROCC
BLIINDER [66]	0.782	0.763	0.576	0.876	0.864	0.668
DeepRN [67]	0.784	0.753	0.579	0.866	0.880	0.666
BLIINDS-II [14]	0.473	0.442	0.291	0.574	0.575	0.414
BMPRI [68]	0.541	0.487	0.333	0.637	0.619	0.421
BRISQUE [15]	0.524	0.497	0.345	0.707	0.677	0.494
CurveletQA [11]	0.636	0.621	0.421	0.730	0.718	0.495
DIIVINE [9]	0.617	0.580	0.405	0.709	0.693	0.471
ENIQA [69]	0.596	0.564	0.376	0.761	0.745	0.544
GRAD-LOG-CP [70]	0.607	0.604	0.383	0.705	0.696	0.501
MultiGAP-NRIQA [36]	0.841	0.813	0.626	0.915	0.911	0.732
MSDF-IQA [40]	0.831	0.801	0.607	0.901	0.885	0.703
NBIQA [71]	0.629	0.604	0.427	0.771	0.749	0.515
PIQE [72]	0.172	0.108	0.081	0.208	0.246	0.172
OG-IQA [73]	0.545	0.505	0.364	0.652	0.635	0.447
SSEQ [74]	0.487	0.436	0.309	0.589	0.572	0.423
UNIQUE [75]	0.891	0.855	0.633	0.900	0.897	0.664
CONTRIQUE [76]	0.857	0.845	-	0.906	0.894	-
DB-CNN [77]	0.869	0.851	-	0.884	0.875	-
DeepFL-IQA [78]	0.769	0.734	-	0.887	0.877	-
KonCept512 [79,82]	0.848	0.825	-	0.937	0.921	-
MLSP [78,80]	0.769	0.734	-	0.887	0.877	-
PaQ-2-PiQ [46]	0.850	0.840	-	0.880	0.870	-
PQR [81]	0.882	0.857	-	0.884	0.880	-
WSP [43]	-	-	-	0.931	0.918	-
DF-CNN-IQA	0.859	0.849	0.630	0.949	0.931	0.738

. From these results, it can be observed that the proposed method outperforms all the other considered methods on KonIQ-10k [4] by a large margin. Specifically, it outperforms the second- and third-best methods by approximately 0.01 in terms of PLCC and SROCC. On the smaller CLIVE, the proposed method gives the second-best results among the methods which were tested by us. According to our experiments, the UNIQUE [75] method was able to exceed DF-CNN-IQA.

The experimental results obtained on SPAQ [5] and TID2013 [51] are summarized in

Table 5. Comparison of DF-CNN-IQA to the state of the art on SPAQ [5] and TID2013 [51]. Median PLCC, SROCC and KROCC values were measured over 100 random train–validation–test splits. The best results are typed in bold, and the second-best results are underlined. We denoted by ‘-’ if the data were not available.

	SPAQ [5]			TID2013 [51]
Method	PLCC	SROCC	KROCC	PLCC	SROCC	KROCC
BLIINDER [66]	0.872	0.869	0.683	0.834	0.816	0.720
DeepRN [67]	0.870	0.850	0.676	0.745	0.636	0.560
BLIINDS-II [14]	0.676	0.675	0.486	0.558	0.513	0.339
BMPRI [68]	0.739	0.734	0.506	0.701	0.588	0.427
BRISQUE [15]	0.726	0.720	0.518	0.478	0.427	0.278
CurveletQA [11]	0.793	0.774	0.503	0.553	0.505	0.359
DIIVINE [9]	0.774	0.756	0.514	0.692	0.599	0.431
ENIQA [69]	0.813	0.804	0.603	0.604	0.555	0.397
GRAD-LOG-CP [70]	0.786	0.782	0.572	0.671	0.627	0.470
MultiGAP-NRIQA [36]	0.909	0.903	0.693	0.710	0.433	0.302
MSDF-IQA [40]	0.900	0.894	0.692	0.727	0.448	0.311
NBIQA [71]	0.802	0.793	0.539	0.723	0.628	0.427
PIQE [72]	0.211	0.156	0.091	0.464	0.365	0.257
OG-IQA [73]	0.726	0.724	0.594	0.564	0.452	0.321
SSEQ [74]	0.745	0.742	0.549	0.618	0.520	0.375
UNIQUE [75]	0.907	0.906	0.687	0.812	0.826	0.578
CONTRIQUE [76]	0.919	0.914	-	0.857	0.843	-
DB-CNN [77]	0.915	0.914	-	0.865	0.816	-
DeepFL-IQA [78]	-	-	-	0.876	0.858	-
KonCept512 [79]	-	-	-	-	-	-
MLSP [78,80]	-	-	-	-	-	-
PaQ-2-PiQ [46]	-	-	-	-	-	-
PQR [81]	-	-	-	0.798	0.740	-
WSP [43]	-	-	-	-	-	-
DF-CNN-IQA	0.921	0.915	0.693	0.743	0.709	0.496

. From these results, it can be observed that DF-CNN-IQA excels the performance of all the other considered NR-IQA algorithms on SPAQ [5], while it performs weaker than the best state-of-the-art method on TID2013 [51], which in contrast to SPAQ [5] contains artificial distortions.

The direct and weighted averages obtained on these IQA databases are summarized in

Table 6. Comparison of the proposed DF-CNN-IQA to the state of the art. The weighted and direct average of measured PLCC and SROCC values are reported. The best results are typed in bold, and the second-best results are underlined. We denoted by ‘-’ if the data were not available.

	Direct Average			Weighted Average
Method	PLCC	SROCC	KROCC	PLCC	SROCC	KROCC
BLIINDER [66]	0.841	0.828	0.662	0.865	0.856	0.676
DeepRN [67]	0.816	0.780	0.620	0.850	0.832	0.654
BLIINDS-II [14]	0.570	0.551	0.383	0.612	0.605	0.431
BMPRI [68]	0.655	0.607	0.422	0.685	0.660	0.455
BRISQUE [15]	0.609	0.580	0.409	0.680	0.658	0.472
CurveletQA [11]	0.678	0.655	0.445	0.732	0.713	0.479
DIIVINE [9]	0.698	0.657	0.455	0.731	0.704	0.482
ENIQA [69]	0.694	0.667	0.480	0.758	0.740	0.545
GRAD-LOG-CP [70]	0.692	0.677	0.481	0.732	0.721	0.523
MultiGAP-NRIQA [36]	0.844	0.765	0.588	0.885	0.846	0.659
MSDF-IQA [40]	0.840	0.757	0.578	0.877	0.833	0.647
NBIQA [71]	0.731	0.694	0.477	0.772	0.747	0.511
PIQE [72]	0.264	0.219	0.150	0.238	0.214	0.142
OG-IQA [73]	0.622	0.579	0.432	0.669	0.646	0.493
SSEQ [74]	0.610	0.568	0.414	0.656	0.634	0.467
UNIQUE [75]	0.878	0.871	0.641	0.892	0.891	0.662
CONTRIQUE [76]	0.885	0.874	-	0.904	0.894	-
DB-CNN [77]	0.883	0.864	-	0.894	0.884	-
DeepFL-IQA [78]	-	-	-	-	-	-
KonCept512 [79]	-	-	-	-	-	-
MLSP [78,80]	-	-	-	-	-	-
PaQ-2-PiQ [46]	-	-	-	-	-	-
PQR [81]	-	-	-	-	-	-
WSP [43]	-	-	-	-	-	-
DF-CNN-IQA	0.868	0.851	0.639	0.908	0.894	0.685

. It can be seen that the direct average PLCC, SROCC and KROCC values of the proposed method are larger than those of the majority of the considered state-of-the-art methods. If we take a look at the weighted average by the number of images found in the applied IQA benchmark databases, the difference between the proposed method and the other methods is even more significant. Specifically, DF-CNN-IQA can outperform all the other algorithms in this regard. Moreover, it can be concluded that the proposed method tends to give a better performance on larger IQA benchmark databases with authentic distortions.

To ascertain whether the achieved results are significant or not, one-sided t-tests [83] were carried out between the 100 SROCC values produced by our and the other state-of-the-art methods tested by us. Namely, the null hypothesis was that the mean SROCCs of two sets containing 100–100 correlation values are equal with a confidence of 95%. The results of the one-sided t-tests are summarized in

Table 7. Results of the one-sided t-tests. Symbol 1 (−1) means that the proposed DF-CNN-IQA method is statistically significantly $(95\%$ confidence interval, $p=0.05$ significance level) better (worse) than the NR-IQA method in the row on the IQA benchmark database in the column.

Method	CLIVE [50]	KonIQ-10k [4]	SPAQ [5]	TID2013 [51]
BLIINDER [66]	1	1	1	−1
DeepRN [67]	1	1	1	1
BLIINDS-II [14]	1	1	1	1
BMPRI [68]	1	1	1	1
BRISQUE [15]	1	1	1	1
CurveletQA [11]	1	1	1	1
DIIVINE [9]	1	1	1	1
ENIQA [69]	1	1	1	1
GRAD-LOG-CP [70]	1	1	1	1
MultiGAP-NRIQA [36]	1	1	1	1
MSDF-IQA [40]	1	1	1	1
NBIQA [71]	1	1	1	1
PIQE [72]	1	1	1	1
OG-IQA [73]	1	1	1	1
SSEQ [74]	1	1	1	1
UNIQUE [75]	−1	1	1	−1

. In this table, symbol 1 $(-1)$ stands for that the proposed method is statistically better (worse) at a confidence interval of $95\%$ than the other examined NR-IQA method. From these results, it can be concluded that the proposed method is able to significantly outperform the state of the art on large databases with authentic distortions (KonIQ-10k [4] and SPAQ [5]), while it is outperformed by two methods on artificial distortions (TID2013 [51]).

To further demonstrate the effectiveness of the proposed method, a cross-database test was also carried out, where the examined algorithms were trained on the large KonIQ-10k [4] and tested on the smaller CLIVE [50]. The results of this cross-database test are summarized in

Table 8. Cross-database test. Methods were trained on KonIQ-10k [4] and tested on CLIVE [50]. The best results are typed in bold, and the second-best ones are underlined. We denoted by ‘-’ if the data were not available.

Method	PLCC	SROCC	KROCC
BLIINDER [66]	0.748	0.730	0.503
DeepRN [67]	0.746	0.725	0.481
BLIINDS-II [14]	0.107	0.090	0.063
BMPRI [68]	0.453	0.389	0.298
BRISQUE [15]	0.509	0.460	0.310
CurveletQA [11]	0.496	0.505	0.347
DIIVINE [9]	0.479	0.434	0.299
ENIQA [69]	0.428	0.386	0.272
GRAD-LOG-CP [70]	0.427	0.384	0.261
MultiGAP-NRIQA [36]	0.841	0.813	0.585
MSDF-IQA [40]	0.764	0.749	0.552
NBIQA [71]	0.503	0.509	0.284
OG-IQA [73]	0.442	0.427	0.289
SSEQ [74]	0.270	0.256	0.170
UNIQUE [75]	0.842	0.826	0.589
CONTRIQUE [76]	-	0.731	-
DB-CNN [77]	-	0.755	-
DeepFL-IQA [78]	-	0.704	-
KonCept512 [79]	0.848	0.825	-
PaQ-2-PiQ [46]	-	-	-
PQR [81]	-	0.770	-
WSP [43]	0.840	0.820	-
DF-CNN-IQA	0.854	0.831	0.598

. From the presented results, it can be seen that the proposed method is able to outperform all the other considered algorithms by a large margin. This means that the proposed method possesses a rather strong generalization capability.

4. Conclusions

In this paper, we proposed a novel deep learning-based approach for NR-IQA. Specifically, a decision fusion of different deep convolutional architectures, such as VGG16 [56], ResNet50 [57], InceptionV3 [42], InceptionResNetV2 [59], DenseNet201 [60] and NASNetMobile [61], were carried out to for perceptual image quality estimation. The main motivation behind this layout was the following. The first layers of a CNN frequently capture basic image structures, i.e., edges, blobs or color patterns, which are quality-aware features. Using a diverse set of pretrained CNNs allowed us to better characterize natural images’ authentic distortions. This motivation was confirmed in a parameter study where it was demonstrated that a decision fusion of CNNs is able to considerably improve the performance of estimation. In a series of comparative experiments, it has been shown that the proposed method is able to reach or outperform the state of the art on two IQA benchmark databases with authentic distortions. The obtained results were confirmed in significance and cross-database tests. Future work will focus on a feature-level fusion and joint training of CNNs to reduce the training time and to achieve simpler architectures. Another direction of research is to generalize the achieved results for specific applications, such as the quality assessment of computer tomography (CT) or magnetic resonance imaging (MRI) images. The source code of the proposed is available at: https://github.com/Skythianos/DF-CNN-IQA, accessed on 28 November 2021.