Full-Reference Image Quality Assessment Based on Grünwald–Letnikov Derivative, Image Gradients, and Visual Saliency

Abstract

The purpose of image quality assessment is to estimate digital images’ perceptual quality coherent with human judgement. Over the years, many structural features have been utilized or proposed to quantify the degradation of an image in the presence of various noise types. Image gradient is an obvious and very popular tool in the literature to quantify these changes in the images. However, gradient is able to characterize images locally. On the other hand, results from previous studies indicate that global contents of a scene are analyzed before the local features by the human visual system. Relying on these features of the human visual system, we propose a full-reference image quality assessment metric that characterizes the global changes of an image by the Grünwald–Letnikov derivatives and the local changes by image gradients. Moreover, visual saliency is also utilized for weighting the changes in the images and emphasizing those areas of the image which are salient to the human visual system. To prove the efficiency of the proposed method, massive experiments were carried out on publicly available benchmark image quality assessment databases.

1. Introduction

Image quality assessment (IQA) is still a serious research challenge due to the difficulties of modelling the enormous complexity of the human visual system and perception. Presently, IQA algorithms are divided into two distinct classes, i.e. subjective and objective IQA. Specifically, subjective IQA focuses on collecting subjective quality scores from human participants in a laboratory environment [1] or an online crowdsourcing experiment [2]. Subsequently, users’ individual quality ratings are averaged into mean opinion scores (MOS) that are later considered as a direct measure of image quality. Besides, subjective IQA studies in detail the effects of viewing distances, display devices, lighting conditions, and participants demographical and physical features. Many benchmark IQA databases [3,4,5] can be found online which are the results of subjective quality experiments. Specifically, these databases consist of a number of digital images with their corresponding MOS values.

In contrast to subjective IQA, the aim of objective IQA is devising mathematical algorithms and methods which are capable of predicting perceptual image quality. In the literature, objective IQA is classified into three broad groups. The first group is full-reference image quality assessment (FR-IQA) where the algorithms estimate the quality of distorted images with full access to the distortion-free, reference images. In contrast, no information is available about reference images in no-reference image quality assessment (NR-IQA). Finally, reduced-reference image quality assessment (RR-IQA) corresponds to a transition between NR-IQA and FR-IQA. Although full information about the reference images is not available, but some features derived from the reference images can be applied in RR-IQA.

Over the years, many structural features have been utilized or proposed to quantify image degradations. Image gradient, which characterizes image locally, is a very popular tool in the literature for this purpose [6,7,8,9]. Results of previous studies indicate that global contents of a scene are analyzed by the human visual system before the local features [10]. The main contribution of this study is an FR-IQA metric that characterizes the global changes of an image by the Grünwald–Letnikov derivative and the local changes by image gradients. Thus, a combined approach is proposed in this regard. Moreover, visual saliency is also utilized for weighting the changes in the images and emphasizing those image regions which are salient to the human visual system.

1.1. Literature Review

In the literature, numerous FR-IQA algorithms and metrics have been proposed in recent decades [11]. Further, these methods can be divided into five classes, such as (i) error visibility, (ii) structural similarity, (iii) information-theoretic, (iv) learning-based, and (v) fusion-based methods. The main idea of error visibility methods is to devise a distance measure between pixel values or between the transformed representations of the reference and the distorted images to quantify perceptual quality. The most well-known example is the simple mean square error which correlates weakly with the perceptual quality but is still widely used owing to its simplicity [12]. Another well-known example is the peak signal-to-noise ratio (PSNR) which is commonly applied to quantify the quality of image reconstruction and lossy compression [13]. Ponomarenko et al. developed further PSNR by taking the discrete cosine transform (DCT) coefficients and the contrast sensitivity function [14].

Structural similarity methods try to measure similarity between the corresponding image regions of the reference and the distorted image. The representative example of this approach and probably the most well-known FR-IQA metrics is the structural similarity index measure (SSIM) [15] which compares the reference and the distorted images in respect of luminance, contrast, and structure. Over the years many extensions and modifications of SSIM have been proposed in the literature. For example, Zhou et al. [16] calculated SSIM over multiple scales of an input image. In contrast, Li and Bovik [17] determined SSIM for three distinct image regions, such as textures, edges, and smooth regions, and took their weighted average as perceptual quality metric. Later, this approach was further developed by dividing edges into preserved and changed categories [18]. To achieve higher accuracy, Liu et al. [19] performed SSIM in the wavelet domain. The authors’ approach was further developed in the complex wavelet domain by Sampat et al. [20] Wang and Li [21] measured the information content of the input images and used it to weight SSIM. Sun et al. [22] proposed to use superpixels [23] to segment the reference and distorted images first, since they provide a more meaningful representation of images than rectangular pixel grids. This method was further improved by Frackiewicz et al. [24] by using other color spaces and comparing similarity maps by mean deviation similarity index.

Information-theoretic FR-IQA approaches measure some kind of mutual information between the reference and the distorted image to quantify perceptual image quality. A representative example is the visual information fidelity (VIF) model [25]. Specifically, the authors applied Gaussian scale mixtures in the wavelet domain to model the reference and the distorted images. Mutual information was measured between the two Gaussain scale mixtures to quantify perceptual quality.

Recently, deep learning has gained popularity in the field of visual quality assessment as well [26,27,28]. Learning-based methods apply some kind of machine or deep learning algorithm to learn relationships between image features and perceptual quality. For example, Tang et al. [29] extracted spatial and frequency domain features from reference-distorted image pairs and combined them. The obtained features were projected onto perceptual quality scores with the help of a trained random forest regressor. In contrast, Bosse et al. [30] used a convolutional neural network (CNN) as feature extractor. More specifically, deep features were extracted from a distorted and a reference image patch by a CNN and were fused together. Subsequently, the fused feature vectors were projected onto patch-wise quality scores. To get the perceptual quality of an input image, the arithmetic mean of the patch-wise scores was determined. In contrast, Ahn et al. [31] predicted a distortion sensitivity map with a three-stream CNN using as input the distorted image, the reference image, and the spatial error map. To get the perceptual quality, the sensitivity map is multiplied by the spatial error map.

Fusion-based methods take existent FR-IQA metrics to compile a new image quality evaluator. The main idea behind fusion-based methods is similar to those of boosting in machine learning. For example, Okarma et al. [32] studied the properties of MS-SSIM, VIF, and R-SVD FR-IQA metrics thoroughly and proposed the fusion of these three metrics by particular arithmetic expression containing productions and powers. Later, Okarma proposed different regression techniques for a more effective FR-IQA metrics fusion [33,34]. Based on the results of Okarma, Oszust [35] and Yuan et al. [36] introduced other regression based fusion techniques. Specifically, in [35] traditional FR-IQA metrics were used as predictor variables in a multiple linear regression model, while Yuan et al. [36] utilized kernel ridge regression for combining predefined local structures and local distortion measurements. In [37], a support vector regression based fusion was carried out based on ten FR-IQA metrics. In contrast, Lukin et al. [38] trained a neural network to fuse the results of six traditional FR-IQA metrics. Instead of machine learning techniques, Oszust [39] implemented a genetic algorithm for the decision fusion of multiple metrics. This approach was further developed in [40] by applying multi-gene genetic programming. Amirshahi et al. [41] compared the feature maps, extracted from an AlexNet [42] convolutional neural network model, of the reference and the distorted image using traditional FR-IQA metrics. To obtain the perceptual quality of the distorted image, the quality scores of the feature maps were aggregated using different types of averages, such as arithmetic and geometric mean.

For comprehensive surveys about FR-IQA, we refer readers to [43,44,45,46].

1.2. Organization of the Paper

The remaining parts of this study is organized as follows. After this introduction and literature review, Section 2 introduces briefly the mathematical preliminaries, i.e. Grünwald–Letnikov derivative, and our proposed method in detail. Next, Section 3 gives the definitions of the applied evaluation metrics and presents a comprehensive comparison to the state-of-the-art. Lastly, the paper is concluded in Section 4.

2. Proposed Method

2.1. Preliminaries

In this section, some mathematical concepts and definitions are introduced which have vital importance in our proposed FR-IQA metrics. The Grünwald–Letnikov derivative, introduced by Anton Karl Grünwald Austrian and Aleksey Vasilievich Letnikov Russian mathematicians, is a basic extension of the definition of the derivative in fractal calculus. Specifically, it enables to take the derivative of a function a non-integer number of times [47]. The definition of Grünwald–Letnikov derivative is conducted from the integer-order calculus in the literature. The starting point is the definition of the first-order derivative of a one-dimensional signal $f\left(x\right)$ which is determined as:

$$\frac{\mathrm{d}f\left(x\right)}{\mathrm{d}x}=\underset{h\to 0}{lim}\frac{f\left(x\right)-f(x-h)}{h}.$$

(1)

Based on this, the second-order derivative can be expressed as

$$\frac{{\mathrm{d}}^{2}f\left(x\right)}{\mathrm{d}{x}^2}=\underset{h\to 0}{lim}\frac{f\left(x\right)-2f(x-h)+f(x-2h)}{h^2}.$$

(2)

In general, for any positive integer n, we can derive the following formula

$$\frac{{\mathrm{d}}^{n}f\left(x\right)}{\mathrm{d}{x}^n}=\underset{h\to 0}{lim}\frac{{\sum }_{j=0}^n{(-1)}^j\left(\genfrac{}{}{0pt}{}{n}{j}\right)f(x-jh)}{h^2},$$

(3)

where

$$\left(\genfrac{}{}{0pt}{}{n}{j}\right)=\frac{n(n-1)...(n-j+1)}{j!}.$$

(4)

Eliminating the restriction that n must be a positive by an $\alpha$ non-integer number, it is reasonable to define

$${{}_{x_0,x}{}^{GL}D}^{\alpha }f\left(x\right)=\underset{h\to 0}{lim}\frac{1}{h^{\alpha }}\sum _{j=0}^{[\frac{x-x_0}{h}]}{(-1)}^j\left(\genfrac{}{}{0pt}{}{\alpha }{j}\right)f(x-jh),$$

(5)

where ${{}_{x_0,x}{}^{GL}D}^{\alpha }f\left(x\right)$ is the $\alpha$th order Grünwald–Letnikov derivative of $f\left(x\right)$, x and $x_0$ represent the upper and lower bounds, respectively. Moreover, $[·]$ stands for the rounding operator. By replacing ${(-1)}^j\left(\genfrac{}{}{0pt}{}{n}{j}\right)$ with $\frac{{(-1)}^j\Gamma (\alpha +1)}{\Gamma (j+1)\Gamma (\alpha -j+1)}$, where $\Gamma (·)$ is the Gamma-function, we can define Grünwald–Letnikov derivative as:

$${{}_{x_0,x}{}^{GL}D}^{\alpha }f\left(x\right)=\underset{h\to 0}{lim}\frac{1}{h^{\alpha }}\sum _{j=0}^{[\frac{x-x_0}{h}]}\frac{{(-1)}^j\Gamma (\alpha +1)}{\Gamma (j+1)\Gamma (\alpha -j+1)}f(x-jh).$$

(6)

It is essential to highlight one important difference between ordinary and Grünwald–Letnikov derivatives. As one can see from Equation (6), the calculation of the Grünwald–Letnikov derivative of $f\left(x\right)$ at x requires all function values from $x_0$ to x. As a consequence, Grünwald–Letnikov derivative is considered to have memory. In the literature, this property is also formulated as that Grünwald–Letnikov derivative requires non-local information [48]. As an illustration, Figure 1 depicts the fractional derivatives of the sine function with order between $0.1$ and $0.9$.

Next, we have to define the Grünwald–Letnikov derivative of a two dimensional signal which is given as

$$I(x,y)=\left(\begin{array}{cccc}{I}_{11}& I_{12}& \cdots & I_{1N}\\ I_{21}& I_{22}& \cdots & I_{2N}\\ ⋮& ⋮& \ddots & ⋮\\ I_{M1}& I_{M2}& \cdots & I_{MN}\end{array}\right),$$

(7)

where M and N stand for the number of rows and columns in $I(x,y)$. Similarly to the ordinary derivative, the Grünwald–Letnikov derivative has to be defined in two dimensions, i.e., $x$- and $y$-directions [49,50]. In $x$-direction, it can be defined as follows

$${{}^{GL}D}^{\alpha }{I}_x(x,y)=I(x,y)-\alpha I(x-1,y)+\frac{\alpha (\alpha -1)}{2}I(x-2,y).$$

(8)

Similarly in $y$-direction

$${{}^{GL}D}^{\alpha }{I}_y(x,y)=I(x,y)-\alpha I(x,y-1)+\frac{\alpha (\alpha -1)}{2}I(x,y-2).$$

(9)

Hence, the Grünwald–Letnikov fractional derivative can be given as

$${{}^{GL}D}^{\alpha }I(x,y)=\sqrt{{\left({{}^{GL}D}^{\alpha }{I}_x(x,y)\right)}^2+{\left({{}^{GL}D}^{\alpha }{I}_y(x,y)\right)}^2}.$$

(10)

Figure 2 shows a grayscale test image and its Grünwald–Letnikov derivatives with different values of $\alpha$.

2.2. Proposed Metric

Results of previous studies indicate that global contents of a scene are analyzed by the human visual system before the local features [10]. In this study, we propose an FR-IQA metric that combines global and local information of an image by applying Grünwald–Letnikov derivatives and ordinary image derivatives (high-level overview is depicted in Figure 3). In the followings, $R(x,y)$ stands for the pristine, reference image, while $D(x,y)$ denote the distorted image generated from $R(x,y)$.

Global similarity (denoted by $S_G(x,y)$) between $R(x,y)$ and $D(x,y)$ is expressed as the similarity between the Grünwald–Letnikov derivatives of $R(x,y)$ and $D(x,y)$

$$S_G(x,y)=\frac{2·{{}^{GL}D}^{\alpha }R(x,y)·{{}^{GL}D}^{\alpha }D(x,y)+c_1}{{\left({{}^{GL}D}^{\alpha }R(x,y)\right)}^2+{\left({{}^{GL}D}^{\alpha }D(x,y)\right)}^2+c_1}$$

(11)

where $c_1$ is a constant number to manage numerical stability [15]. In our MATLAB implementation $\alpha =0.6$ fractional derivative order was used. To characterize the similarity between local changes, gradient operators are applied. The literature [51,52] recommends the Scharr operator, since it has a good effect on image quality estimation. Specifically, a $3\times 3$ Scharr operator was applied in our method whose horizontal $\left(S_x\right)$ and vertical $\left(S_y\right)$ templates can be given as

$$S_x=\frac{1}{16}\left(\begin{array}{ccc}3& 0& -3\\ 10& 0& -10\\ 3& 0& -3\end{array}\right),$$

(12)

$$S_y=\frac{1}{16}\left(\begin{array}{ccc}3& 10& 3\\ 0& 0& 0\\ -3& -10& -3\end{array}\right).$$

(13)

These templates can be applied separately to obtain gradient components of an image I in each orientation:

$$G_x=S_x·I,$$

(14)

$$G_y=S_y·I,$$

(15)

where * stands for the convolution operator. These can be put together to get the gradient magnitude:

$$G=\sqrt{G_x^2+G_y^2}.$$

(16)

Figure 4 depicts an illustration of the Scharr operator.

To characterize the local similarity $\left(S_L(x,y)\right)$ between the reference and distorted images, the gradient magnitudes are utilized as follows:

$$S_L(x,y)=\frac{2·G_R(x,y)·G_D(x,y)+c_2}{G_R^2(x,y)+G_D^2(x,y)+c_2},$$

(17)

where $G_R(x,y)$ and $G_D(x,y)$ stand for reference and distorted gradient magnitude maps, respectively. Moreover, $c_2$ is a constant number to manage numerical stability.

The similarity map (denoted by $S(x,y)$) between a reference and a distorted image using the preceding equations is defined as

$$S(x,y)={\left(S_G(x,y)\right)}^{\lambda }·{\left(S_L(x,y)\right)}^{1-\lambda },$$

(18)

where $\lambda$ is used to fine-tune the respective weights of the importance of global and local information. In our MATLAB implementation, $\lambda =0.7$ was applied. To get the local global variation $\left(LGV\right)$ quality score, we take the average of $S(x,y)$. Formally, it can be written

$$LGV=\frac{1}{M·N}\sum _{x=1}^M\sum _{y=1}^{N}S(x,y).$$

(19)

In the saliency weighted local global variation (SWLGV) quality score, the visual attention mechanism is also taken into account. Namely, the differences between the reference and the distorted images are emphasized in the salient regions. Let denote the saliency maps of the reference and distorted images by $S{M}_R(x,y)$ and $S{M}_D(x,y)$, respectively. In our metric, the algorithm of Imamoglu et al. [53] was used to generate saliency maps. The saliency map of a reference-distorted image pair (denoted by $SM(x,y)$) is the elementwise maximum of $S{M}_R(x,y)$ and $S{M}_D(x,y)$,

$$SM(x,y)=max(S{M}_R(x,y),S{M}_D(x,y)).$$

(20)

Specifically, SWLGV corresponds to the weighted average of $S(x,y)$ and $SM(x,y)$, where $SM(x,y)$ represents the weights. Formally, it can be written:

$$SWLGV=\frac{{\sum }_{i=1}^M{\sum }_{j=1}^{N}SM(x,y)·S(x,y)}{{\sum }_{i=1}^M{\sum }_{j=1}^{N}SM(x,y)}.$$

(21)

3. Experimental Results and Analysis

This section presents our obtained experimental results. First, Section 3.1 describes the applied evaluation metrics and protocol. Subsequently, the used benchmark IQA databases are introduced in Section 3.2. Finally, a comparison of $LGV$ and $SWLGV$ to the state-of-the-art is presented in Section 3.3.

3.1. Evaluation Metrics and Protocol

The performance of an FR-IQA metric is given by correlation coefficients in the literature which are measured between predicted and ground-truth quality scores [54]. With this end in view, three correlation coefficients, such as Pearson’s linear correlation coefficient (PLCC), Spearman’s rank order correlation coefficient (SROCC), and Kendall’s rank order correlation coefficient (KROCC), are widely used in the literature. Thus, these evaluation metrics are also applied in this paper. The PLCC between two vectors (denoted by $\mathbf{x}$ and $\mathbf{y}$) is defined as:

$$PLCC(\mathbf{x},\mathbf{y})=\frac{{\sum }_{i=1}^N(x_i-\overline{x})(y_i-\overline{y}))}{\sqrt{{\sum }_{i=1}^N{(x_i-\overline{x})}^2{\sum }_{i=1}^N{(y_i-\overline{y})}^2}},$$

(22)

where $\overline{x}=\frac{1}{N}{\sum }_{i=1}^N{x}_i$ and $\overline{y}=\frac{1}{N}{\sum }_{i=1}^N{y}_i$. Following the recommendations of Sheikh et al. [55], a non-linear mapping is applied between the predicted and ground-truth scores before the calculation of PLCC using the following formula:

$$Q={\beta }_1\left(\frac{1}{2}-\frac{1}{e^{-{\beta }_2(Q_p-{\beta }_3)}}\right)+{\beta }_4{Q}_p+{\beta }_5,$$

(23)

where ${\beta }_i$’s $(i$ goes from 1 to 5) represent the fitting parameters. Moreover, $Q_p$ and Q denote the predicted and mapped quality scores, respectively. Similarly, SROCC is defined as:

$$SROCC(\mathbf{x},\mathbf{y})=\frac{{\sum }_{i=1}^N(x_i-x^{\prime })(y_i-y^{\prime }))}{\sqrt{{\sum }_{i=1}^N{(x_i-x^{\prime })}^2{\sum }_{i=1}^N{(y_i-y^{\prime })}^2}},$$

(24)

where $x^{\prime }$ and $y^{\prime }$ stand for the middle ranks of $\mathbf{x}$ and $\mathbf{y}$, respectively. Finally, KROCC is defined as

$$KROCC(\mathbf{x},\mathbf{y})=\frac{C-D}{\left(\genfrac{}{}{0pt}{}{N}{2}\right)},$$

(25)

where C stands for the number of concordant pairs between $\mathbf{x}$ and $\mathbf{y}$, while D denotes the number of discordant pairs.

The proposed methods were implemented in MATLAB R2020a using a STRIX Z270H Gaming personal computer with Intel(R) Core(TM) i7-7700K CPU 4.20 GHz (8 cores) and 15 GB memory.

3.2. Databases

Benchmark IQA databases used for developing, testing, and ranking FR-IQA methods contain a small group of reference images whose perceptual quality are believed flawless. Moreover, distorted images are artificially generated from the reference images applying several levels and types of distortions, such as motion blur, JPEG compression, or salt & pepper noise. Implicitly, MOS values belong to the distorted images. In this study, we utilized four popular IQA benchmark databases, i.e. KADID-10k [5], TID2013 [3], TID2008 [56], and CSIQ [57], to evaluate the proposed LGV and SWLGV metrics. The empirical MOS distributions of these databases are depicted in Figure 5, while their main properties are outlined in

Table 1. Summary about the main properties of the applied publicly available IQA benchmark databases.

Attribute	KADID-10k [5]	TID2013 [3]	TID2008 [56]	CSIQ [57]
Year	2019	2013	2008	2010
Number of reference images	85	25	25	30
Number of distorted images	10,125	3000	1700	866
Number of distortion types	25	24	17	6
Number of distortion levels	5	5	4	4–5
Subjective testing method	DCR	Custom	Custom	Custom
Resolution	$512\times 384$	$512\times 384$	$512\times 384$	$512\times 512$
MOS range	1–5	0–9	0–9	0–1

. Figure 6 depicts some sample distorted images from KADID-10k [5] as an illustration of IQA databases.

3.3. Comparison to the State-of-the-Art

As it can be seen in the previous section, the proposed method possesses several adjustable parameters to determine global and local similarity between the reference and the distorted images, such as $\alpha$ and h (Equation (6)) for the Grünwald–Letnikov derivative and $\lambda$ that is applied to weight the importance of global and local information. To determine optimal values for these parameters, eight random reference images and their corresponding 544 distorted counterparts were taken and numerical experiments were carried out on this subset of TID2008 [56]. Namely, $\alpha$ and $\lambda$ were varied from 0 to 1 in steps of 0.1, while h was varied from 10 to 100 in steps of 10. During the numerical experiments, we monitored the SROCC values. Finally, we choose $\alpha =0.6$, $\lambda =0.7$, and $h=80$ where the maximum of SROCC was measured.

To compare the previously presented LGV and SWLGV FR-IQA metrics to the state-of-the-art, nine other state-of-the-art FR-IQA metrics were collected, i.e., 2stepQA [58], CSV [59], DISTS [60], GSM [8], MAD [57], MS-SSIM [16], ReSIFT [61], RVSIM [62], and SSIM [15], whose source codes were made online available for the research community. The results measured on KADID-10k [5] and TID2013 [3] are outlined in

Table 2. Comparison of LGV and SWLGV to the state-of-the-art on KADID-10k [5] and TID2013 [3]. The highest values are typed in bold, while the second highest ones are underlined.

	KADID-10k [5]			TID2013 [3]
FR-IQA Metric	PLCC	SROCC	KROCC	PLCC	SROCC	KROCC
2stepQA [58]	0.768	0.771	0.571	0.736	0.733	0.550
CSV [59]	0.671	0.669	0.531	0.852	0.848	0.657
DISTS [60]	0.809	0.814	0.626	0.759	0.711	0.524
GSM [8]	0.780	0.780	0.588	0.789	0.787	0.593
MAD [57]	0.716	0.724	0.535	0.827	0.778	0.600
MS-SSIM [16]	0.819	0.821	0.630	0.794	0.785	0.604
ReSIFT [61]	0.648	0.628	0.468	0.630	0.623	0.471
RVSIM [62]	0.728	0.719	0.540	0.763	0.683	0.520
SSIM [15]	0.670	0.671	0.489	0.618	0.616	0.437
LGV	0.640	0.820	0.630	0.832	0.801	0.631
SWLGV	0.685	0.840	0.655	0.855	0.804	0.637

. The results of TID2008 [56] and CSIQ [57] are presented in

Table 3. Comparison of LGV and SWLGV to the state-of-the-art on TID2008 [56] and CSIQ [57]. The highest values are typed in bold, while the second highest ones are underlined.

	TID2008 [56]			CSIQ [57]
FR-IQA Metric	PLCC	SROCC	KROCC	PLCC	SROCC	KROCC
2stepQA [58]	0.757	0.769	0.574	0.841	0.849	0.655
CSV [59]	0.852	0.851	0.659	0.933	0.933	0.766
DISTS [60]	0.705	0.668	0.488	0.930	0.930	0.764
GSM [8]	0.782	0.781	0.578	0.906	0.910	0.729
MAD [57]	0.831	0.829	0.639	0.950	0.947	0.796
MS-SSIM [16]	0.838	0.846	0.648	0.913	0.917	0.743
ReSIFT [61]	0.627	0.632	0.484	0.884	0.868	0.695
RVSIM [62]	0.789	0.743	0.566	0.923	0.903	0.728
SSIM [15]	0.669	0.675	0.485	0.812	0.812	0.606
LGV	0.778	0.874	0.687	0.779	0.926	0.760
SWLGV	0.811	0.884	0.705	0.776	0.922	0.755

. From the presented results, it can be concluded that the SWLGV metric provides the best outcomes in terms of SROCC and KROCC on KADID-10k [5] and TID2008 [56]. Furthermore, it gives the best PLCC value and the second best SROCC and KROCC values on TID2013 [3]. Interestingly, the saliency weighting step do not improve the performance of estimation on CSIQ [57], while it significantly improves the estimation accuracy on the other applied databases.

Table 4. Comparison of LGV and SWLGV to the state-of-the-art. Direct and weighted average PLCC, SROCC, and KROCC values are reported. Measured on KADID-10k [5], TID2013 [3], TID2008 [56], and CSIQ [57]. The highest values are typed in bold, while the second highest ones are underlined.

	Direct Average			Weighted Average
FR-IQA Metric	PLCC	SROCC	KROCC	PLCC	SROCC	KROCC
2stepQA [58]	0.776	0.781	0.587	0.765	0.768	0.572
CSV [59]	0.827	0.825	0.653	0.740	0.738	0.582
DISTS [60]	0.801	0.781	0.601	0.795	0.785	0.599
GSM [8]	0.814	0.815	0.622	0.789	0.789	0.596
MAD [57]	0.831	0.820	0.643	0.763	0.758	0.573
MS-SSIM [16]	0.841	0.842	0.656	0.821	0.822	0.633
ReSIFT [61]	0.697	0.688	0.530	0.655	0.641	0.483
RVSIM [62]	0.801	0.762	0.589	0.752	0.725	0.549
SSIM [15]	0.692	0.694	0.504	0.668	0.669	0.485
LGV	0.757	0.855	0.677	0.699	0.828	0.644
SWLGV	0.782	0.863	0.688	0.736	0.842	0.662

summarizes the direct and weighted averages of the PLCC, SROCC, and KROCC values measured on KADID-10k [5], TID2013 [3], TID2008 [56], and CSIQ [57]. It can be observed that the proposed SWLGV provides the best results in terms of SROCC and KROCC results, while the proposed LGV gives the second best results in terms of SROCC and KROCC.

Table 5. Comparison of SROCC of each FR-IQA metrics on TID2013’s [3] distortion levels (Level 1 represents the lowest level of degradation, while Level 5 represents the highest one). The highest values are typed in bold, while the second highest ones are underlined.

	2stepQA [58]	CSV [59]	DISTS [60]	GSM [8]	MAD [57]	MS-SSIM [16]	ReSIFT [61]	RVSIM [62]	SSIM [15]	LGV	SWLGV
Level 1	0.246	0.424	0.235	0.372	0.388	0.166	0.181	0.248	0.204	0.401	0.398
Level 2	0.394	0.626	0.440	0.512	0.368	0.049	0.401	0.430	0.276	0.605	0.610
Level 3	0.539	0.635	0.367	0.523	0.442	0.240	0.415	0.416	0.084	0.630	0.632
Level 4	0.571	0.749	0.606	0.669	0.284	0.172	0.699	0.702	0.208	0.728	0.735
Level 5	0.663	0.787	0.664	0.745	0.308	0.397	0.788	0.803	0.202	0.746	0.747
All	0.733	0.848	0.711	0.787	0.778	0.785	0.623	0.683	0.616	0.801	0.804

and

Table 6. Comparison of SROCC of each FR-IQA metrics on TID2008’s [56] distortion levels (Level 1 represents the lowest level of degradation, while Level 5 represents the highest one). The highest values are typed in bold, while the second highest ones are underlined.

	2stepQA [58]	CSV [59]	DISTS [60]	GSM [8]	MAD [57]	MS-SSIM [16]	ReSIFT [61]	RVSIM [62]	SSIM [15]	LGV	SWLGV
Level 1	0.470	0.638	0.566	0.639	0.432	0.067	0.457	0.634	0.368	0.644	0.649
Level 2	0.619	0.683	0.381	0.636	0.520	0.221	0.437	0.513	0.105	0.701	0.703
Level 3	0.573	0.774	0.581	0.677	0.239	0.059	0.707	0.761	0.190	0.779	0.782
Level 4	0.610	0.829	0.628	0.718	0.232	0.275	0.788	0.825	0.241	0.841	0.845
All	0.769	0.851	0.668	0.781	0.829	0.846	0.632	0.743	0.675	0.874	0.884

summarizes the SROCC values which were measured separately on the distortion levels of TID2013 [3] and TID2008 [56]. As mentioned in Section 1, TID2013 [3] and TID2008 [56] have five and four different distortion levels, respectively. It can be observed that LGV and SWLGV give in general higher performance on higher distortion levels. Moreover, SWLGV provides the second best SROCC values on 4 out of 5 distortion levels of TID2013 [3], while it is the best performing method on all distortion levels of TID2008 [56]. On the other hand, LGV provides the second best result on the lowest distortion level of TID2013 [3] and the second highest SROCC values on all distortion levels of TID2008 [56].

Table 7. Comparison on TID2013’s [3] distortion types. SROCC values are given. The highest values are typed in bold, while the second highest ones are underlined.

	2stepQA [58]	CSV [59]	DISTS [60]	GSM [8]	MAD [57]	MS-SSIM [16]	ReSIFT [61]	RVSIM [62]	SSIM [15]	LGV	SWLGV
AGN	0.817	0.938	0.845	0.899	0.912	0.624	0.831	0.886	0.848	0.921	0.936
ANC	0.590	0.862	0.786	0.823	0.800	0.387	0.749	0.836	0.779	0.911	0.904
SCN	0.860	0.939	0.859	0.927	0.929	0.683	0.839	0.868	0.851	0.887	0.930
MN	0.395	0.748	0.814	0.704	0.658	0.372	0.702	0.734	0.775	0.821	0.842
HFN	0.828	0.927	0.868	0.884	0.902	0.704	0.869	0.895	0.889	0.916	0.874
IN	0.715	0.848	0.674	0.813	0.743	0.766	0.824	0.865	0.810	0.759	0.795
QN	0.886	0.892	0.810	0.911	0.895	0.720	0.745	0.869	0.817	0.866	0.956
GB	0.853	0.933	0.926	0.954	0.915	0.762	0.937	0.970	0.910	0.952	0.961
DEN	0.900	0.952	0.899	0.955	0.922	0.819	0.907	0.926	0.876	0.984	0.976
JPEG	0.867	0.944	0.897	0.933	0.924	0.784	0.905	0.930	0.893	0.970	0.952
JP2K	0.891	0.966	0.931	0.934	0.929	0.790	0.928	0.946	0.806	0.945	0.968
JGTE	0.806	0.800	0.906	0.866	0.768	0.582	0.712	0.831	0.701	0.895	0.900
J2TE	0.854	0.887	0.865	0.893	0.854	0.742	0.835	0.882	0.813	0.918	0.874
NEPN	0.775	0.811	0.833	0.804	0.803	0.792	0.693	0.771	0.634	0.719	0.795
BLOCK	0.044	0.183	0.302	0.588	−0.322	0.382	0.440	0.545	0.564	0.603	0.601
MS	0.660	0.654	0.752	0.728	0.708	0.732	0.418	0.559	0.738	0.677	0.756
CC	0.430	0.227	0.464	0.466	0.420	0.027	−0.055	0.132	0.355	0.659	0.667
CCS	−0.258	0.809	0.789	0.676	−0.059	−0.055	−0.209	0.366	0.742	0.750	0.758
MGN	0.747	0.884	0.790	0.831	0.888	0.653	0.765	0.853	0.804	0.819	0.841
CN	0.858	0.924	0.907	0.902	0.904	0.596	0.882	0.914	0.797	0.838	0.859
LCNI	0.902	0.965	0.932	0.945	0.950	0.713	0.897	0.933	0.877	0.873	0.917
ICQD	0.808	0.919	0.832	0.901	0.867	0.739	0.770	0.871	0.820	0.845	0.864
CA	0.702	0.845	0.879	0.835	0.760	0.568	0.838	0.871	0.740	0.793	0.788
SSR	0.926	0.976	0.944	0.961	0.949	0.801	0.944	0.956	0.822	0.800	0.810
All	0.733	0.848	0.711	0.787	0.778	0.785	0.623	0.683	0.616	0.801	0.804

and

Table 8. Comparison on TID2008’s [56] distortion types. SROCC values are given. The highest values are typed in bold, while the second highest ones are underlined.

	2stepQA [58]	CSV [59]	DISTS [60]	GSM [8]	MAD [57]	MS-SSIM [16]	ReSIFT [61]	RVSIM [62]	SSIM [15]	LGV	SWLGV
AGN	0.766	0.922	0.812	0.855	0.872	0.610	0.771	0.840	0.805	0.913	0.922
ANC	0.627	0.893	0.811	0.821	0.803	0.354	0.762	0.829	0.780	0.898	0.897
SCN	0.814	0.932	0.838	0.904	0.901	0.727	0.810	0.837	0.800	0.917	0.914
MN	0.450	0.781	0.830	0.736	0.673	0.304	0.728	0.760	0.797	0.809	0.843
HFN	0.818	0.936	0.870	0.889	0.894	0.749	0.881	0.886	0.871	0.918	0.908
IN	0.659	0.819	0.626	0.764	0.650	0.767	0.777	0.836	0.776	0.786	0.786
QN	0.850	0.894	0.770	0.903	0.851	0.708	0.730	0.836	0.784	0.887	0.941
GB	0.877	0.923	0.909	0.948	0.896	0.759	0.904	0.963	0.866	0.959	0.960
DEN	0.919	0.970	0.931	0.971	0.928	0.786	0.923	0.939	0.873	0.967	0.971
JPEG	0.895	0.948	0.894	0.937	0.931	0.774	0.914	0.926	0.880	0.953	0.948
JP2K	0.910	0.984	0.953	0.949	0.941	0.837	0.935	0.970	0.745	0.975	0.984
JGTE	0.851	0.790	0.907	0.871	0.781	0.606	0.735	0.860	0.666	0.879	0.901
J2TE	0.845	0.852	0.833	0.880	0.802	0.742	0.778	0.854	0.769	0.902	0.901
NEPN	0.803	0.752	0.882	0.784	0.801	0.749	0.761	0.732	0.588	0.727	0.793
Block	0.441	0.770	0.618	0.843	−0.362	0.765	0.743	0.782	0.804	0.896	0.897
MS	0.655	0.594	0.681	0.638	0.563	0.711	0.322	0.525	0.629	0.699	0.774
CC	0.597	0.330	0.649	0.634	0.548	0.042	−0.018	0.194	0.502	0.669	0.693
All	0.769	0.851	0.668	0.781	0.829	0.846	0.632	0.743	0.675	0.874	0.884

presents the results on TID2013 [3] and TID2008 [56] in detail for every distortion types found in these IQA benchmark databases. As mentioned in Section 3.2, TID2013 [3] contains 24 distinct distortion types, i.e., AGN (additive Gaussian noise), ANC (additive noise in color components), SCN (spatially correlated noise), MN (masked noise), HFN (high frequency noise), IN (impulse noise), QN (quantization noise), GB (Gaussian blur), DEN (image denoising), JPEG (JPEG compression), JP2K (JPEG2000 compression), JGTE (JPEG transmission errors), J2TE (JPEG2000 transmission errors), NEPN (Non-eccentricity pattern noise), BLOCK (local block-wise distortions of different intensity), MS (Mean shift), CC (contrast change), CCS (change of color saturation), MGN (multiplicative Gaussian noise), CN (comfort noise), LCNI (lossy compression of noisy images), ICQD (image color quantization with dither), CA (chromatic aberrations), and SSR (sparse sampling and reconstruction). On the other hand, TID2008 [56] contains a narrower set of distortions than TID2013 [3]. Specifically, it includes the first 17 distortion types of TID2013 [3]. It can be seen that SWLGV and LGV are the best performing method on 5 out of 24 distortion types of TID2013 [3]. On the other hand, SWLGV provides the highest SROCC values on 8 out of 17 distortion types of TID2008 [56] and gives on 6 distortions the second best results.

4. Conclusions

In the present study, an innovative FR-IQA metric was proposed relying on Grünwald–Letnikov derivative, image gradients, and visual saliency. The starting point was the observation of previous studies that the human visual system analyzes the global features of a scene before the local ones. However, image gradients, which are very popular in the literature to quantify image degradations, characterize the image locally. Our main contribution was a metric that describes the global changes of an image relying on Grünwald–Letnikov derivative, while the local changes are quantified by image gradients. Next, the combination of local and global changes were weighted by visual saliency to estimate perceptual image quality. The proposed metric was compared with several other state-of-the-art algorithms on major standard IQA databases. It was demonstrated that the proposed method is able to surpass or approach the state-of-the-art performance.