Fuzzy Windows with Gaussian Processed Labels for Ordinal Image Scoring Tasks

Round 1

Reviewer 1 Report

Strong points of the paper：

1. This manuscript proposed a FW-GPL approach for the ordinal classification issue. This method including a defuzzifier window and a learning strategy of Gaussian processed labels. It solved the problem of ambiguous and overlapped features in the ordinal data. The method is novel.

2. The manuscript introduces some basic knowledge in the related work, and explains some of the principles in a combination of pictures and pictures, step by step and easy to understand.

Weak points of the paper:

1. The figures mentioned in the manuscript do not correspond to the one actually intended to use. For example, the "we can find A and B in Figure 2" statement is Figure 2, but what you actually want to use is Figure 3. There are many similar places in the article, it is recommended to read carefully and modify.

2. In the manuscript, there are also problems of inconsistent symbols and unexplained symbols of the formula. It is suggested to unify the symbols and explain the symbols used in the formula as detailed as possible. Meanwhile, it is suggested to add a comma or period after the formula to become a complete sentence.

3. Please note that the order of the chart should be consistent with the order of the article, and the chart should not be too far away from the corresponding text. It is recommended to reformat it.

4. Some figures in the manuscript are incomplete or difficult to understand. For example, Figure 1 lacks the annotation of the X axis and Y axis; Figure 2 involves the mismatch between some text and illustrations of age groups, which is difficult to understand; Figure 5 is not cited or mentioned in the text. It is recommended that some pictures be redrawn. 

5. Since the experimental results of this manuscript are only competitive, it is suggested to state more clearly the specific reasons for the competitiveness and the advantages over other methods.

 

Overall the method is novel and I recommend that the authors revise the manuscript according to the above-mentioned questions. 

Author Response

Strong points of the paper：

1. This manuscript proposed a FW-GPL approach for the ordinal classification issue. This method includes a defuzzifier window and a learning strategy of Gaussian processed labels. It solved the problem of ambiguous and overlapped features in the ordinal data. The method is novel.
2. The manuscript introduces some basic knowledge in the related work, and explains some of the principles in a combination of pictures and pictures, step by step and easy to understand.

Weak points of the paper:

1. The figures mentioned in the manuscript do not correspond to the one actually intended to use. For example, the "we can find A and B in Figure 2" statement is Figure 2, but what you actually want to use is Figure 3. There are many similar places in the article, it is recommended to read carefully and modify.

Reply: Thank you for your such good comments. We have carefully revised this type of error. To avoid repeating, we also have removed Figure 3.

2. In the manuscript, there are also problems of inconsistent symbols and unexplained symbols of the formula. It is suggested to unify the symbols and explain the symbols used in the formula as detailed as possible. Meanwhile, it is suggested to add a comma or period after the formula to become a complete sentence.

Reply: Thank you very much. We have revised the inconsistent symbols and described the unexplained ones. For example, we have unified the prediction probability that the input image belongs to the label l_k as P_k . K is the total number of ordinal categories. Revised μ_o to μ in Eq (8) to be consistent with Eq (5). Moreover, we also revised Eq (10) to be more readable, as well as Eq (11) and Eq (12). In line 265, we have added more to explain Frk and Bk.

3. Please note that the order of the chart should be consistent with the order of the article, and the chart should not be too far away from the corresponding text. It is recommended to reformat it.

Reply: Thank you for your kind reminder. We have revised the locations of the figures and charts to be consistent with the order of this article.

4. Some figures in the manuscript are incomplete or difficult to understand. For example, Figure 1 lacks the annotation of the X-axis and Y-axis; Figure 2 involves the mismatch between some text and illustrations of age groups, which is difficult to understand; Figure 5 is not cited or mentioned in the text. It is recommended that some pictures be redrawn.

Reply: Thank you for your kind reminder. Figure 1 and Figure 2 have been redrawn. Moreover, please find the cited Figure 5 in section 6.3. Ablation Study III (incomplete ordinal image data) “We manually remove some age segments of the IMDB-WIKI to train the model and test it in the complete ordinal text data, as shown in Figure 5. From Table 11, we can find that when the number of neurons is 100, the most proper window is 20.”

5. Since the experimental results of this manuscript are only competitive, it is suggested to state more clearly the specific reasons for the competitiveness and the advantages over other methods.

Reply: Thank you for your comment. In 6. Ablation and Discussion, we have added one discussion to clarify the advantage and the limitation of our method.

“6.4. Advantage and Limitation

By directly facing the challenge of ordinal image classification, our method attempts to reduce the influence of the overlapped features. The length of the window controls the defuzzification of the ordinal neighbour categories. JREAE [ 13] used two covariance matrices to capture the underlying correlations from both aspects of input facial features and output age labels, but this family of methods (e.g., DRF [6] and AVDL[10 ]) should first take the age distribution of the dataset into account. Because after fitting the distribution of the facial age dataset, there is an inevitable deviation between the real distribution of the age and the fitted one. To avoid such a problem, our method used a Gaussian distribution within the window to approximate the relationship between input facial features and output age labels. From Table 5, our method outperforms other LDBL methods and presents the advantage of using label distribution-based learning methods.

However, the limitation is that we only use a naive fuzzy logic window to leverage the challenge of ordinal image classification tasks. By adaptively adjusting the distance between the real age and the centre of the moving window, MWR [ 18 ] moved the window to fit the ρ-ranks within entire and specific age ranks. Our method constrained the centre of the window by using naive fuzzy logic to adjust the distribution of the facial age in the window. That would ignore the influence of the remote but highly related feature which is beyond the window. Even though we tried to use longer windows, our method failed to overcome this problem.”

Reviewer 2 Report

The authors proposed a method that leveraged the so-called fuzzy window with Gaussian processed labels (FW-GPL) to mitigate the impact of overlapping features in ordinal image classification tasks. A big problem is the results do not seem impressive at all compared with prior works, begging questions on the merit of this study.

Table 4: given the size of the dataset, the reported improvement is marginal, and it is hard to believe they convey any statistical significance.

Table 5: The MAE performance is mediocre in comparison.

Tables 6 and 7: The improvement is marginal. The authors are responsible for convincing readers such small difference is statistically significant, rather than just claiming “weak but obvious improvement”.

 

The authors should also refine the presentation of information. Fewer jargons and more intuition would help readers better understand their method, as well as more specific analysis of their results. Currently, the abstract is too detail-oriented, the introduction provides too little high-level overview of the manuscript, figure and table captions are not self-sustaining, and there is no discussion in Section 6… All these factors add to the poor readability of the manuscript.

Author Response

The authors proposed a method that leveraged the so-called fuzzy window with Gaussian processed labels (FW-GPL) to mitigate the impact of overlapping features in ordinal image classification tasks. A big problem is the results do not seem impressive at all compared with prior works, begging questions on the merit of this study.

 

Table 4: given the size of the dataset, the reported improvement is marginal, and it is hard to believe they convey any statistical significance.

Reply: Thank you for your comment. To make a fair comparison, we eventually reported the average MAE and average RMSE by using 10-Fold Cross-Validation. In the 4.3. Experiment Settings, we have added “Finally, the 10-fold cross-validation is performed.”

 

 

 

Table 5: The MAE performance is mediocre in comparison.

Reply: Thank you for your comment. From Table 5, after comparing with the SOTA results, we found that our method can rank in the top 20%. Furthermore, we have discussed the advantage and limitations in In 6. Ablation and Discussion.

 

Tables 6 and 7: The improvement is marginal. The authors are responsible for convincing readers such small difference is statistically significant, rather than just claiming “weak but obvious improvement”.

Reply: Thank you for your comment. To discuss the advantage and limitation of our method, we have added the more into 6. Ablation and Discussion. Even though FW-GPL cannot outperform other outstanding SOTA models, our method can rank at the top 20% and top one when using the label distribution-based learning methods.

 “6.4. Advantage and Limitation

By directly facing the challenge of ordinal image classification, our method attempts to reduce the influence of the overlapped features. The length of the window controls the defuzzification of the ordinal neighbour categories. JREAE [ 13] used two covariance matrices to capture the underlying correlations from both aspects of input facial features and output age labels, but this family of methods (e.g., DRF [6] and AVDL [10]) should first take the age distribution of the dataset into account. Because after fitting the distribution of the facial age dataset, there is an inevitable deviation between the real distribution of the age and the fitted one. To avoid such a problem, our method used a Gaussian distribution within the window to approximate the relationship between input facial features and output age labels. From Table 5, our method outperforms other LDBL methods and presents the advantage of using label distribution-based learning methods.

However, the limitation is that we only use a naive fuzzy logic window to leverage the challenge of ordinal image classification tasks. By adaptively adjusting the distance between the real age and the centre of the moving window, MWR [ 18] moved the window to fit the ρ-ranks within entire and specific age ranks. Our method constrained the centre of the window by using naive fuzzy logic to adjust the distribution of the facial age in the window. That would ignore the influence of the remote but highly related feature which is beyond the window. Even though we tried to use longer windows, our method failed to overcome this problem.”

We also have revised the location of tables and figures to improve the readability.

 

Reviewer 3 Report

They propose a method - FuzzyWindow with the Gaussian Processed Labels 1 (FW-GPL) to mitigate the overlapping problem of the neighbour ordinal category when scoring images. More detailed comments are given as follows:

 1-    The title of figure 3 and 4 need to reduce.

2-    In introduction, add to the final paragraph the structure of the paper section.

3-    In 4.2. Experimental assessment criteria. Add the citation.

4-    Add the future works in end of conclusion section.

5-    Discuss the limitations of the proposed method.

6-    The recent references such as 2022 is very few. There are many works in 2022 therefore add some of them.

7-    The recent references ratio below 50% around 43%. Add more recent references.

8-    Discuss the limitations of the proposed method.

9-    In training phase, dataset partition is randomly or not? I suggest u to used K-Fold Cross Validation. And how many K-Fold used?

10-In experiments results, the evaluation (train-test round) must repeated for N round. I suggest to repeat for many rounds to ensure that the bias was minimized.

Author Response

1-    The title of figure 3 and 4 need to reduce.

Reply: Thank you for your comment. We have reduced the title of Figure 3 and Figure 4. But to avoid repeating, we have removed Figure 3.

2-    In introduction, add to the final paragraph the structure of the paper section.

Reply: Thank you for your comment. Please find the Figure 2 (which previously was Figure 5) from the Introduction.

 

3-    In 4.2. Experimental assessment criteria. Add the citation.

Reply: Thank you. In 4.2, we have added two citations.

“Willmott, C.J.; Matsuura, K. Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Climate research 2005, 30, 79–82.

Chai, T.; Draxler, R.R. Root mean square error (RMSE) or mean absolute error (MAE)?–Arguments against avoiding RMSE in theliterature. Geoscientific model development 2014, 7, 1247–1250”

 

4-    Add the future works in end of conclusion section.

Reply: Thank you for your nice comment. At end of the conclusion section, we have added our future work.

“The idea of using fuzzy logic and a Gaussian process strategy to guide ordinal image classification is inspirational and we will explore more possibilities for it. There are many directions for future work. (1) There are many other ordinal medical tasks, for example, scoring the severity of depression and grading the injury of spinal cords. We will use this method on such medical tasks in the coming research. (2) Our method cannot achieve the best SOTA result. We will try to overcome this challenge by infusing FW-GPL into other SOTA models. (3) To save the computing cost, we will fine-tune the pre-trained models which have been inserted with FW-GPL.”

 

5-    Discuss the limitations of the proposed method.

Reply: Thank you again. We have discussed the advantage and the limitation in section 6.4. Advantages and Limitations.

 

6-    The recent references such as 2022 is very few. There are many works in 2022 therefore add some of them. And 7-    The recent references ratio below 50% around 43%. Add more recent references.

Reply: Thank you for your excellent comment. In Table 5, we have removed some old citations.

 

“[1] Gao, B.B.; Xing, C.; Xie, C.W.;Wu, J.; Geng, X. Deep label distribution learning with label ambiguity. IEEE Transactions on Image Processing 2017, 26, 2825–2838. ”

“[7] Liu, H.; Lu, J.; Feng, J.; Zhou, J. Ordinal Deep Feature Learning for Facial Age Estimation. 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017) 2017, pp. 157–164. ”

“[43] Agustsson, E.; Timofte, R.; Gool, L.V. Anchored Regression Networks Applied to Age Estimation and Super Resolution. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), 2017.”

“[44] Zhang, Y.; Liu, L.; Li, C.; Loy, C.C. Quantifying Facial Age by Posterior of Age Comparisons. ArXiv 2017, abs/1708.09687.”

In the introduction, we have added one recent review article (2023) to summarize the ordinal model and plugged two recent ordinal research articles (2021) referring to the label-processed methods. We also have added one recent citation that used two symmetric covariance matrices to capture the underlying correlations from both aspects of input facial features and output age labels.

“Tutz, G. Ordinal regression: A review and a taxonomy of models. Wiley Interdisciplinary Reviews: Computational Statistics 2022, 14, e1545.”

“Chen, G.; Peng, J.; Wang, L.; Yuan, H.; Huang, Y. Feature constraint reinforcement based age estimation. Multimedia Tools and Applications 2022, pp. 1–22”

“Berg, A.; Oskarsson, M.; O’Connor, M. Deep ordinal regression with label diversity. In Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 2021, pp. 2740–2747.”

“Li, W.; Huang, X.; Lu, J.; Feng, J.; Zhou, J. Learning probabilistic ordinal embeddings for uncertainty-aware regression. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 13896–13905.”

 

9-    In training phase, dataset partition is randomly or not? I suggest u to used K-Fold Cross Validation. And how many K-Fold used?

Reply: Thank you for this excellent comment. To make a fair comparison, we eventually reported the average MAE and RMSE by using 10-Fold Cross-Validation. In the 4.3. Experiment Settings, we have added “Finally, the 10-fold cross-validation is performed.”

 

10-In experiments results, the evaluation (train-test round) must repeated for N round. I suggest to repeat for many rounds to ensure that the bias was minimized.

Reply: Thank you again for this nice comment. After adding “Finally, the 10-fold cross-validation is performed.” into the 4.3. Experiment Settings, we have clarified and reported the 10-fold cross-validation result to avoid the bias of the result.