Sim-to-Real Dataset of Industrial Metal Objects

Round 1

Reviewer 1 Report (New Reviewer)

Comments and Suggestions for Authors

This paper presents an image dataset with 6D object pose labels and a public tool for 6D pose labeling of multi-view data. The dataset consists of both real-world collected data and synthetic data, which will be useful for computer vision tasks.

The paper has presented the setup and procedures for data collection, the organization of the dataset, and the usage of the labeling tool. The paper presentation is clear and well-organized.

Author Response

Dear Reviewer,

Thank you for your positive feedback. We appreciate your recognition of the clarity and organization of our work. Your comments encourage us to continue to strive towards contributing to our field.

Best regards,
Peter De Roovere

Reviewer 2 Report (New Reviewer)

Comments and Suggestions for Authors

The manuscript presented a diverse dataset of industrial, reflective objects with real-world and synthetic data. Real-world data is obtained by recording multi-view images of scenes with varying object shapes, materials, carriers, compositions, and lighting conditions. Synthetic data is obtained by carefully simulating real-world and varying environmental conditions in a controlled and realistic way.

It seems that this paper is an intermediate version, and some important phrases are still marked in blue. I think there is still some room for improvement in this paper, which is mainly reflected in the following aspects:

1. I suggest the author add some experiments to verify the effectiveness of the dataset in 6D pose estimation, object detection, instance segmentation, 3D reconstruction, or active perception. It becomes more convincing when combined with applications.

2. The motivation of the dataset should be clarified. I suggest the authors show some examples that existing methods for 3D reconstruction works well in existing dataset. However, these methods fail to work in the proposed dataset.

3. The references are somewhat weak. It is recommended to add the latest literature from top journals.

Comments on the Quality of English Language

Minor editing of English language required.

Author Response

Dear Reviewer,
Thank you for taking the time to review our manuscript and for providing your valuable comments. We appreciate the opportunity to improve our paper based on your feedback. Please find below our responses to the issues you raised:
0. Regarding the Blue Marked Phrases: Thank you for pointing out the blue-marked phrases. We intentionally kept these markings to indicate the sections revised following the last round of reviews. Similarly, any changes made after your last review are now highlighted.
1. Addition of Experiments: We agree with your suggestion to include experiments demonstrating the effectiveness of our dataset in various applications. Accordingly, we have added a new section (3.3 Usage) to our manuscript, which showcases recent peer-reviewed research that utilized our dataset, underscoring its unique value. Notably, one study used our dataset to enhance object detection by exploring synthetic variations. Another research investigated the challenges in 6D pose estimation methods using our dataset, highlighting its utility in advancing reflective and textureless object recognition and multi-view data analysis.
2. Clarification of the Dataset's Motivation: We have revised the introduction to better explain the motivation behind our dataset. We have included a specific example (Figure 2) to illustrate the limitations of current 3D reconstruction and 2D segmentation methods when applied to reflective objects, compared to their performance with textured, non-reflective objects.
3. Strengthening the References: We have taken your advice to reinforce our reference list. We have added recent publications from high-impact journals (references [3] and [14]).
We believe these revisions address your concerns and significantly improve the manuscript's contribution to the field. We are grateful for the opportunity to improve our work and welcome any further suggestions you may have.
Sincerely, Peter De Roovere

Reviewer 3 Report (New Reviewer)

Comments and Suggestions for Authors

Dear authors 

This is indeed an interesting read and the paper is scientifically sound. There are ethical concerns with regards to using picture/ pictures unless a written permission is granted from the owner. 

Comments on the Quality of English Language

English is fine.

Author Response

Dear Reviewer,
Thank you very much for taking the time to review our manuscript. We really appreciate your positive feedback and the opportunity to address any concerns you may have raised.
We understand that you had some ethical concerns regarding the use of images in our manuscript. We want to assure you that we have all the necessary permissions and own the rights to all the pictures used. We have also taken great care to ensure that all ethical guidelines were followed.
We appreciate your feedback on the language used in our manuscript. We noted that you marked the checkbox indicating "English very difficult to understand/incomprehensible" but also stated that the language in the manuscript is acceptable. To clarify, we have thoroughly reviewed the manuscript and ensured that the English language used meets academic standards. We believe that the language is clear, comprehensible, and meets the requirements of the publication.
We are committed to addressing any further queries or concerns you may have and are open to making any necessary revisions. We appreciate your guidance on the next steps in the publication process.
Once again, thank you for your valuable feedback and for considering our work for publication.
Sincerely,

Reviewer 4 Report (New Reviewer)

Comments and Suggestions for Authors

I have the following concerns that need to be addressed before the paper can be accepted for publication.

1. Please void clumped references, e.g., [3-7]. It makes it difficult for the audience/readers to ascertain why each reference was cited or what contribution they make to your paper. Please keep it to a maximum of 2 references, if absolutely essential, when clumping references.

2. In table 1, it is unclear how the last column, i.e., ease of use, is decided. It appears to the general audience/reader to be subjective. There needs to be a more detailed explanation on this so that it is clear for the audience to understand and to make clear this is not extremely subjective.

3. Under materials and methods there is a section called 'Way of working'. This section feels inadequate. The structure of the paper and its presentation is unclear, i.e., not easy for the reader to follow. I would recommend possibly including a flowchart and/or algorithm that is simple for the reader to follow to understand the steps involved in applying your approach.

4. Future planned worked by the authors is unclear in the conclusion, although they do mention the broader impact of their work (this appears more for the general public to pursue). It is important that the authors clarify what their next steps are.

Author Response

Dear Reviewer,

Thank you for taking the time to review our manuscript and providing us with useful feedback. We have made the following changes to address your comments:

1. Clumped references: We have removed all clumped references from our paper.

2. Ease of Use column in Table 1: We appreciate your feedback on the subjectivity of this assessment. To address this issue, we have included a detailed rubric in Appendix A that evaluates the ease of use of 6D object pose labeling tools. Additionally, we have added scores for each labeling tool on each category introduced in the rubric.

3. Way of Working section under Materials and Methods: We have recognized the need for clarity in this section and have therefore restructured the paragraph and incorporated a flowchart. These improvements aim to make the methodology more accessible and easier to follow for the reader.

4. Clarity of future work: We have added a new paragraph in Section 4 (Conclusion), which clearly outlines our future research directions. This addition ensures that our planned next steps are explicitly stated and align with the broader impact of our work.

We appreciate the opportunity to revise our manuscript based on your comments and hope that our changes meet with your approval.

Thank you for your guidance and consideration.

Sincerely,
Peter De Roovere

Round 2

Reviewer 2 Report (New Reviewer)

Comments and Suggestions for Authors

The authors have resolved all my concerns. The paper is ready for publication.

Reviewer 4 Report (New Reviewer)

Comments and Suggestions for Authors

Thank you for making the suggested edits.

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The paper presents a dataset of industrial metal objects with characteristics such as symmetry, texturelessness, and high reflectiveness. The dataset comprises real-world and synthetic multi-view RGB images with 6D object pose labels. Also, the paper presents a tool that enables quick annotation of 6D object pose labels in multi-view images. The tool can provide 6D object pose labels for all real-world images.

The dataset per si can not be considered a scientific or technological contribution. Also, the paper needs to describe references for a comparative analysis with state-of-the-art works. There are plenty of tools for labeling data for machine learning techniques, and they also were not used for analysis. 

Finally, it could be more technologically relevant to describing a dataset construction methodology considering the industrial environment's requirements. 

Comments on the Quality of English Language

The paper is well written, permitting an easy reading process. 

Reviewer 2 Report

Comments and Suggestions for Authors The text is interesting in the sense of showing the methodology for creating the database, but it is very specific in its structure, which makes the real usefulness of the database debatable.

Reviewer 3 Report

Comments and Suggestions for Authors

The topic is promising. The manuscript presents a Dataset helpful to several industrial applications, simplifying the complex problem of metal capture. The abstract section describes the problem-solution, methodology, results, and applications. The conclusion section describes clearly the scientific contribution, scope, limits, and industrial application. 

Comments on the Quality of English Language

The Quality of the English Language requires minimum improvements.