Evaluating the Effect of Planned Missing Designs in Structural Equation Model Fit Measures

Round 1

Reviewer 1 Report (New Reviewer)

Thanks for affording me the opportunity to review this study. I believe that the findings of this research could offer valuable insights into the utilisation of structural equation model. I have some questions for your consideration to potentially improve the quality of this manuscript.

 

Abstract

1. You should include a brief description of the methods used to generate the data, and a data analysis plan.

Introduction

2. Line 69: It’s important to provide a clear definition of “omission be design” in the introduction. Offer examples to illustrate this concept and discuss its impact on research. Does it affect data quality, efficiency, or any other aspects of the study?

3. Line 73-77: If the content in this paragraph is more methodological in nature, it would be better placed it in the Methods section

Methods

4. Line 98: Review the entire manuscript to eliminate any duplicate phrases or sentences, there is a duplicate “is present” here

5. Clearly state your data analysis plan in the Methods section. Specify which statistical techniques you employed to handle missing data and ensure valid inferences. Since you've mentioned Full Information Maximum Likelihood (FIML) and Multiple Imputation (MI), explain how these were applied to your data.

6. Ensure that figures and tables are presented in a clear and organized format.

7. In the Discussion section, provide a dedicated subsection for discussing the limitations of your study. Additionally, offer insights into future directions for research based on your findings.

8. Summarize the main findings of your study in the Conclusions section. Offer concise statements about the implications of your research and its contributions to the field.

Revise

Author Response

Dear Reviewer

I would like to express my gratitude for the time and effort you have invested in reviewing my manuscript titled “Evaluating the effect of planned missing designs in the structural equation model fit measures” submitted to psych. Your insightful comments and constructive feedback have greatly contributed to the improvement of my research, and I sincerely appreciate your expertise in the field.

I have carefully considered all your comments and suggestions, and I have made the necessary revisions to address the concerns raised. I will address each of the points mentioned and explain the changes I have made accordingly.

1. Comment: Abstract – “You should include a brief description of the methods used to generate the data, and a data analysis plan.”

Response: In response to your request, I have added one sentence at the end of the abstract describing the methods and data analysis plan.

2. Comment: Introduction, line 69 – “It’s important to provide a clear definition of “omission be design” in the introduction. Offer examples to illustrate this concept and discuss its impact on research. Does it affect data quality, efficiency, or any other aspects of the study?”

Response: I have introduced an entire paragraph in the introduction to meet this request. (Now is line 67).

3. Comment: Introduction, line 73-77 – “If the content in this paragraph is more methodological in nature, it would be better placed it in the Methods section”

Response: I have rewritten this paragraph in which I present the structure of the manuscript. (Lines 76-80).

4. Comment: Methods, line 98 – “Review the entire manuscript to eliminate any duplicate phrases or sentences, there is a duplicate “is present” here“

Response: The entire manuscript has now been revised and all spelling and grammatical mistakes have been corrected. The numbering of the lines has thus changed.

5. Comment: Methods – “Specify which statistical techniques you employed to handle missing data and ensure valid inferences. Since you've mentioned Full Information Maximum Likelihood (FIML) and Multiple Imputation (MI), explain how these were applied to your data.”

Response: In the Methods section, I present the FIML and MI methods and discuss the differences between them (Lines 136-149).

6. Comment: Results – “Ensure that figures and tables are presented in a clear and organized format”

Response: I have replaced all the figures with tables (Tables 4 to 7), aiming at clarifying the results section.

7. Comment: Discussion – “Provide a dedicated subsection for discussing the limitations of your study. Additionally, offer insights into future directions for research based on your findings.”

Response: In the Discussion section, I discuss the limitations of my work and, based on my findings, present some ideas for future work. 

8. Comment: Conclusions – “Summarize the main findings of your study in the Conclusions section. Offer concise statements about the implications of your research and its contributions to the field.”

Response: I have summarized my results and also highlighted how they contribute to the field of Psychometrics and Educational Measurement.

Once again, I sincerely appreciate your time and effort in reviewing my work and I believe that the revised version incorporates the suggested changes and addresses the concerns raised during the review process.  I am available to clarify any doubts or misunderstandings.

Reviewer 2 Report (New Reviewer)

Thank you for the opportunity to review this paper. The manuscript entitled "Evaluating the effect of planned missing designs in the structural equation model fit measures" aimed to explore the effect of the non-responses due to a specific planned missing design on the mentioned fit indexes when adjusting a structural equation model.

 

I find that the authors have done an excellent job and I congratulate them. I think the manuscript is almost ready for publication. I just think the manuscript needs proofreading according to journal guidelines.

 

For example, the author's name + "et al." is listed throughout the manuscript. in the text. When we are in the text, it is better to write "and colleagues" rather than "et al.", which is preferred in quotations in parentheses.

 

Even when two authors are repeated, for example in the case of line 63, "Liu et al. [19] and Liu et al. [20]" it is better to avoid repetitions and write "Liu et al. [19; 20]".

 

All the yellow highlighting must be removed and only one line spacing must be kept, as indicated by the journal. Editing is an important part.

 

I hope that these comments can enrich your work even more and I congratulate you on the excellent manuscript received. Thank you.

Author Response

Dear Reviewer

I would like to express my gratitude for the time and effort you have invested in reviewing my manuscript titled “Evaluating the effect of planned missing designs in the structural equation model fit measures” submitted to psych. Your insightful comments and constructive feedback have greatly contributed to the improvement of my research, and I sincerely appreciate your expertise in the field.

I have carefully considered all your comments and suggestions, and I have made the necessary revisions to address the concerns raised. I will address each of the points mentioned and explain the changes I have made accordingly.

1. Comment: “When we are in the text, it is better to write "and colleagues" rather than "et al.", …”

Response: I have made the suggested change.

2. Comment: “Even when two authors are repeated, for example in the case of line 63, "Liu et al. [19] and Liu et al. [20]" it is better to avoid repetitions and write "Liu et al. [19; 20]".”

Response: I have made the suggested change.

3. Comment: “All the yellow highlighting must be removed and only one line spacing must be kept, as indicated by the journal.”

Response: I have made the suggested modifications.

Once again, I sincerely appreciate your time and effort in reviewing my work and I believe that the revised version incorporates the suggested changes and addresses the concerns raised during the review process. I am available to clarify any doubts or misunderstandings.

Reviewer 3 Report (New Reviewer)

In this paper, the authors investigate the impact of planned missing designs on fit measures in structural equation modeling. A planned missing design intentionally introduces non-responses as per the researcher's discretion, aiming to enhance data quality and reduce inquiry efforts. The assessment of model fit in structural equation modeling involves various criteria, including the root mean square error of approximation (RMSEA), standardized root mean square residual (SRMR), comparative fit index (CFI), and Tucker-Lewis index (TLI).

 

The primary objective of this study is to analyze the influence of non-responses stemming from a specific planned missing design, specifically the 3-form design, on the aforementioned fit indices during the process of adjusting a structural equation model. To achieve this, a simulation study is conducted, encompassing both correctly specified models and models with misspecified correlations among factors.

 

The findings highlight that the CFI, TLI, and SRMR indices exhibit sensitivity to non-responses, particularly in scenarios involving small sample sizes, low factor loadings, and a high number of observed variables. Additionally, the presence of non-responses in models with misspecified correlations leads to unsatisfactory values for all four fit indices under consideration. This effect is more pronounced when there exists a strong correlation between factors.

 

Overall, this study contributes to our understanding of the ramifications of planned missing designs on fit measures in structural equation modeling. The results underscore the necessity of considering non-response patterns, especially when dealing with misspecified models and factors with substantial correlations.

ok

Author Response

Dear Reviewer

I would like to express my gratitude for the time and effort you have invested in reviewing my manuscript titled “Evaluating the effect of planned missing designs in the structural equation model fit measures” submitted to psych. Your insightful comments and constructive feedback have greatly contributed to the improvement of my research, and I sincerely appreciate your expertise in the field.

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

The simulation study is well designed and the results are clearly presented. 

Nevertheless,  I think the manuscript can be improved by strengthening the justification of the study.  Why are planned missing designs preferable to a reduced number of items, especially when the forced response options are available?

Another suggestion is to improve the discussion section. It is important to discuss the limitations of the current study and provide suggestions for further research. 

 

Writing clarity could be improved. Some phrases are not commonly used and seem awkward. 

Author Response

Dear Reviewer

I would like to express my gratitude for the time and effort you have invested in reviewing my manuscript titled “Evaluating the effect of planned missing designs in the structural equation model fit measures” submitted to the special issue of psych, Psychometrics and Educational Measurement. Your insightful comments and constructive feedback have greatly contributed to the improvement of my research, and I sincerely appreciate your expertise in the field.

I have carefully considered all your comments and suggestions, and I have made the necessary revisions to address the concerns raised. I will address each of the points mentioned and explain the changes I have made accordingly.

1. Comment: Why are planned missing designs preferable to a reduced number of items, especially when the forced response options are available?

Response: I appreciate your suggestion regarding the topic forced response options. In response, I have introduced a paragraph in section 2.1 Planned Missing Design, where I talk about this topic and show why are planned missing design a better option. I have also included additional references to support my arguments.

2. Comment: Another suggestion is to improve the discussion section. It is important to discuss the limitations of the current study and provide suggestions for further research. 

Response: I appreciate your comments about the discussion section. I have revised all this section to include a broader discussion of the implications of my results and how they align or not with other relevant methodological works in this area of knowledge. Also, I include a new section Conclusions and Future Work, where I present the main conclusions and show the limitations of my study, as well some ideas for future work.

Once again, I sincerely appreciate your time and effort in reviewing my work. Your valuable feedback has significantly contributed to the improvement of I article. I believe that the revised version incorporates the suggested changes and addresses the concerns raised during the review process, but I am available to clarify any doubts or misunderstandings.

 

Reviewer 2 Report

Dear authors,

I have the pleasure to revise your manuscript "Evaluating the effect of planned missing designs in the structural equation model fit measures".

In my opinion the manuscript is valuable for the journal and for the academic community. I suggest some minor revision in order to improve your paper.

1) Add a conclusion for your paper 

2) add a part with specific implication

Author Response

Dear Reviewer

I would like to express my gratitude for the time and effort you have invested in reviewing my manuscript titled “Evaluating the effect of planned missing designs in the structural equation model fit measures” submitted to the special issue of psych, Psychometrics and Educational Measurement. Your insightful comments and constructive feedback have greatly contributed to the improvement of my research, and I sincerely appreciate your expertise in the field.

I have carefully considered all your comments and suggestions, and I have made the necessary revisions to address the concerns raised. I will address each of the points mentioned and explain the changes I have made accordingly.

1. Comment: Add a conclusion for your paper 

Response: I appreciate your comments about the discussion section. I have revised all this section to include a broader discussion of the implications of my results and how they align or not with other relevant methodological works in this area of knowledge. Also, I include a new section Conclusions and Future Work, where I present the main conclusions and show the limitations of my study, as well some ideas for future work.

2. Comment: add a part with specific implication

Response: I appreciate your suggestion. I revised the results section, which is now simpler and clearer. Thus, I believe that your recommendation is followed.

Once again, I sincerely appreciate your time and effort in reviewing my work. Your valuable feedback has significantly contributed to the improvement of I article. I believe that the revised version incorporates the suggested changes and addresses the concerns raised during the review process, but I am available to clarify any doubts or misunderstandings.

Reviewer 3 Report

The issue raised in this manuscript is important and novel. Planned missing designs are a valuable tool for researchers as it enables them to answer more questions with the same amount of effort on the part of participants. The present manuscript contributes evidence regarding the trustworthiness of fit indices in CFAs involving planned missing data. Here are some suggestions I have for improving the manuscript:

(1) The models being used could be more closely aligned with those that would actually be used in research involving planned missing designs. Planned missing design is not a likely choice for a measurement study. Instead, planned missing is likely to be used in studies where structural questions are being asked (i.e., the relationships amongst substantively relevant variables is of interest). More generally, the choice of models considered was not justified.

(2) The ratio of numbers of variables in form X to forms A, B, and C was also not justified. I am not an expert in planned missing designs, but in my limited experience, they involve more missingness than was considered in this study. In any case, the amount of planned missingness should be manipulated - a good question to ask would be: at what point is there too much missingness?

(3) Comparing fit of complete and planned missing designs is a major strength of this study. However, modern psychometric research on fit testing should be considered including (for example) equivalence testing forms of fit indices or model-specific fit index testing (e.g., recent work by McNeish). The growing inaccuracy of fit indices as indicator quality diminishes is well documented by Hancock and others

(4) The result section is awfully wordy and not very well organized. The focus should be on comparing complete and planned missing results, as is done in the figures. The tables are largely extraneous. It needs to be clear from the results which factors (and interactions of factors!) matter. 

(5) It's not clear what is being reported in the results - are the listed values mean values? median values? Would proportion of values classified as not fitting according to a cutoff be a better thing to track? 

(6) Since several factors appear to be irrelevant, it might be worth considering an ANOVA approach to analyzing the simulation data. A finding of non-significance (or very small effect size) for certain factors/interactions would justify marginalizing across those irrelevant factors and greatly simplify the presentation of results.

(7) The discussion section seems... thin. A discussion section is not a retelling of the results, but rather an interpretation and contextualization of them. What new questions arise because of this work? What additional concerns need to be addressed for researchers to be confident using planned missing designs? 

The quality of writing in this manuscript is adequate, but it is apparent the author is not a native English speaker.

Author Response

Dear Reviewer

I would like to express my gratitude for the time and effort you have invested in reviewing my manuscript titled “Evaluating the effect of planned missing designs in the structural equation model fit measures” submitted to the special issue of psych, Psychometrics and Educational Measurement. Your insightful comments and constructive feedback have greatly contributed to the improvement of my research, and I sincerely appreciate your expertise in the field.

I have carefully considered all your comments and suggestions, and I have made the necessary revisions to address the concerns raised. I will address each of the points mentioned and explain the changes I have made accordingly.

1. Comment: The models being used could be more closely aligned with those that would actually be used in research involving planned missing designs. Planned missing design is not a likely choice for a measurement study. Instead, planned missing is likely to be used in studies where structural questions are being asked (i.e., the relationship amongst substantively relevant variables is of interest). More generally, the choice of models considered was not justified.

Response: I appreciate your suggestion regarding the models under analysis. I include some new references of applied studies that use a CFA with two factors, to justify the choice. In fact, my choice was following the work of Muthén and Muthén 2002. In their work, a CFA with two factors is considered. In this way, I have considered a CFA with one factor to understand the effect of the number of factors in the model.

2. Comment: The ratio of numbers of variables in form X to forms A, B, and C was also not justified. I am not an expert in planned missing designs, but in my limited experience, they involve more missingness than was considered in this study. In any case, the amount of planned missingness should be manipulated - a good question to ask would be: at what point is there too much missingness?

Response: I appreciate your question, and you are right, the percentage of missing data in the 3-form considered, is a limitation of the study. I follow the work of Schoemann 2014. I have now introduced a new section Conclusions and Future Work and I mentioned this fact as a limitation.

3. Comment: Comparing fit of complete and planned missing designs is a major strength of this study. However, modern psychometric research on fit testing should be considered including (for example) equivalence testing forms of fit indices or model-specific fit index testing (e.g., recent work by McNeish). The growing inaccuracy of fit indices as indicator quality diminishes is well documented by Hancock and others.

Response: I appreciate your suggestion, and, in the discussion section, that was reformulated, I compare my results with McNeish et al., 2018, and Hancock and Mueller, 2010. Furthermore, I mention the reliability paradox.

4. Comment: The result section is awfully wordy and not very well organized. The focus should be on comparing complete and planned missing results, as is done in the figures. The tables are largely extraneous. It needs to be clear from the results which factors (and interactions of factors!) matter. 

Response: I appreciate your suggestion, and revised the results section, I agree with you that was a little confusing. I maintained the tables with the results to justify what I say. However, for the comparison of complete versus missing data, the figures show in a quick way the differences between complete data and missing data.

5. Comment: It's not clear what is being reported in the results - are the listed values mean values? median values? Would proportion of values classified as not fitting according to a cutoff be a better thing to track? 

Response: I appreciate your question, in the results section, the mean values for each index over 1000 replications are presented and, in parenthesis, the respective standard deviation. These are the results disposable by package Simsem of R.

6. Comment: Since several factors appear to be irrelevant, it might be worth considering an ANOVA approach to analyzing the simulation data. A finding of non-significance (or very small effect size) for certain factors/interactions would justify marginalizing across those irrelevant factors and greatly simplify the presentation of results.

Response: I appreciate your suggestion and I perform an ANOVA analysis to understand whether the values for CFI and SRMR indexes are affected by the correlation parameter. This is the only considered parameter as this was the variable considered in the simulation study for which the conclusion was not clear.

7. Comment: The discussion section seems... thin. A discussion section is not a retelling of the results, but rather an interpretation and contextualization of them. What new questions arise because of this work? What additional concerns need to be addressed for researchers to be confident using planned missing designs? 

Response: I appreciate your comments about the discussion section. I have revised all this section to include a broader discussion of the implications of my results and how they align or not with other relevant methodological works in this area of knowledge. Also, I include a new section Conclusions and Future Work, where I present the main conclusions and show the limitations of my study, as well some ideas for future work.

 

Once again, I sincerely appreciate your time and effort in reviewing my work. Your valuable feedback has significantly contributed to the improvement of I article. I believe that the revised version incorporates the suggested changes and addresses the concerns raised during the review process, but I am available to clarify any doubts or misunderstandings.