Performance Evaluation of Oral Health Teams in Brazil: An Item Response Theory Approach
, Juliana V. M. Mambrini 3, LetÃcia C. Pinheiro 3, Antônio Thomaz G. Matta-Machado 4 and Renata C. Martins 2,*Round 1
Reviewer 1 Report
Comments and Suggestions for AuthorsIt is a well-written manuscript and could be interesting for dentistry researchers. I have the following suggestions:
1. Please review the meaning of the abbreviations in the abstract and in the whole text when they are mentioned for the first time.
2. Please follow the strobe guidelines for observational studies.
Thank you.
Author Response
It is a well-written manuscript and could be interesting for dentistry researchers. I have the following suggestions
- Please review the meaning of the abbreviations in the abstract and in the whole text when they are mentioned for the first time.
Response: Thanks for the comment. The meaning of the abbreviations in the abstract and in the whole text were checked.
- Please follow the strobe guidelines for observational studies.
Response: Thanks for the comment. The manuscript has been revised according to the strobe guidelines.
Reviewer 2 Report
Comments and Suggestions for AuthorsThe present study aims to describe the actions performed by OHTs within PHC in Brazil and the relationship of contextual aspects that lead to different levels of OHT performance in the 3rd cycle of evaluation of PMAQ-AB. The null hypothesis of this study was that there is no impact of contextual factors on the performance of Brazilian OHTs.
This is a very interesting study.
Could the authors please describe more aspects about the instrument developed by MofH?
Could the authors insert more informations in the results.
Could the authors please add more recent published articles in the discussion.
Comments on the Quality of English LanguageModerate
Author Response
The present study aims to describe the actions performed by OHTs within PHC in Brazil and the relationship of contextual aspects that lead to different levels of OHT performance in the 3rd cycle of evaluation of PMAQ-AB. The null hypothesis of this study was that there is no impact of contextual factors on the performance of Brazilian OHTs.
This is a very interesting study.
- Could the authors please describe more aspects about the instrument developed by MofH?
Response: Thanks for the comment. The text describing more aspects of the instrument and interviews was inserted in the 4th and 5th paragraphs of the Methodology.
Could the authors insert more information in the results?
Response: Thanks for the comment. More information was added to the Results.
Could the authors please add more recent published articles in the discussion?
Response: Thanks for the comment. More recently published articles were added to the Discussion (References 19, 26 and 27).
Reviewer 3 Report
Comments and Suggestions for AuthorsThe submitted manuscript analyzes a questionnaire for the performance evaluation of oral health teams in Brazil utilizing item response theory (IRT) models. The ms is clearly structured. I have a few comments:
1. Abstract, line 26: Also report the size of the correlation in the abstract.
2. 30: Write “p < .001”.
3. 117: It is better to characterize IRT models as “statistical models” instead of “mathematical models.”
4. 136: It is unreasonable that ability scores can only range between -4 and +4. I thought that a normal distribution for the ability variable \theta would be assumed, which means that \theta would have an unrestricted range.
5. 142: Write “ltm” package.
6. Table 1: Also present item p-values (i.e., proportion correct).
7. 168: Also report skewness and kurtosis of the scores. Including a histogram of the scores in the ms might be desirable.
8. Sect. 3: The authors should indicate which ability estimate was used (e.g., the EAP estimate?).
9. Did the authors rely on the normal distribution assumption of \theta in the estimation using the ltm package? Notably, variants for skewed ability variables can be (preferably) employed for estimation (von Davier, 2008; Xu & von Davier, 2008).
von Davier, M. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61(2), 287–307. https://doi.org/10.1348/000711007X193957
Xu, X., & von Davier, M. (2008). Fitting the structured general diagnostic model to NAEP data. ETS Research Report ETS RR-08-27.
Author Response
The submitted manuscript analyzes a questionnaire for the performance evaluation of oral health teams in Brazil utilizing item response theory (IRT) models. The ms is clearly structured. I have a few comments:
- Abstract, line 26: Also report the size of the correlation in the abstract.
Response: Thanks for the comment. This information was added.
- 30: Write “p < .001”.
Response: Thanks for the comment. This correction was done.
- 117: It is better to characterize IRT models as “statistical models” instead of “mathematical models.”
Response: Thanks for the comment. This correction was done.
- 136: It is unreasonable that ability scores can only range between -4 and +4. I thought that a normal distribution for the ability variable \theta would be assumed, which means that \theta would have an unrestricted range.
Response: Thanks for the comment. We agree with the reviewer that the scores can an unrestricted range. The text was corrected.
- 142: Write “ltm” package.
Response: Thanks for the comment. This correction was done.
- Table 1: Also present item p-values (i.e., proportion correct).
Response: Thanks for the comment. We included the proportion of correct response for each item.
- 168: Also report skewness and kurtosis of the scores. Including a histogram of the scores in the ms might be desirable.
Response: Thanks for the comment. The histogram and values were included at the Results.
- Sect. 3: The authors should indicate which ability estimate was used (e.g., the EAP estimate?).
Response: Package ltm fits the graded response model using Marginal Maximum Likelihood Estimation (MMLE). For ability estimate was used the Empirical Bayes method.
- Did the authors rely on the normal distribution assumption of \theta in the estimation using the ltm package? Notably, variants for skewed ability variables can be (preferably) employed for estimation (von Davier, 2008; Xu & von Davier, 2008).
von Davier, M. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61(2), 287–307. https://doi.org/10.1348/000711007X193957
Xu, X., & von Davier, M. (2008). Fitting the structured general diagnostic model to NAEP data. ETS Research Report ETS RR-08-27.
Response: Thanks for the opportunity of clarifying this important issue. We haven't identified in the literature any specific studies that address the effect of deviation from the Normality of q for the graded response model (Samejima Model), but the literature shows that both the estimates of the item parameters and the estimates of abilities are robust to deviations from the normality of the distribution of scores (Xu & Jia, 2011). Although we observed asymmetry in the distribution, there is no evidence of the presence of multiple populations, with multimodal distribution, for example, which could significantly impact the estimates of the item parameters and the score. Taking this into account, and considering the size of the sample used in the study, we believe that, although it is possible to adjust q by other models, the impact on the estimates presented in the manuscript will be substantively irrelevant. Furthermore, the score is not being used for resource allocation purposes, team ranking, or other purposes whose "fine-tuning" could have some impact on the health service. Finally, considering the asymmetry of the distribution of the scores, the calculation of the correlation between the HDI and Gini indicators and the performance of the teams (score) was redone using Spearman's correlation coefficient.
Round 2
Reviewer 3 Report
Comments and Suggestions for Authors
I disagree with the authors regarding the response to my previous Comment 9. The observed distribution of scores is skewed, and the items are relatively easy. In this case (given the fact that it is short), it is quite likely that the assumed theta distribution can impact item parameters. Moreover, you incorrectly argued that von Davier (2008) and Xu and von Davier (2008) did not address the graded response model. This is a nonsensical statement. You have dichotomous data. In this case, the graded response model is the two-parameter logistic model. This model is addressed in the work of von Daveir. You are advised to redo the analysis using more flexible distributions for the ability theta. There are several R packages that are able to do so. But, of course, the ltm package does not offer this functionality.
von Davier, M. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61(2), 287–307. https://doi.org/10.1348/000711007X193957
Xu, X., & von Davier, M. (2008). Fitting the structured general diagnostic model to NAEP data. ETS Research Report ETS RR-08-27.
Author Response
Dear Reviewer,
we would like to thank you for your revision. All the analyses were redone, using the IRTest package (Li, S. (2023). IRTest: Parameter estimation of item response theory with an estimation of latent distribution (Version 1.12.0). R package.). This package uses latent distribution estimation methods that enhance the estimation accuracy and free the normality assumption on the latent distribution. All new results were included in the study.
Round 3
Reviewer 3 Report
Comments and Suggestions for Authorsno further comments
