Model Building in Forensic Psychiatry: A Machine Learning Approach to Screening Offender Patients with SSD
Round 1
Reviewer 1 Report
The paper aims to develop ML models to predict the outcome of adverse treatment responses for patients with Schizophrenia spectrum disorders, and also to identify the major predictive factors.
The description of the dataset and the methodologies are clear and convincing. Although the models only achieve moderate accuracy for the intended task, the work is interesting and presents a good degree of novelty.
The paper woul benefit from the review/discussion from the literature about previous works published in the field of machine learning in fosensic psychiatry.
Author Response
Reviewer 1:
The paper would benefit from the review/discussion from the literature about previous works published in the field of machine learning in forensic psychiatry.
Dear Sir or Madam, thank you very much for the diligent review of our manuscript, your positive feedback about it and the helpful suggestions. We have added a paragraph in the discussion in which we provided a summary of previous works in ML and forensic psychiatry and elaborate on the different approach of the current manuscript in comparison to the aforementioned ones (line 244 ff):
“So far, the application of ML is rare in the field of forensic psychiatry. Previous studies have mainly explored heterogenous forensic populations, e. g. for purposes of recidivism risk prediction and have not focused on patients with SSD in particular. The authors’ former publications, which evaluated a more homogenous population of offender patients with SSD exclusively, mainly focused on providing a better under-standing of complex, multifactorial phenomena, such as stress, criminal recidivism, migration experience, self-harm and aggressive behavior.”
Reviewer 2 Report
Dear editors,
Thank you very much for the opportunity to review diagnostics-1915070 Model Building in Forensic Psychiatry: A Machine Learning Approach to Screening Offender Patients with SSD.
The study tackles a highly relevant (scientific) development and is remarkably well written. Additionally the context in which the data has been collected (of course) provides the opportunity to develop and test models under reasonable controlled conditions, what enhances the validity of the findings though obviously restricts the overall applicability with regards to other patient / convict populations. Nevertheless an excellent testing ground to develop the decision support technologies such as the ones being presented here.
Despite the overall merit of the presented scientific work done, some considerations however remain. Most notably how well it discriminates itself from previously published work. A concern that I consider to be a major issue to be covered before publication can be considered. See comments below on how this could be resolved. All other comments can be considered ‘minor’ (though it would of course be appreciated if they would be used to strengthen the manuscript).
Major point:
Lines 84 – 91 It would / could be very relevant to extent this paragraph with some insight in how the used (analytical / mathematical / statistical) methodology in the present study differentiates from the (analytical) methodologies applied in the other (referred) studies.
Line 92-92: Consequently, what is than the relevance / necessity / rationale for the present study? See previous comment, please indicate and emphasize more clearly how the present study distinguishes itself from the previously published ones. And, of course, what new insights it aims to provide that can be specifically associated with here presented alternative analytical strategy.
Minor points:
Line 59: abbreviations (provide proper definitions first time an abbreviation is being used).
Line 116 table 1. From the socio-demographic information I understand that a. males are highly over-represented (not necessarily a surprise), though wouldn’t it consequently make sense to exclude the very small female sample? Why (not)? And b. I interpret the information presented in the two bottom rows of the table as all (diagnosed) SSD patients were ‘single (at offence)’ and all non-diagnosed (or diagnosed but classified as ‘negative’ on SSD) did have a relationship at the time of offence. That is correct? Please clarify in the manuscript.
Line 122 What method was used to calculate / estimate / correct for (the values of the) missing data? How was this approached?
Line 131 – 134 This statement seems to very much address the fundamental relevance of studies like the present one: in a complicated environment / under challenging circumstances finding the smallest set of factors that can nevertheless with (reasonable) accuracy predict / indicate where specific risks may present themselves and how scarce resources (e.g. man hours, medical supplies and attention, detention infrastructure, etc.) could best be distributed to come to optimal results (i.e. not being structurally ‘overextended’). Challenge than becomes: what methodological approach appears to be able to do so (as said, for ML analytics on typically relatively small datasets), without significantly compromising validity and/or reliability? For that reason I suppose that scientific efforts like the present one are highly relevant.
Line 171 Table 3: Variable code is not relevant for the reader (and/or for the interpretation of the results)? Assessment method (of that specific variable) nevertheless is. More specifically, please indicate (e.g. in a separate column) what assessment instrument (e.g. a specific behavioral observation format, (self-report) questionnaire, records from justice department, reports of observations of professionals working with the convict, … etc.) was used to obtain every specific factor (incorporated in the model(s)).
Line 192 Please provide factor definitions in the figure itself instead of variable codes (with the legend under the figure).
Line 210 -223 and apart from the very true arguments already provided, in the field of forensic psychology / psychiatry there is (from personal professional experience) at least one additional relevance, primarily from the public prosecution perspective: accountability. Providing the scientifically sound decision support method that helps to come to (well) specified decisions to be made on individual cases (focusing on circumventing potential risks, for the patient / convict involved, the related professionals and society). That justifies a more ‘personal’ medical and juridical judgement and might consequently be a juridical very valid argument to differentiate in imposed penitentiary regimes.
Author Response
Dear Sir or Madam, thank you very much for your diligent and meticulous review and helpful comments, which we highly appreciate. We are very pleased to hear your positive feedback on our manuscript, including your feedback on the distinct suitability of our study population.
Please find our reply below each paragraph:
Major point:
Lines 84 – 91 It would / could be very relevant to extent this paragraph with some insight in how the used (analytical / mathematical / statistical) methodology in the present study differentiates from the (analytical) methodologies applied in the other (referred) studies.
Line 92-92: Consequently, what is than the relevance / necessity / rationale for the present study? See previous comment, please indicate and emphasize more clearly how the present study distinguishes itself from the previously published ones. And, of course, what new insights it aims to provide that can be specifically associated with here presented alternative analytical strategy.
Thank you for pointing this out. We have now tried to include a short paragraph differentiating out previous ML approaches from the current manuscript (line 87 ff, see below):
“Whereas the aforementioned studies by the authors have so far aimed to exploratively examine the complex interplay of various influential factors in different phenomena to provide a better understanding of them and have tested other outcome variables (e. g. self-harm), the present work is a first approach to develop a clinical model for the prediction of certain events during the hospitalization in question.”
We have also inserted a short paragraph in the discussion section discussing how previous works have focused more on exploring phenomena rather than on the development of predictive tools (with the exception of studies on more heterogenous offender populations, see line 244 ff). Please let us know if you deem a more detailed differentiation to be sensible.
Minor points:
Line 59: abbreviations (provide proper definitions first time an abbreviation is being used).
Thank you for pointing this out. We have included the full name of the risk assessment tools as well as the abbreviation in brackets:
“Analogous to the risk assessment tools commonly used in forensic psychiatry to assess the risk of reoffending, for example the Historical Clinical and Risk Management 20 (HCR-20) or the Violence Risk Appraisal Guide (VRAG)G, there is thus a need for a tool to predict adverse treatment courses in the correctional system”
Line 116 table 1. From the socio-demographic information I understand that a. males are highly over-represented (not necessarily a surprise), though wouldn’t it consequently make sense to exclude the very small female sample? Why (not)? And b. I interpret the information presented in the two bottom rows of the table as all (diagnosed) SSD patients were ‘single (at offence)’ and all non-diagnosed (or diagnosed but classified as ‘negative’ on SSD) did have a relationship at the time of offence. That is correct? Please clarify in the manuscript.
We have clarified the information of the two bottom rows of table 1 in the continuous text as following (line 121 ff):
“The majority of patients was single at the time of the offence leading to the referenced forensic hospitalization, and was diagnosed with schizophrenia (F20.x acc. to ICD-10), while other diagnoses from the psychosis spectrum, for example schizoaffective disorder, were less prevalent (see Table 1).»
We decided to not exclude female subjects as, just like you pointed out, a predominantly male population with few women best reflects the reality in forensic psychiatry (we have included a short sentence about our rationale in this decision in the limitations section, see line 346 ff).
Line 122 What method was used to calculate / estimate / correct for (the values of the) missing data? How was this approached?
We have elaborated on statistical procedures regarding missing values (line 131 ff):
“To enable the flexible application of all ML algorithms, imputation of missing values was carried out by mean for continuous variables and by mode for categorical variables included in the MLR package, and imputation weights saved for later were reused on the validation subset (see Figure 1, Step 3a).»
Line 131 – 134 This statement seems to very much address the fundamental relevance of studies like the present one: in a complicated environment / under challenging circumstances finding the smallest set of factors that can nevertheless with (reasonable) accuracy predict / indicate where specific risks may present themselves and how scarce resources (e.g. man hours, medical supplies and attention, detention infrastructure, etc.) could best be distributed to come to optimal results (i.e. not being structurally ‘overextended’). Challenge than becomes: what methodological approach appears to be able to do so (as said, for ML analytics on typically relatively small datasets), without significantly compromising validity and/or reliability? For that reason I suppose that scientific efforts like the present one are highly relevant.
An excellent summarized statement of the core of our research concern. We have taken the liberty to include this in the conclusions section (line 350ff, with your permission).
We would also like to refer to line 335 ff in the paragraph on limitations, where, if we understand your concern correctly, we tried to address this as well.
Line 171 Table 3: Variable code is not relevant for the reader (and/or for the interpretation of the results)? Assessment method (of that specific variable) nevertheless is. More specifically, please indicate (e.g. in a separate column) what assessment instrument (e.g. a specific behavioral observation format, (self-report) questionnaire, records from justice department, reports of observations of professionals working with the convict, … etc.) was used to obtain every specific factor (incorporated in the model(s)).
Absolutely – the assessment method of each variable seems very valuable to the reader, especially for the sake of reproducibility. The reason we decided to include the variable code as well is that the reader can access a detailed table with definitions and assessment methods of all 209 variables which were tested (we have provided the link in the data availability statement). In this document, navigation is easiest using the variable code. In our opinion, due to the rather detailed description of each variable, providing the assessment method for each variable within the table compromises readability and clarity of the table
- g. definition of DZ10 Rule Breaking: ‘Breaking rules’ refers to a lack of obedience to ward rules of the hospital setting (does not include rule violations listed in DZ11 und DZ12; does not include the law violation(s) listed in his/her federal central criminal registry; and does not include the violation of social norms)
Rating:
(0) If no report in his/her file indicates that he/she has ever broken rules
(1) If one report in his/her file indicates that he/she has broken rules at least once
(2) If two reports in his/her file indicate that he/she has broken rules at least twice
(3) If three or more reports in his/her file
However, we have referred to the detailed list of all our variables including rating procedures for each variable in the legend of the table as well. Please let us know if you consider another presentation of this information to be more useful.
Line 192 Please provide factor definitions in the figure itself instead of variable codes (with the legend under the figure).
We have made the necessary adjustments in figure 2.
Line 210 -223 and apart from the very true arguments already provided, in the field of forensic psychology / psychiatry there is (from personal professional experience) at least one additional relevance, primarily from the public prosecution perspective: accountability. Providing the scientifically sound decision support method that helps to come to (well) specified decisions to be made on individual cases (focusing on circumventing potential risks, for the patient / convict involved, the related professionals and society). That justifies a more ‘personal’ medical and juridical judgement and might consequently be a juridical very valid argument to differentiate in imposed penitentiary regimes.
Very much so! Thank you for providing your own, very valuable clinical insight. We have included your very valid point in our discussion (see line 229 ff).
Round 2
Reviewer 2 Report
Dear Authors,
Thank you very much for your efforts to revise youe manuscript. It was a pleasure to review and believe all my concerns are covered appropriately.
Looking forward to your study being publish, as to have the opportunity to refer to it.
