On the Black-Box Challenge for Fraud Detection Using Machine Learning (II): Nonlinear Analysis through Interpretable Autoencoders

Round 1

Reviewer 1 Report

Aiming to solve the problem of Credit Fraud Detection, this paper proposes a method in three steps: 1. to reduce the dimensionality by selecting the informative features; 2. to efficiently compress and encode data to isolate fraud transactions from non-fraudulent ones; 3. to propose, and eventually, evaluate, novel techniques to offer a comprehensive explanatory model in CFD.

 

The grammar and the structure of the sentences must be improved.

An example below, this long sentence is not very well readable nor grammatically accurate.

 

We introduce the Single Transaction-level Explanation (STE), which is a technique that identifies for each informative variable how it affects the final decision through small-scale fluctuations in the inputs and in the latent spaces.

 

Some parts are lengthy and can be more concise. e.g. Abstract

Also, the introduction seems to miss the link between the concepts that are exactly related to the problem of the paper. discussing many different things while the main contribution is not based on conventional methods.

 

The introduction section should indicate the challenges and contributions.

 

some important articles can be added to related work to highlight the role of the

 

The paper must include a framework overview to highlight the components in online and offline mode.

 

Some missing references:

About Embedding:

(1) SoulMate: Short-text author linking through Multi-aspect temporal-textual embedding

Najafipour et al.

 

About intrusion detection that is related to CFD:

(2) Improving network intrusion detection classifiers by non-payload-based exploit-independent obfuscations: An adversarial approach

Homoliak et al.

 

And finally, the role of other parameters must be noted. For example, the temporal correlation between the data entities can also highlight the latent cues for CFD. So would better be indicated in the literature.

 

(3) TEAGS: time-aware text embedding approach to generate subgraphs

Hosseini et al.

 

The paper seems to be more applicational, using the components like autoencoders can not represent a research work. For example, step 1 merely executes a few existing algorithms: CME, GB, SVC, LDA, LR that are not even state-of-the-art.

 

Some formulas are simple and some others belong to other works. You may remove those unnecessary and add the citations instead of them.

 

Figure 3 represents a few lines of code in deep learning, hence the scientific contribution is unclear.

 

Some steps are not clear, for example in Algorithm 4: Depending on the weights of each feature, we obtain its numerical position in the significance ranking.

Where the ranking is important, the methodology is not clear.

 

For the experiment, the structure should be improved. Add sections for the benchmark, baselines, and parameter settings, where you should defend that the baselines are current.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

Writing style is good. However the followings may be considered for betterment:

Formatting should be checked.

Grammar can be enhanced.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 3 Report

ABSTRACT

-Line 1: too generic, please rephrase more appropriately 

-Line 2: Algorithmic society -> needs to be changed, not a proper definition 

-Line 7: Many problem domains -> several problem domains

-Line 13: who are the supervisors? Please specify

-line 23: describe the local perturbation defined here

 

-too many general sentences (for example line 30, similar datasets, which ones? At least add a reference).

 

-line 31-32 do not only specify the improvement, but also the baseline from which this paper proposal improves from

 

-from line 33 to 35 rephrase -> too general and not well specified

 

-general comment: the abstract is too long, I think it would be good to shorten its length

 

INTRODUCTION

 -LINE 42: add immediately after the mention, the relative reference of the Alan Turing Institute report you are referrig to

-line 54: what are the checklists of risks factors? Please specify accordingly, and make examples of such lists

-lin 58-59 please rephrase and improve the English

-line 64: emerged in this space-> please rephrase

-line 85/86 what are the wrong algorithms mentioned here? Please rephrase and specify

-line 92 enhance more the interpretability aspect. Add references and say why it is so important.

 

RELATED WORK

-line 152/153 the fact that in the latent space the information is more easily detectable is not always the case. Please justify better this part and rewrite.

-line 182/183 the LIME model needs to be explained better. Please add more information concerning this

-I think a table would be needed to summarize the various features selection methods mentioned in this section 

-I think this section lacks more references to autoencoders applications where explainability is performed. For example: 

 

   -Arora, Sanjeev, et al. "A latent variable model approach to pmi-based word embeddings." Transactions of the Association for Computational Linguistics 4 (2016): 385-399.

   -Dimitri, Giovanna Maria, et al. "Unsupervised stratification in neuroimaging through deep latent embeddings." 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 2020.

  -Tshitoyan, Vahe, et al. "Unsupervised word embeddings capture latent knowledge from materials science literature." Nature 571.7763 (2019): 95-98.

  - Gasparetti, Fabio. "Discovering prerequisite relations from educational documents through word embeddings." Future Generation Computer Systems 127 (2022): 31-41.

  -Pancino, Niccolò, et al. "A Mixed Statistical and Machine Learning Approach for the Analysis of Multimodal Trail Making Test Data." Mathematics 9.24 (2021): 3159.

 

MATERIALS AND METHODS

-line 239: please specify better the notation. The usual notation implies to have N observations of L features. In other words, usually rows are the observations and columns of the matrix represent the features. If this conventional notation is not used, please specify accordingly and modify 

-Figure 1: please improve. Specify what are X’ and X’’ in the figure, and make the image more visibile

-line 249-250 make the notation consistent with the figure and with the rest of the text

-line 195 specify what resampling means 

-Figure 2: improve deeply the caption. The figure needs to be redone Moreover the caption is too long. Move some of the details to the text 

-Using the algorithmic section to describe step 1 I think is redundant, please modify 

-Section 3.5 would need an example, to make it more comprehensible. In particular it would be useful to have an estimate of how much the dimensionality reduces to have a better idea of the utility of the algorithm 

-line 382 please rephrase 

-line 384 -> participating TO the system 

-between line 392 and 412 there is a lack of notation explanation that makes the whole section hardly readable. Please amend, rewriting the section and adding relevant information considering the notation proposed 

-Our proposal 1 section looks a bit disconnected from the rest. Please rephrase and reorganize the section 

-Algorithm 3: the description of the process in this format makes it hardly readable. Please rephrase, but also make examples. The whole methodology in which the scores are found should be described. Leaving the whole description to the algorithmic section makes it hardly readable

-Same can be said for Algorithm 5 

 

EXPERIMENTS AND RESULTS 

-There is a lack of comparison with other methods. The method should be compared with alternatives and it should be therefore evaluated the improvement brought by the new procedure proposed 

-Figure 4: please add details on all of the features reported in this section. Moreover which is the colour legend? 

-Same can be said also for Table 2, and the rest of this section. There is a lack of explanation of the features, and this makes it hardly readable 

-Figure 7 and Figure 8, add more explanation on how were the figures obtained, Moreover multiple panels are shown, but it is not clear what do they represent. 

-line 648: misspelling (transations -> transactions )

-The whole experiment section needs reorganization. Is too long and hardly readable 

 

 

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

The authors have addressed most of my previous concerns.

I advise that they add a bulleted point list to signify the contributions of the article. Also consider my previous commend on the framework to elucidate it as requested in the previous revision.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 3 Report

I thank the authors for having addressed the comments, and the manuscript have definetely improved. However I still have some concerns, that I list hereby: 

 

Line 26: we developed of a novel -> remove of

Line 33: remove comma after e-commerce

Lin 166: say in which earlier section you have introduced this concept

Line 184: lineal -> linear

Line 200: which literature? In case specify (also according to THE literature)

Table 1: I think this helps the reader a lot. Probably I think it would be good to have references also added to Table 1

Lines 252-253: those weights really summarize-> remove really

Line 262: remove mainly        

Line 274: reformulate the sentence. Said at the beginning is not correct

Line 292 and following: please add references to where the equations 1 and 2 are references in the text

Line 298: fine tuning has been employed not only for autoencoders, please amend and explain better this sentence

Algorithm1: further explanation is needed to better let the reader understand algorithm 1

Line 312: simulation of what? Please rephrase

Line 324: please add a small sentence introducing why now we are talking about Kendall  in the paper

Equation 3: define what is the denominator of the formula

Figure 2: remove companion paper in the Figure. There is a mistake in the caption on the roman number 5 (which appears as a u instead of V). Also the acronyms used in the figure should be explained in the caption (MIFF, etc)

Line 360: I would add Features Selection extended in this line

Line 357 to line 373: please re-read and rephrase. There is a problem with the tenses used for the verbs in this area (not consistent with before)

Line 374: specify why you consider it as an Abstract space

Figure 3: not clear what is the layer box. Please specify also in the figure. The figure otherwise is too vague

Example 1: should not be in italic to avoid confusing it with the assumption paragraph

Line 442: how did you create the dataset

Line 445-447: not clear this part. Try to rephrase. In which sense the weights are considered

Line 459: how did you define the best autoencoder?

Algorithm 2 needs further explanation in the text

Line 473-474: too vague this sentence. Please rephrase

Definition 2 needs a relative example (as you did for definition 1)

Line 501-502. In which sense opening the door? Please explain more in details with an example

Describe Algorithm4 also with words in the text

Section 4.1 should stay in the materials and methods section, as it describes the dataset i.e. the materials used. Please reorganize accordingly

Line 526: what are f0,f1,f2,f3 and f4? Please specify. Also the features selection process is not completely clear (how did you obtain 23 from 485?)

For the three datasets, even if there are references to the original work, I would suggest to also add information here in the text on the meaning of the features (to give an example Cash-in, Cash-out and so on for Paysim). You could add a table in the supplementary or here, for the sake of completeness

Line 572: please argument on the sentence: “These results are extensible to real data sets”

Line 597: what are meagre values? Also please argument on the sentence added at lin 596 in which you mention the fact that the achieved results are statistically stable.

Figure 4: which is the threshold for determining relevance? Also please add a table in which acronyms of features used also in Figure 4 and later in the text are described.

Lines 599 and later: when using variables acronyms names there is the need of defining what they are first (so use a table or a section in the supplementary material). Same can be said for variables in Table 2 and Table 3

Line 634: Hypotesis ?? -> please amend

Figure 5: type_PAYMENT and type_TRANSFER have no variation to the position in the latents space? Only one dot is visible in the plot

Figure 6 needs more explanation in the Caption

Figure 8 needs furthere refinement. From a graphical point of view is stretched. Also for what concerns the non

Figure 9: Please add explanation of the features acronyms we see in the figure

 

Author Response

Please see the attachment

Author Response File: Author Response.pdf