MFPCDR: A Meta-Learning-Based Model for Federated Personalized Cross-Domain Recommendation
Round 1
Reviewer 1 Report
Title: MFPCDR：A meta-learning-based model for federated personalized cross-domain recommendation
Manuscript Number: applsci-2256108
Comment for Authors
The core objective of this work is to solve the cold start problem in recommendation systems and the privacy preservation challenge in cross-domain recommendation systems. To this end, we propose a novel meta-learning-based framework for federated personalized cross-domain recommendation systems, namely MFPCDR, in which a federated learning approach is used to train the model on the local embedding module, user behaviour data is stored on the client machine to participate in the training, upon completion of training, data and settings are transferred to a central server for update, without the user behaviour data leaving the client machine, thus protecting the user's privacy data. After implementing the local embedding module, we may gain user and project embedding as well as a customized conversion of user embedding for cold start users, after which we can obtain transfer of cold start users through the meta-recommendation module. After the carefully evaluations of this article I concluded that this work cannot be reached at the acceptable level before fulfil the requirements as follows:
This paper cannot be accepted by the following reasons.
Ø The originality of the proposed novel meta-learning-based framework is weak, because it is just a complex model consisting of the existing ideas.
Ø Although the framework is a bit interesting and the results are promising, the manuscript is not clear enough to allow other researchers to understand and reproduce the authors' method. In my opinion, the details of new modification and experiments are needed for more enhancement in the text, which is precisely the core of the document and needs a major improvement.
Ø Statistics results analysis also missing in the text.
Ø Results should be compared on the fresh published works.
Ø Standard suites comparisons should be included in this work.
Ø Only cold start problem has been considered in this work, as per my opinion this work is not sufficient so some more problems and results should be added in it.
Author Response
We would like to thank you for your careful reading, helpful comments, and constructive suggestions, which has significantly improved the presentation of our manuscript.
Point 1: The originality of the proposed novel meta-learning-based framework is weak, because it is just a complex model consisting of the existing ideas.
Response 1: Thank you for your suggestion. The innovation of our method lies in the use of federated learning to solve the problem of privacy leakage in traditional recommendation systems. User data is stored on the client machine for training and never leaves the local. We use the meta-network in the meta-recommendation module to learn the user's personalized transferable embedding, enabling the transfer of users' personalized preferences across domains. We use the attention mechanism in the local embedding module, and assign larger coefficients to those items that contribute a lot to personalized transferable preferences, which is more helpful in mining transferable features in the source domain that are conducive to knowledge transfer, and more Good job getting item embedding.
Point 2: Although the framework is a bit interesting and the results are promising, the manuscript is not clear enough to allow other researchers to understand and reproduce the authors' method. In my opinion, the details of new modification and experiments are needed for more enhancement in the text, which is precisely the core of the document and needs a major improvement.
Response 2: Thank you for your suggestion. In order to make our manuscript clearer, we have revised our manuscript. In 3.1 Problem Formulation, we gave a simple example to make the problem description easier to understand, as follows:
“Let's take a simple example of a music datasets as the source domain and a book datasets as the target domain. Both the music datasets and the book datasets have a user set, an item set, and a rating matrix. An overlapping user is a user who is present in both the music datasets and the book datasets, but only has a rating on the music datasets. We need to use the overlapping users' ratings on the music data to learn the preferences of the overlapping users, and then we can use the overlapping users' preferences to make recommendations to them in the book data set, thus solving the cold start problem”.
We added instructions to filter users with less than 10 interactions and items with less than 30 interactions in 4.1.Datasets to make the manuscript more understandable, as follows:
“People with fewer than 10 interactions were deleted because they provided too few reviews or feedback to provide enough information for analysis. And items with fewer than 30 interactions were removed because the item's sales or market performance is insufficient to support the poor analysis, deeming or irrelevant. These filtering criteria ensure that only high-quality data is used for analysis, improving the credibility of the data and the accuracy of the results”.
We have modified the chapter 4.3 Experimental Setup and added many experimental details. The newly rewritten part is as follows:
“We used pytorch to implement our proposed MFPCDR model, preprocessing all user and project documents in the datasets to remove deactivated words and words with high document frequency. We used grid search to adjust the hyperparameters of all methods, Adam to optimize the model, and set the learning rate to {0.001,0.005,0.01,0.05,0.1,0.5}, the embedding dimension range for users and items to {8,16,32,64}, and the batch size test range to {64,128,256,512}. The activation function was set to sigmoid and the performance of all methods was reported after 5 runs. We found that the MFPCDR model works well when the Adam learning rate is set to 0.01, the user and item embeddings are set to 32, and the batch size is set to 128.
For assessment metrics, we considered RMSE, MSE, MAE and Top-K. MSE is the abbreviation of mean square error, which is the average value of the square of the difference between the predicted value and the actual value. It is a measure of the difference between the predicted value and the actual value. The smaller the MSE, the smaller the difference between the predicted result and the real result, indicating that the model's prediction effect is better. RMSE is less sensitive to outliers than MSE because it operates on the square root. MAE maximises user satisfaction but does not minimise prediction error. top-K measures how many items in a given recommendation the recommendation algorithm matches the user's true preferences, often with a long tail of users problem. The MSE is sensitive to outliers and computationally efficient, and can be effectively applied to our model”.
Point 3: Statistics results analysis also missing in the text.
Response 3: Thank you for your suggestion. Our analysis of statistical results is as follows. First, we added the descriptive statistics section in Section 4.4, as follows:
“It can be seen from the data that MFPCDR performs better than other baseline models in three cross-domain recommendation scenarios. The mean MSE of MFPCDR in the first scenario is 1.086 and the median is 1.081. The mean MSE of MFPCDR in the second scenario is 1.126 and the median is 1.128. The mean MSE of MFPCDR in the third scenario is 0.8896 and the median is 0.895”.
We also conducted exploratory data analysis. Through data visualization, we drew a line graph, as shown in Figure 4 in the manuscript, and analyzed the relationship and outliers between the data and explored the causal relationship, as follows:
“In Scenario 1 when the proportion of overlapping users reaches about 20%, the fold starts to level off gradually. This phenomenon is also observed in Scenario 2 and Scenario 3 when the proportion of overlapping users reaches about 10%. We find that it is at the beginning of the model, as the proportion of overlapping users increases, the model is more likely to capture the personalized transferable preferences of users, which helps our model solve the user cold start problem. And as the proportion of overlapping to users continues to increase, the discounting starts to level off, which we find is due to several reasons, firstly, due to the overlap between users who may be interested in the same products, secondly, when the proportion of overlapping users increases, more users in the MFPCDR model will generate similar behaviors, and finally, when the proportion of overlapping users is high, the MFPCDR model is susceptible to the influence of popular recommendations”.
We further explain the effects and application scenarios of the MFPCDR model, and the added parts are as follows:
“We find that the MFPCDR model works better than other models because it uses an attention mechanism on the item embedding by performing a weighted summation, which reduces the weight given to learning the user's preferences for useless items and thus improves the model's effectiveness. ”
“Our models are applicable to different regions or domains, but the effectiveness of the recommendations depends on the quality of the data, feature selection and user feedback. The data should adequately reflect the interests and behaviour of users. Different domains and regions may have different characteristics. For example, in a stock exchange recommendation system, trading history and stock fundamentals indicators may be important features, while in a location-based recommendation system, user location and attached shops may be important features, and applying MFPCDR in these scenarios requires appropriate feature selection”.
“Because our meta-recommendation module needs to learn from interacting users, our model is not suitable for extreme cold start situations. Also, because a federated learning approach is used to protect user privacy, this inevitably introduces noise that reduces the effectiveness of the model training”.
Point 4: Results should be compared on the fresh published works.
Response 4: Thank you for your valuable suggestions. The ANR model is a neural network-based recommendation system model. The main idea is to use the attention mechanism to realize feature interaction and weight learning in the recommendation process. In the recommendation stage, the ANR model can calculate the similarity between the user's representation and the item's representation, and use the similarity as the score of the recommendation result. Finally, the ANR model sorts the ratings and recommends the top K items with the highest ratings to the user.
Compared with other recommendation system models, the ANR model has better performance and interpretability, which can help the recommendation system better understand user behavior and preferences, thereby improving recommendation accuracy and user satisfaction. So we choose the ANR model as our comparison model, and CMF and DFM are classic models in the field of traditional recommendation systems. Many previous works choose to compare with them. EMCDR and CDLFM are classic works in the field of cross-domain recommendation systems. This is the reason we choose these models as baseline models.
Because we adopt the method of federated learning, noise will inevitably be introduced in the process of data training, so the model needs to make choices in terms of accuracy and privacy protection.
Point 5: Standard suites comparisons should be included in this work.
Response 5: Thanks to your suggestion, we compare our model with five baseline models, including traditional recommendation system model and federated recommendation system model. The datasets uses the Amazon datasets. The Amazon datasets is a widely used recommendation system datasets consisting of product purchase and review data on the Amazon e-commerce platform. It contains purchase history, reviews, and metadata information for millions of items by many users. This datasets can be used for training and evaluation of recommendation system algorithms. We selected three cold-start recommendation scenarios, compared our proposed MFPCDR with five baseline models, and achieved good results.
Point 6: Only cold start problem has been considered in this work, as per my opinion this work is not sufficient so some more problems and results should be added in it.
Response 6: Thank you for your suggestion. Our model is designed for the cold start of the recommendation system. The federated learning method is used to protect privacy. The model needs to make a trade-off between accuracy and robustness, so the performance of MFPCDR may be better in the cold start scenario. good. However, we explored the influence of the number of overlapping users on the model, and wrote it in Chapter 4.5, as follows:
“To investigate the influence of the number of overlapping users, further tests were done at five percentage levels of 5%, 10%, 20%, 50%, and 60% in three cross-domain recommendation situations. Figure 4 depicts the outcomes Observations indicate that MSE lowers as the fraction of overlapping users increases, which means that the larger the proportion of overlapping users, the more user preferences the model can learn and the better the model accuracy. The setting of the proportion of overlapping users directly affects the number of users from which the system learns similar preferences for cross-domain recommendations. Clearly more overlapping users allows the model to better understand the preferences of the cross-domain recommendation domain, thus improving the accuracy of the recommendations. Intuitively, the curve of our proposed MFPCDR model is relatively flat”.
We would like to thank the referee again for taking the time to review our manuscript.
Reviewer 2 Report
Manuscript id - applsci-2256108
In this work the authors have proposed a unique meta-learning-based federated personalized cross-domain recommendation model that discovers the personalized preferences of cold-start users via a server-side meta-recommendation module.
Comment -1
In the above description, a minor change is required -------
………. recommendation model that discovers the personalized preferences for (the word ‘for’ is more appropriate and conveys the intended meaning rather than ‘of’) cold-start users -- (as authors given in line 300)
The above change is mandatorily required as it affects intended technical meaning
Comment -2
In what way, the proposed method is unique when compared to existing works? -- The authors can explicitly describe the unique features
Comment -4
Line 505 – How the method is novel? How do authors claim for it?
Otherwise, the word may be removed
Comment -5
What are the limitations of the proposed method? What are the practical challenges during implementation of federated design?
Comment -6
Citations references are missing
Author Response
Respose to the Review Comments
We would like to thank you for your careful reading, helpful comments, and constructive suggestions, which has significantly improved the presentation of our manuscript.
We have described our changes as follows.
Point 1: In the above description, a minor change is required -------
………. recommendation model that discovers the personalized preferences for (the word ‘for’ is more appropriate and conveys the intended meaning rather than ‘of’) cold-start users -- (as authors given in line 300).
The above change is mandatorily required as it affects intended technical meaning.
Response 1: We have changed of to for in line 18. The whole sentence reads: Therefore, we offer a unique meta-learning-based federated personalized cross-domain recommendation model that discovers the personalized preferences for cold-start users via a server-side meta-recommendation module.
Point 2: In what way, the proposed method is unique when compared to existing works? -- The authors can explicitly describe the unique features
Response 2: We briefly describe the characteristics of our proposed model in lines 23 to 26, with the following changes: “Our model can effectively protect user privacy while solving the user cold start problem compared to traditional recommendation system models, while we use an attention mechanism in the local embedding module to mine the source domain for transferable features that contribute to knowledge transfer”.
Point 3: Line 505 – How the method is novel? How do authors claim for it?
Otherwise, the word may be removed
Response 3: We have removed this paragraph: “which indicates that MFPCDR is less impacted by the fraction of overlapping users and can better tackle the cold start problem while safeguarding user privacy”.
Point 4: What are the limitations of the proposed method? What are the practical challenges during implementation of federated design?
Response 4: We present the limitations of our proposed approach and the challenges of implementing an l-federal design in lines 495-499, as follows: “Because our meta-recommendation module needs to learn from interacting users, our model is not suitable for extreme cold start situations. Also, because a federated learning approach is used to protect user privacy, this inevitably introduces noise that reduces the effectiveness of the model training”.
Point 5: Citations references are missing
Response 5: We have added to the cited references.
We would like to thank the reviewer again for taking the time to review our manuscript.
Reviewer 3 Report
1. The Introduction part should be strongly strengthened. I suggest that the major research questions of is paper should be emphasized in this part. In addition, I suggest that the author should describe the contribution of the paper from the perspective of practice.
2. A real data set from Amazon E-commerce has been used to carry out the experiments. Why is the Amazon data set selected for validation in this paper, and are other E-commerce industry datasets also applicable to the proposed system in this paper?
3. In each domain, we deleted entries that had no review language and then filtered out people with fewer than 10 interactions and goods with fewer than 30 interactions.
The above sentence should be justified. Why people with less than 10 interactions have been removed? Why goods with less than 30 interactions have been removed?
4. Evaluating the proposed approach by using one performance metric (MSE) is not sufficient to validate the superiority of the proposed approach. More relevant performance metrics should be included. This includes P, R, and/or F-measures which are widely used by most of the previous works and their definitions can be found in many textbooks. It is a common practice to use these metrics when evaluating any algorithm in the area of recommendation systems (Please see the following works):
https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9026823
https://www.mdpi.com/2071-1050/15/4/2947
https://ieeexplore.ieee.org/document/9224132
https://ieeexplore.ieee.org/document/9514087
5. The list of references mentioned in point 4 should be included in the paper as they are highly relevant to the topic of the paper. These are some of the recent highly relevant works.
6. Please include a discussion on the selection of the evaluation metrics. If the authors believe that the evaluation metrics are the only ones used by the previous works or the only possible metrics, this needs to be stated.
7. Please provide a discussion of how the proposed method is not positioned with respect to the existing methods (not with respect to its performance, but with respect to how unique/novelty the new method is, or whether it is similar to the most recent relevant approaches).
8. Give a brief discussion about the process when it is used in different areas or domains like stock exchange, location-based recommendations, etc. How can it be more effective?
9. The motivation is weaker. Instead of highlighting the general problems of the recommender system in the introduction section, some focus should be on the problems/weaknesses of the existing systems from the literature.
10. The experiments don't discuss the effectiveness of the proposed approach regarding the cold start of the new user.
11. The novelty of the manuscript should be improved.
12. The problem statement of the paper needs further elaboration. It is hard to understand the issue addressed in the paper. Providing a real-world example to depict the problem would be a good way to clarify the problem.
13. It is not clear what are the differences between the proposed Local Embedding Module from the work proposed by Yan et al, and [24]. The author mentioned that their work has been inspired by the work of (Yan et al, and [24]). However, it is not yet clear how their work is different from these two works! Are there any modifications or changes made to these two techniques?
14. Similarly, it is not clear what the novelty of the Meta Recommendation Module presented in subsection 3.3. How this part of your proposed solution differs from the attention technique proposed in [27]?
15. The authors need to explain the reason why their approach outperforms the other approaches. The discussion of the result should be in-depth to elaborate on the uniqueness of the proposed approach.
Here are more technical and grammatical concerns/comments that should be addressed in the revised version of the manuscript.
1. The two sentences between lines 10 – 15 are almost identical. You should either remove one of them or rephrase it.
2. The term Meta-learning needs some elaboration in the abstract. It is unclear what it means in the context of the paper.
3. The reference in line 42 is missing Error! Reference source not found.
4. Figure 1 has not been mentioned in the text. You must explain Figure 1 in the text.
5. The Workflow diagram for cold start user cross-domain recommendation presented in Figure 1 should be explained in the text.
6. The first paragraph in Subsection 2.1. should be supported with some relevant references. Most of the text in the paragraph requires supporting evidence.
7. The five classifications of meta-learning mentioned in lines (85- 87) need supporting references. Who proposed these techniques? Where are the references of these methods?
8. Meta-learning has been shown to be effective in image classification, recommendation systems, and small sample learning. Is there any evidence that supports this allegation?
9. The references mentioned between lines 92 – 112 are not properly cited. It seems that there are some bugs in the software that has been used to cite the references.
10. The reference mentioned in line 127 is not properly cited. It seems that there are some bugs in the software that has been used to cite the reference.
11. The references mentioned between lines 140 – 149 are not properly cited. It seems that there are some bugs in the software that has been used to cite the references.
12. The references mentioned between lines 157 – 180 are not properly cited. It seems that there are some bugs in the software that has been used to cite the references.
13. The reference mentioned in line 208 is not properly cited. It seems that there are some bugs in the software that has been used to cite the reference.
14. The reference mentioned in line 259 is not properly cited. It seems that there are some bugs in the software that has been used to cite the reference.
15. The reference mentioned in line 291 is not properly cited. It seems that there are some bugs in the software that has been used to cite the reference.
16. The reference mentioned in line 232 is not properly cited. It seems that there are some bugs in the software that has been used to cite the reference.
17. The output of Algorithm 1 should be included beside the input. It is uncommon to list the input for the algorithm without its output!
18. Algorithm 1 should be divided into three different algorithms. The first algorithm describes the detailed steps of the Local Embedding Module. Algorithm 2 elaborates on the Meta Recommendation Module, and Algorithm 3 outlines the Prediction Module.
19. Further details on each module should be given in the algorithms
20. Check Section 4. Heading! It seems the head of the section is incomplete! (See line 349
21. All references mentioned in 4.2. Baseline Methods have bugs! Please fix them.
22. The sentence in line 378 should be rephrased. It is not appropriate to be written in this way!
23. The references mentioned in lines 379, 382, 386, 393, and 397 are not properly cited. It seems that there are some bugs in the software that has been used to cite the reference.
24. What is MSE stands for? The full definition of the MSE must e given when the first time appears in the paper.
Author Response
Dear reviewer
Please see the attachment.
Author Response File: 
Author Response.doc
Round 2
Reviewer 3 Report
Thank you for incorporating my comments/suggestions in the revised version of the manuscript. However, I noticed that some of the comments have been ignored. Please improve the quality of the manuscript by addressing the following comments/suggestions.
1. The Introduction part should be strongly strengthened. I suggest that the major research questions of the paper should be emphasized in this part.
2. A real data set from Amazon E-commerce has been used to carry out the experiments. Why is the Amazon data set selected for validation in this paper, and are other E-commerce industry datasets also applicable to the proposed system in this paper?
3. Evaluating the proposed approach by using one performance metric (MSE) is not sufficient to validate the superiority of the proposed approach. More relevant performance metrics should be included. This includes P, R, and/or F-measures which are widely used by most of the previous works and their definitions can be found in many textbooks. It is a common practice to use these metrics when evaluating any algorithm in the area of recommendation systems (Please see the following works):
https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9026823
https://www.mdpi.com/2071-1050/15/4/2947
https://ieeexplore.ieee.org/document/9224132
https://ieeexplore.ieee.org/document/9514087
4. Please include a discussion on the selection of the evaluation metrics. If the authors believe that the evaluation metrics are the only ones used by the previous works or the only possible metrics, this needs to be stated.
5. Please provide a discussion of how the proposed method is not positioned with respect to the existing methods (not with respect to its performance, but with respect to how unique/novelty the new method is, or whether it is similar to the most recent relevant approaches).
6. The novelty of the manuscript should be improved.
12. The problem statement of the paper needs further elaboration. It is hard to understand the issue addressed in the paper. Providing a real-world example to depict the problem would be a good way to clarify the problem.
7. It is not clear what are the differences between the proposed Local Embedding Module from the work proposed by Yan et al, and [24]. The author mentioned that their work has been inspired by the work of (Yan et al, and [24]). However, it is not yet clear how their work is different from these two works! Are there any modifications or changes made to these two techniques?
8. Similarly, it is not clear what the novelty of the Meta Recommendation Module presented in subsection 3.3. How this part of your proposed solution differs from the attention technique proposed in [27]?
Before I can make a decision, you must incorporate these suggestions into your manuscript!
Author Response
Dear review
First of all, we would like to thank you for your valuable comments and suggestions on our research work. We value your review comments and are deeply grateful for them.
In the review comments, we have taken note of your criticisms and suggestions on our research findings. We have read your comments carefully and have built on them for further discussion and improvement. We have made changes in response to your suggestions and have tried our best to meet the requirements you raised. We have included the answers to your questions in the attached document.
At the same time, we are aware of the shortcomings in our research, which are well reflected in your review comments. We will carefully consider your comments and improve them in our future research in order to better improve the quality of our research.
Thank you again for your attention and support to our research work, your review comments are of great importance to our research. We hope that we can receive more guidance and support from you in our future research, so that we can jointly promote the development of science.
We sincerely wish you all the best!
Author Response File: Author Response.docx
Round 3
Reviewer 3 Report
I think the paper has been significantly improved and could be accepted for publication.
