A Qualitative and Comparative Performance Assessment of Logically Centralized SDN Controllers via Mininet Emulator
, I-Shyan Hwang 1,*, Muhammad Saibtain Raza 1 and Mohammad Syuhaimi Ab-Rahman 2
Round 1
Reviewer 1 Report
Comments and Suggestions for AuthorsThe authors present a complete review of papers related to SDN. The structure of the paper needs some small changes. English sounds good. The references are many, some more than 2023 are missing, but in general terms they are adequate. The simulation part should be better explained so that the experiments are reproducible.
Some improvements to the paper are below.
1. Abstract. Line 11. The industry is already moving towards SD-WAN and intent-based networking. The 'expected' comment could be changed.
2. Abstract. Line 26. OpendayLight is the best in half of the metrics, not in all of them as it seems to imply.
3. REFERENCE 1. It could be changed for something more recent.
4. Reference 3. Likewise, you should look for a more current one.
5. Figure 1. It doesn't look good when printed, luckily there is no problem with the pdf. Change the colors in case it looks better.
6. Section 2. Line 169. Add a reference to 'hcprobe' or delete the phrase.
7. Section 2. Lines 153 to 223. There is no full stop, which makes reading more complicated. This part of the section should be divided into several paragraphs.
8. Section 2. Line 217. Add a reference to 'cbench' or delete the phrase.
9. Section 2. Line 235. Sometimes it is said paper, sometimes article.
10.Section 2. Line 238. Table 2 is referenced before table 1.
11. Section 3.3. Line 314. There is a change in the argument. Add a phrase to connect with the above.
12. Section 3.3. Lines 314-373. Divide into several paragraphs.
13. Section 4. Line 386. Do the authors know of any telecommunications operators or parts of the industry that have used these drivers?
14. Section 4.2 Why have these scenarios been used and not others?
15. Section 4.3. There is a lack of basic information to know if the metrics used are reasonable or not. For example, it is not known the number of sources/sinks used, bandwidth of the links, type of traffic, whether there is fixed background traffic, ...
16. Section 5. Peaks are found in all metrics. It doesn't make much sense to view metrics for all packages over time. Mean values of the metric with their confidence interval must be included. Furthermore, the number of experiments must be high, at least 30 simulations with different seeds.
17. Section 5. Figure 7. Why are there so many losses? Why doesn't Ryu have them? Do you start counting metrics before the switches are fully configured? So many losses in wired links are not normal.
18. Section 6. It is not necessary to go figure by figure giving data. Comments should be more qualitative than quantitative.
19. Section 6.1 and 6.2. The simulations do not refer to these two subsections, why do they go here?
20. Line 637. Errata in the section number.
21. References. There are already some articles that could be included that take stock of some drivers, for example, the following two:
RAM, Anil; DUTTA, Manash Pratim; CHAKRABORTY, Swarnendu Kumar. A Flow-Based Performance Evaluation on RYU SDN Controller. Journal of The Institution of Engineers (India): Series B, 2024, p. 1-13.
NAIM, Naimullah, et al. POX and RYU Controller Performance Analysis on Software Defined Network. EAI Endorsed Transactions on Internet of Things, 2023, vol. 9, no 3.
Author Response
Please see the attachment.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Comments and Suggestions for AuthorsThe presented paper is devoted to the analysis of available SDN controllers that use the OpenFlow protocol. The introduction is an important part of the article because it contains the basic classification of the architecture of the control layer in SDN. An extensive state-of-the-art was also performed, with references to individual literature items many times later in the article. The paper was written in a quite readable way, maintaining the correctness of the language and the order in which the theses and results of the analyzes and simulations were presented. The reviewer believes that the article can be published after introducing a few additional descriptions and explanations:
1. The purpose and motivation of the analyzes and simulations performed must be clearly distinguished and justified.
2. The diagram in Figure 1 must be inserted in the object-oriented version (increased readability is needed), paying particular attention to the correct font proportions.
3. Figure 1: Is it intentional that there is a lack of examples for "Machine Learning Based Optimization"? The text refers to reference [11].
4. Lines 251-252: The sentence should be rebuilt to make it logical.
5. Line 282: Should be SD-WAN. The shortcut should be expanded.
6. Generally, it is advisable to create a collective list of abbreviations with their expansions.
7. Lines 374-379: The text should be removed because it is a repetition of what has appeared many times before.
8. Page 11: The table should be numbered by 3. Is this table necessary? This is a selective repetition of the data from Table 1. The new component is only "Application Area".
9. Subsection 4.3: Many key network model configuration parameters are missing: virtualization platform for Ubuntu, switch parameters, switch field speed and interfaces speeds, distances between hosts and switches, and so on. It is best to summarize it in a table.
10. The Section 5 beginning: Does this mean that during the simulation only the controllers were swapped, or were there several controllers connected together that co-worked on the east-west API? How were these controllers organized in the MiniNet application?
11. Was the network loaded with additional traffic while performing the tests?
12. Figures 3-8: How do these results apply to a network whose physical parameters are unknown? The measures used are unquestionably physical.
13. Page 16: A table should be marked with the number 4 if the previous table remains, but in a reduced version or with other more complete data.
Comments on the Quality of English LanguageThe text of the article should be reviewed for the so-called typos.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Reviewer 3 Report
Comments and Suggestions for AuthorsIn this paper, the authors have performed a qualitative and quantitative investigation of SDN controllers. Further, the authors also discuss an emulation tool for performance assessment. The paper provides a good overview of the SDN controllers and has also described the performance metrics in detail. I would recommend the paper for publication. Following are my comments to improve the quality of the manuscript.
The introduction effectively provides a solid foundation for understanding the significance of SDN. The authors can discuss the challenges that organizations might face during the transition from traditional networks to SDN-based architectures in terms of cost, complexity, and interoperability.
Also, the authors may point out how SDN may differ from traditional networks in terms of scalability, agility, and performance.
In the related work section, the authors can attempt to delineate that section into multiple sub-sections for better readability.
This section also provides a good comparison between different taxonomies and classification schemes in the earlier literature for SDN control architecture. Please ensure that the comparison is succinct and also only focuses on highlighting the unique contribution of the studies.
Also, highlight the limitations of the previous studies on SDN controller evaluation. This would help emphasize the significance of your research.
In the “controller classification and design choices” section, the authors can elaborate on how the choice of programming languages and API influences controller performance, and interoperability with different network devices.
Similarly, the authors can also provide any case study to illustrate how platform compatibility and modularity contribute to controller flexibility and functionality.
Also, in this section, the authors can briefly explain the limitations and capabilities of Mininet. I am guessing the authors have done this in the later section. However, they can provide a brief overview of the limitations and advantages in this section.
The authors can highlight how your study builds upon previous research and addresses the gaps by examining controller performance across different network scenarios and performance metrics.
The authors have considered five controllers for evaluation. Consider providing a brief rationale for selecting these controllers. What are the reasons for selecting these controllers specifically?
Also, the authors may need to explain why custom topologies were chosen over standard topologies and how they better represent real-world networks.
The same goes for performance metrics. Why these metrics were chosen for controller evaluation?
Are the findings from the performance analysis limited to the specific experimental setup and conditions? The controller performance may vary based on network size, topology, traffic patterns, and other factors. How can we generalize the findings? Is that a limitation of the study?
Discuss the relevance of each performance metric to real-world deployments.
Similarly, the authors can further elaborate on how the performance of the controller in a given metric impacts their performance in the real world.
If possible, a suggestion would be for the authors to show any statistical analysis or significance testing to show the differences observed between the controller performances.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
Comments and Suggestions for AuthorsThe authors have made the proposed changes correctly.
Reviewer 2 Report
Comments and Suggestions for AuthorsAlmost all the recommended corrections indicated in the previous version of the review were taken into account. Thank you for meticulously correcting the paper. I have no other additional comments.
