Improved G-Optimal Designs for Small Exact Response Surface Scenarios: Fast and Efficient Generation via Particle Swarm Optimization
Round 1
Reviewer 1 Report
This manuscript comprises a comprehensive literature review of algorithms and results obtained for generating G-optimal exact designs. In addition to this, a well-known algorithm, the Particle Swarm Optimization (PSO), is proposed as a powerful optimization methodology for this kind of problems. Its well performance is shown through numerous examples. The provided algorithm is a valuable tool for experimental designers so that I recommend this paper for publishing after having solved some minor issues that may help to clarify some aspects:
1. Introduction
- p.2 line 48: Simulated annealing is one of the most popular meta-heuristics methods, as mentioned in the paper, and has been extensively used for experimental designers. Nevertheless, it was not used as a benchmark for comparisons in this paper. In your review I find that it was not previously used for generating G-optimal exact design, but it can seem a competitive algorithm for the PSO that you provide.
- p.2 lines 58-59: “the literature is lacking demonstration of PSOs efficacy and cost to generating exact optimal designs, …” In this recent paper:
, , & (2022). Particle swarm optimization for searching efficient experimental designs: A review. Wiley Interdisciplinary Reviews: Computational Statistics, 14(5), e1578. https://doi.org/10.1002/wics.1578
You can find an updated review about PSO applied to generate exact optimal designs which is missed in this work. Please, complete this gap.
2. G-optimal Design for Small Exact Response Surface Scenarios
- p. 2 line 75: “as is standard at the optimization step of response surface methodology, we well be working with the second-order linear model under standard assumptions” It is a strong statement which should be at least supported by the literature (justify, for example, using Montgomery, Douglas C. Design and analysis of experiments. John Wiley & sons, 2017)
- p. 4 line 96. “These are the only results from the theory of continuous optimal designs that also apply directly to exact optimal designs”. It is necessary to clarify this part. What results are only verified for continuous designs and what implications they have as consequence of assuming them for exact designs?
3. Literature review
- p.4 lines 112, 118. The reference to Borkowski (2003) is missed.
- p.5 line 138. The idea of being [10] and [22] as benchmarks of this work seems to be quite repetitive throughout manuscript. Please, eliminate redundant information.
- p.5 lines 161-165. Please, clarify this phrase.
- p.6 lines 193-194. This argument is vague. Please, argue it with numerical or literature references.
4. Particle Swarm Optimization for Generating Optimal Designs
- p.6 lines 223-224. Why the PSO used for D- and I-optimality exhibited a better searchability to find the globally optimal design in a single run and it cannot be showed for the provided one for G-optimality?
5. G-optimal Design Scenarios and Published Best-Known Exact Designs
- p.7 line 249. Highly efficient G-optimal…?
- What is the main novelty that you bring to the Algorithm 1 (compared to a “general” PSO)? It could help to clarify your contribution.
6. Results
- Two issues need to be reviewed/clarified for a fair comparison of results:
(1) All algorithms should start from the same initial set of random candidate designs.
(2) Precision given in the stopping rule (as well as the stopping rule itself) should be similar between algorithms.
- Figure 1, K=2. Why PSO generates slightly less efficient designs than GA in many cases for K=2?
- Figure 1, K=3. Affirm that PSO obtained efficiencies higher than 100 is a weak result given the variability presented in the distributions for K=3.
- p.9 line 300. What is parallel computing? It is necessary to clarify this since the computational cost could depend on this type of computation rather than the proposed methodology.
- p.9 lines 301-311 and Table 2. I have some doubts about the information reported by the number of function evaluation is relevant to fairly evaluate the computational cost. It is due to each algorithm last different computation time in running one evaluation. Please, justify.
- P. 12 lines 362-363. “These results imply that PSO has more difficulty scaling to higher dimension”. This limitation should be noted in the conclusions.
Author Response
Dear reviewer:
Thank you for the constructive review! We were able to address the majority of your comments and greatly improve the manuscript.
Best,
Steve
Author Response File: 
Author Response.pdf
Reviewer 2 Report
I find this paper an interesting review of G-optimal designs for the quadratic model commonly used in Response Surface Methodology (RSM). After a careful reading of the work let me point out some comments that I think will improve the paper and at the end I will mention some of my concerns about the work.
1.- Introduction presents very nicely optimal experimental design theory, not only exact designs. The two final paragraphs (lines 62-73) are some how redundant and I suggest that they are combined summarizing only once the remainder of the paper.
2.- In lines 50-51 an enumeration is produced using 1) and 2) in an inline format. I rather consider that either an itemize environment is used or “first” and “second” should be used in instead. This is something that happens several times along the paper. I will point it out every time but I would like to applaud the authors' clarity at other times, such as lines 89-92, when they present the parts of the problem they face.
3.- Response Surface Methodology is a wide used methodology in the industry and a general reference to the methodology could be very useful for non-mathematical readers. There are plenty of them, let me suggest for example NIST/SEMATECH e-Handbook of Statistical Methods, http://www.itl.nist.gov/div898/handbook/ but please find the more suitable for you.
4.- Equation (1) might be avoided, I suggest that last paragraph in page 2 is enough clear to understand that the space of all candidate designs is $\mathcal{X}^N$.
5.- I have already mentioned clarity in sections 2.1 and 2.2. Nevertheless, I miss a citation to support the statement made in the final paragraph of page 3, lines 94-96.
6.- Let me point out a notation issue, in Equations (6) and (7) 100 multiplies the efficiency of a design first by using \times and secondly without a symbol. In other places such as line 298 an * is used. The same symbol should be used always, I would prefer that * is avoided.
7.- In section 3 the review of exact optimal designs I wonder if a summary could be arranged in a way of a table with number of factors in rows and experiment sizes in columns. This is just a suggestion.
8.- Statement in line 131 could be supported by some citation?
9.- Avoid to begin a sentence with a reference link such as in line 132. Saleh and Pan [24] should be used instead.
10.- Line 144 has a typing error where it should read “clustering and point”.
11.- Enumeration environment as in my second comment should be used in lines 170-172. Also, in lines 205-213. Different notation is used along the paper, some times 1.) other times 1). If you consider a clear itemize structure is better for the understanding of the ideas use an itemize environment, otherwise make it a text enumeration.
12.- Sentence in lines 221-226 could be rearranged in a more suitable structure, it is not incorrect, but it is difficult to follow in a first approach.
13.- In line 252 it should say $n_{run}=210$ times”
14.- Line 261 should begin with Hernandez and Nachtsheim [11] as in line 288.
15.- It is not clear to me the need of estimating with a confidence interval via Poisson statistics the cost comparison for n=200. Why not running the PSO n=200 instead of 140. Where are the difficulties? The effort of the authors to compare their work with others is very remarkable, and a reason for this estimation should be stated clearly.
16.- Titles above figures 1 and 2 should be removed and captions should provide all the information describing the images.
17.- Table 3 needs a more careful display. In lines 324 they mention that efficiencies of best designs are displayed but then only CEXCH efficiencies are shown. It depends on the source [22] or [11] of the G-optimal desings, but then in the table references are given by the first surname and year. G-releff is then presented compared to previously published designs. To gain clarity Table [1] and Table [3] should have the same information and equivalently arranged.
18.- Again enumeration environment in lines 337-340 should be used.
19.- Conclusions regarding the improvement of the G-optimal designs obtained via PSO in comparison to designs published in the literature could be arranged in a more clear way, for example a table with the number of factors in rows and number of points in columns stating relative efficiency of the PSO design with respect to the best design known in literature. This could easily be a nice picture of the power of PSO algorithm and its implementation in Julia.
20.- Finally let me express my concern about the supplementary material. This material is not available at the reviewing stage, and I consider that it is very important that the provided information is relevant and in a suitable format.
This is an opportunity to provide a catalog of exact optimal designs for the quadratic model in RSM. Authors mention a csv file with the G-optimal designs obtained, this is a very useful tool for the application of the results and should be extremely simple and intuitive for non-mathematics/statistics experts.
R code to verify the G-scores, and therefore reproduce the results is extremely useful and conveys confidence in the results and conclusions.
It would be extremely useful to have in some form the Julia code used to obtain the designs shared in some way, just let me point it out.
Author Response
Dear reviewer:
Thank you for the constructive review! We were able to address the majority of your comments and greatly improve the manuscript.
Best,
Steve
Author Response File: Author Response.pdf
Reviewer 3 Report
The proposed manuscript is aimed at the G-optimal designs issues with Particle Swarm Optimization use. The problem is worth investigating and the manuscript is well written. The structure of the paper is properly designed. The presented review on recent developments in the investigated area significantly improves the paper. The proposed approach is properly presented and the comparison to the widely known algorithms indicates the contribution of this paper.
The paper is well written. The one suggestion is to better highlight the contributions of this paper in the introduction section.
Author Response
Dear reviewer:
Thank you for the constructive review! We were able to address the majority of your comments and greatly improve the manuscript.
Best,
Steve
Author Response File: Author Response.pdf
