On Achilles Heel of some optical network designs and performance comparisons

This non-conventional paper represents the first attempt to uncover a possible vulnerability in some proposals for optical network designs and performance comparisons. While optical network designs and planning lie at the heart of achieving fiber capacity efficiency and/or operational efficiency, its combinatorial nature makes it computationally hard to reach optimal solutions for realistic scenarios. As a consequence, the well-established way that have been taken for granted by not-so-small number of research papers is that an optimization model based on mixed integer linear programming (MILP) formulation is first proposed and then due to the intractability of such combinatorial optimization model, a heuristic algorithm is offered as an approximation. The solution-quality comparison between the MILP and heuristic is then carried out on small-scale instances including topologies and traffic tests to verify the efficacy of the proposed heuristic. Next such allegedly verified heuristic are used for optical network designs of realistic scenarios. This approach may nevertheless leave a critical vulnerability as there is no guarantee that one performs well in small tests will generalize adequately for large-scale cases, a common pitfall widely referred as the peril of extrapolation and/or overfitting. Besides, it is not uncommon that in some research works, for benchmarking purpose, the comparison between a new design proposal whose performance is obtained from on one heuristic and a reference design based on another heuristic is carried out. As the consequence of lacking solution quality check, such performance comparison relied merely on heuristic solutions may be equally vulnerable, resulting to possibly unreliable conclusions. In this work, we pinpoint those issues and provide a realistic case study to highlight and demonstrate the impact of such vulnerabilities.