Conference | seminar
CCLS Seminar with Anna Kononova
- Date
- Monday 13 February 2023
- Time
- Address
- Snellius
Niels Bohrweg 1
2333 CA Leiden - Room
- 312
Benchmarking the benchmark: are heuristic optimisation algorithms designed against the right data?
Benchmarking plays a critical role in the design and development of optimization algorithms. Benchmarks provide information on the performance of an optimisation algorithm and how it compares with others in various situations. For example, one algorithm may excel at optimising separable functions while another performs well on unimodal, highly conditioned functions. The way in which benchmark suites are set up thus influences the set of algorithms recommended to practitioners and biases the goals of algorithm designers. One of these suites is the black-box optimization benchmarking (BBOB) suite of 24 single-objective noiseless functions, which has been a standard in the field for over a decade. Within this problem suite, different instances of a single problem can be created, which is beneficial for testing the stability and invariance of algorithms under transformations.
We investigate closely the BBOB instance creation protocol and show that the distribution of features across BBOB instances is highly diverse. We demonstrate that statistically significant differences in performances across instances can be observed. We argue that, while the transformations applied in BBOB instances do indeed seem to preserve the high-level properties of the functions, their difference in practice should not be overlooked, particularly when treating the problems as box-constrained instead of unconstrained. Finally, we investigate how a particular algorithmic deficiency called structural bias manifests itself on instances of benchmark functions.
This seminar will be followed by drinks and nibbles in the Foobar.