Progress in practical hyperparameter tuning is often hampered by the fact that there are no standardized benchmark problems. To alleviate this problem we maintain HPOBench, a library which provides a unified interface machine learning tasks.
Currently we are working on HPOBench, a new version focusing on reproducible containerized benchmarks.
Previous versions, can be found here (HPOlib1) and here (HPOlib1.5).
Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters [pdf] [bib] [poster]
NeurIPS workshop on Bayesian Optimization in Theory and Practice
This includes results for SMAC, spearmint and TPE on the benchmarks we provided in HPOlib1