The loss landscape of an HPO problem is typically unknown (e.g., we need to solve a black-box function) and expensive to evaluate. Bayesian Optimization (BO) is designed as a global optimization strategy for expensive black-box functions. BO first estimates the shape of the target loss landscape with a surrogate model and then suggests the configuration to be evaluated in the next iteration. By trading off exploitation and exploration based on the surrogate model, it is well known for its sample efficiency.
Our Packages
- SMAC is a versatile tool for optimizing algorithm hyperparameters and implementing different surrogate models, acquisition functions, and model transformations.
- BOHB implements a variant of TPE as a BO approach.
Our Works
- JES (Joint Entropy Search) is a new information theory-based acquisition function for BO.
- Extensions to Tree Parzen Estimators (TPEs)
- c-TPE shows how constraints can be applied to the popular TPE model in BO for HPO.
- MO-TPE meta learns a mult-objective TPE that won the Multi-objective HPO for Transformers competition in AutoML-conf 2022.