fANOVA is a tool for assessing the importance of an algorithm’s hyperparameters. It takes as input performance data gathered with different hyperparameter settings of the algorithm, fits a random forest to capture the relationship between hyperparameters and performance, and then applies functional ANOVA to assess how important each of the hyperparameters and each low-order interaction of hyperparameters is to performance.
Software
The fANOVA package can be obtained on our github page (external link). Documentation is available on github.io (external link).
Reference
Frank Hutter, Holger Hoos, and Kevin Leyton-Brown.
An Efficient Approach for Assessing Hyperparameter Importance [pdf] [pdf long version] [bib]
In: International Conference on Machine Learning (ICML’14).