With machine learning become ever more prevalent in society, it’s important that the methods we use are fair, non-discriminatory and that practitioners are aware of the biases that are present in both their data and the methods they use. We highlight the word fair as there is no single notion of fairness that we can optimize in machine learning that captures all the subtleties that comes with how the actual output of a machine learning model is used. So what can AutoML do? We wrote a blog post about this very topic at the intersection of AutoML and fairness, for which you can find the full paper here.
To highlight, what AutoML can do is be fairness-aware, enabling experts, who really know what concepts of fairness they care about to be in charge. AutoML systems take the human out-of-the-loop but there’s an increasing precedent for putting the human back in-the-loop, putting them in the drivers seat, so to speak. How can AutoML do this?
Multi-Objective and Constrained Optimization
AutoML provides the tools for optimizing for several metrics, such as some computable fairness metric and accuracy. Such metrics are often juxtaposed and there is a trade-off to consider. By using constrained optimization, AutoML finds performant models that hit a minimum threshold of what is deemed a good fairness metric, but it does not allow the domain expert to really understand this trade-off. AutoML tools can provide what is called a Pareto Front, a set of possible models that capture the trade-off inherent to these metrics, allowing the domain-expert to carefully consider what trade-off is suitable for their use case. This puts the domain expert in the driver seat as only they can know the downstream implications of their model, not AutoML.
AutoML for fairness in Face-Recognition
We show that Neural Architecture Search and Hyper Parameter Optimization can be used to discover inherently fair and and accurate architectures.
Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition
.