ICLR 2022 Workshop
2022 | ML Evaluation Standards
Information
- Location: Virtual
- Date: April 2022
- Talk Title: Improving Optimizer Evaluation in Deep Learning
- Website: https://ml-eval.github.io/
- Slides:
- Note: Invited talk and panelist at the ICLR 2022 Workshop for "ML Evaluation Standards" (held virtually).
Improving Optimizer Evaluation in Deep Learning
Although hundreds of optimization algorithms have been proposed for deep learning, there is no widely agreed-upon protocol for evaluating their efficiency, performance, and usability. Instead, the crucial choice of the optimizer is too often done based on personal anecdotes instead of grounded empirical evidence. In this talk, we present strategies for comparing deep learning optimizers which consider the unique challenges of deep learning such as the inherent stochasticity or the crucial distinction between learning and pure optimization. These strategies are formalized and automatized in the Python package DeepOBS, which allows fairer, faster, and more convincing empirical comparisons of deep learning optimizers. Following this protocol, we report insights from our independent, third-party evaluation of the field's current state. A thorough comparison of fifteen popular deep learning optimizers, using roughly 50,000 individual runs, reveals that the comparably traditional Adam optimizer remains a strong but not dominating contender and that newer methods fail to consistently outperform it. As an adjacent research direction to benchmarks, new debugging tools, such as Cockpit, allow for a more detailed evaluation of the training process of neural networks beyond just the loss or the model's performance. These tools could disentangle the many factors contributing to (un)successful neural network training, helping us understand whether training improvements are the result of better models, better algorithms, or better hyperparameters.