Has it Trained Yet? | Frank Schneider

Information

Location: New Orleans, USA
Date: December 2022
Organizers: F. Schneider, Z. Nado, P. Hennig, G. Dahl, N. Agarwal
Website: https://hity-workshop.github.io/NeurIPS2022/
Note: Co-organized a NeurIPS 2022 workshop.

Summary

Training contemporary neural networks is a lengthy and often costly process, both in human designer time and compute resources. Although the field has invented numerous approaches, neural network training still usually involves an inconvenient amount of "babysitting" to get the model to train properly. This not only requires enormous compute resources but also makes deep learning less accessible to outsiders and newcomers. This workshop will be centered around the question "How can we train neural networks faster" by focusing on the effects algorithms (not hardware or software developments) have on the training time of neural networks. These algorithmic improvements can come in the form of novel methods, e.g. new optimizers or more efficient data selection strategies, or through empirical experience, e.g. best practices for quickly identifying well-working hyperparameter settings or informative metrics to monitor during training.
We all think we know how to train deep neural networks, but we all seem to have different ideas. Ask any deep learning practitioner about the best practices of neural network training, and you will often hear a collection of arcane recipes. Frustratingly, these hacks vary wildly between companies and teams. This workshop offers a platform to talk about these ideas, agree on what is actually known, and what is just noise. In this sense, this will not be an "optimization workshop" in the mathematical sense (of which there have been several in the past, of course).
To this end, the workshop's goal is to connect two communities: Researchers who develop new algorithms for faster neural network training, such as new optimization methods or deep learning architectures. Practitioners who, through their work on real-world problems, are increasingly relying on "tricks of the trade". The workshop aims to close the gap between research and applications, identifying the most relevant current issues that hinder faster neural network training in practice.

Poll

For the 2022 NeurIPS workshop “Has it Trained Yet? A Workshop for Algorithmic Efficiency in Practical Neural Network Training” we conducted a survey to gather insights on the current training practices within the community. The poll was distributed through the workshop website, Twitter, email, and at the in-person NeurIPS event.

The survey’s results can be found in the in this repository.

Citation

If you use the results of the poll, please consider citing:

@misc{schneider2022HITY,
    title        = {HITY workshop poll, NeurIPS 2022},
    author       = {Frank Schneider and Zachary Nado and
                   Naman Agarwal and George E. Dahl and
                   Philipp Hennig},
    year         = {2022},
    howpublished = {\url{https://github.com/fsschneider/HITYWorkshopPoll}},
}