How do you assess the performance of a neural network?
Artificial Intelligence & Machine Learning
Deep Learning is all the rage right now
Many teams are training their own Deep Neural Networks (DNNs) from scratch.
Others choose to fine-tune an existing open-source model.
However, the thing is, it is very hard to determine if a model is converging properly.
Moreover, the only way to tell is by monitoring the test (or holdout) accuracy, or, say, some proxy for the model quality. The ‘thing is’, some layers may become overtrained, others undertrained, and as a result, there has been no way to really tell. That is, until now. Enter weightwatcher.
WeightWatcher is an open-source, diagnostic tool for analyzing Deep Neural Networks (DNN).
A tool that can evaluate the quality of a DNN model, layer-=by-layer, without even needing access to the training data.
How is this possible? WeightWatcher implements ideas from Random Matrix Theory which are familiar in quantitative finance but really quite new to the AI community.
The theory behind weightwatcher, the theory of Heavy-Tailed Self-Regularization (HT-SR), has recently been published in JMLR (https://jmlr.org/papers/v22/20-410.html).
And the tool has been tested on hundreds of models, with the remarkably results published
in the highly prestigious journal Nature Communications (https://www.nature.com/articles/s41467-021-24025-8).
The weightwatcher tool offers unique insights into DNN models, both during and after training.
It can find poorly trained layers, overfit layers, and/or layers with training anomalies.
By doing this, it can help you determine if you have trained your model with enough data or you need to add more. It can also help you find, and fix, problems in the model layers that can not be detected using other theories.
And, amazingly, it can become used to estimate, or rank, the quality of a set of models– furthermore, without even needing testing data!
Lastly, you can learn more about this unique and useful tool and how to use it at https://weightwatcher.ai
Enjoy — and happy training.
