How do you prevent overfitting?
Science
Dropout: classic way to prevent over-fitting
Dropout: A Simple Way to Prevent Neural Networks from Overfitting [1]
As one of the most famous papers in deep learning, Dropout: A Simple Way to Prevent Neural Networks from Overfitting gives far-reaching implications for mitigating overfitting in neural networks.
Deep neural nets with many parameters are very powerful machine learning systems, however, overfitting is a serious issue when doing regression or classification problems. Furthermore, to improve the generalization ability of the neural network, the author proposes the Dropout technology. The paper conducted various experiments to prove Dropout`s performance and discussed the effect of Dropout on extracting features and sparsity.
Is Dropout a decent normalization technique?
Given a task, the large-scaled dataset is more likely to contain valuable patterns and a small-scaled dataset often contains noise, which is the pattern that can only be revealed on the small dataset. Moreover, a model with strong fitting ability often has more parameters and higher complexity.
One of the methods to measure a regularization technique is to use a strong fitting model on a small-scaled dataset. In this case, the model is more likely to fit the noise. If the model performs well on the test set, to a certain extent, it shows that Dropout can avoid model fitting noise.
Figure 1: Effect of varying data set size.
The paper compared the network`s classification error under various sizes of datasets using Dropout and without using Dropout and proved that when the dataset is large, Dropout can effectively prevent overfitting (Figure 1).
The author discussed the effect of Dropout on extracting features.
The paper uses an autoencoder with one hidden layer and 256 ReLU units and compares the extracted features using dropout and without using dropout.
Figure 2: Features learned on MNIST with one hidden layer autoencoders having 256 rectified linear units.
Both autoencoders had the same performance on the validation set (Figure 2), however, compared to (a), the autoencoder with Dropout extracted more meaningful features such as contours.
The author discussed the effect of Dropout on sparsity.
In addition, through experiments, the paper proved that the use of dropout can improve the sparsity of the network. The sparsity makes the neural network model more similar to the biological neural network, and at the same time, it makes the network more robust to small changes in the input, thereby improving the generalization ability.
The author uses the encoder trained in the previous section to sample batches on the test set randomly and count the output of the activation function. The specific results are as follows (Figure 3):
Figure 3: Effect of dropout on sparsity. ReLUs were used for both models. Left: The histogram of mean activations shows that most units have a mean activation of about 2.0. The histogram of activations shows a huge mode away from zero. Clearly, a large fraction of units have high activation. Right: The histogram of mean activations shows that most units have a smaller mean activation of about 0.7. The histogram of activations shows a sharp peak at zero. Very few units have high activation.
Moreover, the author compares the histogram of the mean activation value (the average activation function value of a neuron on a random sampling batch) and the histogram of activation value (the activation function output of all neurons on a random sampling batch). The data of autoencoder with Dropout in both histograms are significantly more biased towards 0. Hence the paper proved that Dropout may improve the sparsity of the network, hence improving the generalization of the network.
Although Dropout is a useful way to improve the performance of the network, it also has drawbacks.
Since some neurons are discarded each time during training, it may cause gradient to oscillate and increase the overall training time.
As a classic paper in deep learning, the Dropout technique is certainly revolutionary.
However, as a paper proposed in 2014, some of the assumptions and conclusions made by the authors have been challenged over time. For example, some researchers have argued that Dropout does not always prevent overfitting. Furthermore its effectiveness can vary depending on the specific task and architecture.
Moreover, some studies have found that Dropout can be harmful to performance in certain situations.
Such as small datasets or shallow networks. Additionally, some researchers have suggested that other regularization techniques. Furthermore, such as weight decay and early stopping, can be more effective than Dropout in certain scenarios.
As a beginner who wants to delve deep into machine learning.
Understanding the principles and effectiveness of dropout moreover, can help you in implementing and fine-tuning neural network models.
The paper can also provide insight into the evolution of regularization techniques and the ongoing effort to improve the performance and efficiency of deep learning models.
Dropout is a classic normalization technique in deep learning,
The paper systematically explained the motivation and the principle behind Dropout and used various experiments to successfully verify its generalization ability.
Dropout is a widely used and effective technique for regularizing neural networks, and further research is ongoing to better understand its limitations and optimal usage conditions.
Reference:
[1] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1 (January 2014), 1929–1958.
How do you prevent overfitting?
