What is the difference between backpropagation and gradient descent?
Back-propagation is the process of calculating the derivatives and gradient descent is the process of descending through the gradient. Moreover, adjusting the parameters of the model to go down through the loss function.
๐๐ซ๐๐๐ข๐๐ง๐ญ๐ฌ ๐ฐ๐ข๐ญ๐ก๐จ๐ฎ๐ญ ๐๐๐๐ค๐ฉ๐ซ๐จ๐ฉ๐๐ ๐๐ญ๐ข๐จ๐ง
We cannot think of Neural Nets without backpropagation, right? However, backpropagation has received critique in the neuroscience community as not being biologically plausible because neurons cannot “communicate in reverse mechanism” (i.e., have no backward connection).
Recently, I read this (https://lnkd.in/gr5cRReP) paper from ICLR where the authors suggested a finite differences-based method for estimating gradients: they choose a sample of random vectors (perturbation vectors) in the dimensionality of the parameters, estimate the gradient using finite differences for each of them, then average them to produce an unbiased gradient estimation. Intuitively, rather than having an analytical derivation of the chain rule throughout the neural network. Moreover, the gradient of each parameter with respect to its neighboring parameters โ evaluated at the parameter value โ can be approximated with this numerical technique once the forward pass is completed, the authors call this forward gradient and show some nice theoretical properties such as unbiasedness, and show some toy examples of how gradient descent can be successfully applied with this technique. When compared to backpropagation, this approach has a slightly lower runtime computing cost and similar memory needs.
Could this be a step toward a biologically reasonable learning mechanism?
