Ephemeral_Epoch t1_iqnscns wrote on October 1, 2022 at 7:30 PM Reply to comment by ClearlyCylindrical in [Discussion] If we had enough memory to always do full batch gradient descent, would we still need rmsprop/momentum/adam? by 029187 Seems like you could approximate a minibatch with a full batch + noise? Maybe there's a better noising procedure when using full batch gradients. Permalink Parent 5
Ephemeral_Epoch t1_iqnscns wrote
Reply to comment by ClearlyCylindrical in [Discussion] If we had enough memory to always do full batch gradient descent, would we still need rmsprop/momentum/adam? by 029187
Seems like you could approximate a minibatch with a full batch + noise? Maybe there's a better noising procedure when using full batch gradients.