dpkingma t1_irpaffs wrote on October 10, 2022 at 12:15 AM

Reply to comment by picardythird in [D] What kind of mental framework/thought process the researchers have when working on solving/proving the math of the new algorithms? by aviisu

I agree with the (milder) claim that most methods in the wild are empirical.

With regards to Adam: I wouldn't say that this method was ad hoc. The method was motivated by the ideas in Sections 2 and 3 of the paper (update rule / initialization bias correction), which are correct. The convergence result in Section 4 should be ignored, but didn't play a role in the conception of the problem, and wasn't very relevant in practice anyway due to the convexity assumption.