Submitted by fromnighttilldawn t3_y11a7r in MachineLearning
Critiques on ML approach, technique, implementation, reproducibility or entire field of research, can often be equally (if not more) enlightening as compared to ML surveys.
I think this is because they usually point out what the field is ignoring or if a certain set of popular practice/belief is unsound or useless.
Some famous examples are:
Troubling Trends in ML https://arxiv.org/pdf/1807.03341.pdf
ML that Matters https://arxiv.org/abs/1206.4656
On the Convergence of ADAM https://arxiv.org/abs/1904.09237
On the Information Bottleneck https://iopscience.iop.org/article/10.1088/1742-5468/ab3985
Implementation Matters in Deep Policy Gradients https://arxiv.org/abs/2005.12729 (showed a certain purported algorithm gain is actually mainly due to code-level optimization)
Critique of Turing Award https://people.idsia.ch/~juergen/critique-turing-award-bengio-hinton-lecun.html (basically a critique on the citation practice in ML)
Deep Learning a Critical Appraisal https://arxiv.org/abs/1801.00631
However, these are a little bit dated.
Does anyone have any recent critique papers of similar flavour as the ones I've provided above? (or would you rather offer your original critique in the comments ;) )
_Arsenie_Boca_ t1_irvjdtn wrote
I dont have the papers on hand that investigate this, but here are 2 things that dont make me proud of being part of this field.
Are transformers really architecturally better than LSTMs or is their success mainly due to the huge amount of compute and data we throw at them? More generally, papers tend to make many changes to a system and credit the improvement to the thing they are most proud of without a fair comparison.
Non-opensource models like GPT3 dont make their training dataset public. People evaluate the performance on benchmarks but nobody can say for sure if the benchmark data was in the training data. ML used to be very cautious about data leakage, but this is simply ignored in most cases when its about those models.