[D] Machine Learning conferences/journals with a mathematical slant?

Submitted by vajraadhvan t3_y6v03k in MachineLearning

I'm an undergrad coming from an applied mathematics background, and have been fascinated by mathematical approaches to the foundations of deep learning and ML in general (e.g., geometric deep learning, Ising models).

I'm currently working on a research project which is highly mathematical in flavour, and I was wondering if there are conferences, tracks, and/or journals geared towards more theoretical/mathematical results.

Would also be great to hear about how such results might be received at major ML conferences like ICML. Thanks!

71

Comments

You must log in or register to comment.

Red-Portal t1_isrrljo wrote

ICML, NeurIPS, ICLR all have theory papers. But theory people tend to complain that they often get anti-theory reviews. It's possible since the reviewers are so random that you might end up with strict empiricists. COLT on the other hand is pure theory. JMLR is also the most theory heavy journal in machine learning. Optimization people sometimes veer towards SIAM journals, while stuff closer to statistics would be fit for statistics journals like AoS, Bernoulli, JRSS etc.

47

master3243 t1_iss7aiv wrote

I remember trying to publish a theory paper (statistical learning theory) in ICML and got criticized by two reviewers that complained the paper had no experimental justification (despite being pure information theoretic lower bound of any learnt algorithm which was impossible to justify experimentally??) and my professor and I doubt they understood what was happening.

The third reviewer was extremely knowledgeable in this area and we truly appreciated their comments which definitely helped better the paper.

20

mietminderung t1_iss2l8t wrote

> But theory people tend to complain that they often get anti-theory reviews.

How the turn tables! A couple of decades ago - the boot was on the other foot.

12

Normal_Flan_1269 t1_isvwxli wrote

Hey, I’m actually interested in statistical learning theory in grad school. What kind of math prerequisites do I need to be able to research statistical learning in a phd program?

2

mietminderung t1_iswnzea wrote

A basic course in statistics and the desire for the PhD grind.

1

Normal_Flan_1269 t1_isx9q5s wrote

Dude cmon stop playing. I’ve read some of these papers and the math is crazy. What math courses do you need

2

Red-Portal t1_it5wz61 wrote

I kindda agree that you don't necessarily need a "math course" other than the usual requirements of CS undergrads. You somewhat pick up the rest on the way. I asked the same thing to theory grad students and they said the same thing. One fella that had a math BS actually said he didn't find his undergrad experience to be terribly useful, which kindda says a lot. Taking actual learning theory classes and reading textbooks will be necessary though.

2

Normal_Flan_1269 t1_it5yhim wrote

That doesn’t make sense tho. Math majors have more theory than cs majors. Statistical learning definitely requires high levels of mathematics, like functional analysis. It’s a huge area in statistics.

1

Red-Portal t1_it5zev6 wrote

The thing is, you can't learn everything in advance. And you don't need everything all the time. Some works in learning theory might be mathematically very deep, but taking a few undergrad math courses definitely won't prepare you for those. Although they might help you develop mathematical thinking. But as per the knowledge itself, I don't think taking math major courses is the most efficient way to do it.

2

Normal_Flan_1269 t1_it641pn wrote

So then your saying a cs major is the best major? All they learn how to do is code? How would you not need mathematical maturity to even make contributions to statistical learning theory. Like I would even argue a statistics major is more prepared than cs cause they have the background in stats with computing experience as well. Cs is just a software dev major.

0

Red-Portal t1_it649eh wrote

Because you do learn theory in undergrad CS...? On the contrary, no course teaches you how to code.

2

Normal_Flan_1269 t1_ithut4x wrote

Yeah you learn systems design, not functional analysis, measure theory, and actual mathematics to do the derivations in statistical learning theory. Statistical learning theory is mathematics. Not just coding.

1

Red-Portal t1_ithvm6b wrote

You learn algorithm analysis, computational complexity theory, discrete mathematics, automata, cryptography and whatnot. Do these seem like coding to you?

1

Normal_Flan_1269 t1_iti0m49 wrote

None of those are useful for creating new statistical learning methods or pushing the boundaries of statistical learning as functional analysis, measure theory, real analysis, and statistics. Like cryptography is useless for developing new regularized regression methods, who gives a shit about complexity theory? Like you guys think ML theory and statistical learning is a CS branch. Like you guys coin the term machine learning and think it’s a branch of CS… very far from the truth. Mathematicians and statisticians have been running circles around you guys doing this for decades. Know your place

1

Red-Portal t1_iti0ug1 wrote

Wait what? The core of learning theory is algorithm analysis and complexity theory! Please take any learning theory textbook or course first before making such groundless judgements. God the freakin definition of "PAC learnable" is algorithm-theoric.

1

Normal_Flan_1269 t1_iti100x wrote

It’s literally called statistical learning theory

1

Red-Portal t1_iti13ms wrote

Yes that's why you need at least "undergraduate" statistics knowledge.

1

Normal_Flan_1269 t1_iti2r8e wrote

Lol that’s so false. Undergrad stats is not nearly enough, not even undergrad math. You can get away with just knowing how to code a little bit but it’s way more math and stats. Cs majors are just trained to be software engineers and nothing else. I’m a math and stats major and run circles around them in AI courses because they don’t have any technical depth mathematically.

1

andreichiffa t1_isrrd2s wrote

Among A*, COLT is probably the best venue, with ICML being a great highly visible fit too. NeurIPS will need justification and a great intro, whereas ICLR is would need experimental proofs.

Overall heavily mathematical papers, when properly contextualized and given intuitive understanding of proofs tend to be very popular.

19

vajraadhvan OP t1_isrt93b wrote

Heartening to hear that a wide range of options are open to me. Thanks a bunch!

3

tfburns t1_ist5aup wrote

>Overall heavily mathematical papers, when properly contextualized and given intuitive understanding of proofs tend to be very popular.

Strongly disagree. MLers have a very limited appreciation of 'math'.

3

COPCAK t1_isrltwy wrote

COLT is the best ML theory-oriented conference that I'm aware of.

Theoretical papers are welcome at the major conferences, but not that common.

17

tfburns t1_ist506l wrote

>Theoretical papers are welcome at the major conferences, but not that common.

Agreed, which the caveat that 'theoretical' here rarely means more than 'statistics' / 'optimization'. Math is basically non-existent in ML.

1

TheDeviousPanda t1_isrcx1c wrote

I do not recall seeing any highly mathematical papers at ICML this year. What you are proposing might be better received at a conference like AISTATS, perhaps.

12

vajraadhvan OP t1_isrdpvl wrote

Ah, that's a bit of a shame. I recall seeing a talk called "Towards a Mathematical Theory of Machine Learning" by Weinan E at ICML 2022 (whom I'm citing!), but I'm guessing that's not indicative of the conference as a whole.

1

fhadley t1_isrgpnn wrote

My wife does pretty theoretical statistical learning work and has had success with JMLR

5

hostilereplicator t1_isrzugb wrote

I would echo the others here and say that, depending on the focus of your paper, the big conferences do take maths/theory papers (NeurIPS, ICML, ICLR, COLT, also AIStats and UAI depending on your topic) and JMLR for longer papers. But all of the conferences are both very competitive and have a large random component in what gets accepted… it may also be worth looking at workshops at these conferences to see if anything fits better. Less “prestigious” but also easier to get into and more likely to be reviewed by a suitable/friendly referee.

What’s the topic of your research?

7

vajraadhvan OP t1_iss3bb6 wrote

Approximation theory traditionally looks at the structure of function spaces under addition; but approximation spaces under composition are underexamined. Studying approximation spaces under composition may quantitatively explain the outperformance of neural networks, reveal links to dynamical systems, and suggest related architectures.

(Edit: Following the work of Weinan E, Chao Ma, Lei Wu, Ronald DeVore, Gitta Kutyniok, et al.)

4

tpinetz t1_isrv01o wrote

There are actual math journals such as simods for stuff like that. If you like Weinan E you can take a look at the venues he publishes in.

5

there_are_no_owls t1_isurqev wrote

I was confused why you would mention him in particular before seeing that OP mentioned him before lol. Yes I agree with this advice

2

eraoul t1_isrnh9o wrote

NeurIPS has some pretty technical mathy papers too, right? How does it stack up against others mentioned here?

4

tfburns t1_ist5noh wrote

>NeurIPS has some pretty technical mathy papers too, right?

Examples?

1

Seankala t1_isrzhkb wrote

You're not going to find theory-dense papers at major ML conferences. Most of the reviewers don't bother going through them and people usually find theory boring compared to "super duper cool" architectures that lead to 0.1% increase in performance. Like the top comment said, COLT is a good place to start.

2

tfburns t1_ist5iie wrote

>COLT is a good place to start.

Although it basically limited to optimization/stats, not math.

1

tfburns t1_ist4ozb wrote

Frankly, none of the major venues can reliably evaluate mathematical work. I have seen some reasonable evaluations of work which use some simple objects/concepts which are new to the ML community, but those are rare. There is quite a lot of statistics published, e.g. at COLT. But very little to no math. For that, you're better to go to math venues ime.

2

Tresz65 t1_istpu21 wrote

Aistats man!

2

ZombieRickyB t1_istz4ib wrote

SIMODS is the journal you are looking for. Be warned, the math here is often of a different flavor/level than what's discussed at conferences.

2

vajraadhvan OP t1_iswelgr wrote

Ah, it's by SIAM! Definitely of interest to me, I'll be sure to check the proceedings out.

1

howtorewriteaname t1_isvh9gc wrote

no idea but thanks for asking! I'm finding real good stuff in the answers

2

vajraadhvan OP t1_isvhcdf wrote

I know, right? The responses have been great so far.

1