Viewing a single comment thread. View all comments

yldedly t1_j9j6gk8 wrote

>discover arbitrary functions

Uh, no. Not even close. DL can approximate arbitrary functions on a bounded interval given enough data, parameters and compute.

14

ewankenobi t1_j9jl91t wrote

I like your wording, did you come up with that definition yourself or is it from a paper?

1

yldedly t1_j9jorh1 wrote

It's not from a paper, but it's pretty uncontroversial I think - though people like to forget about the "bounded interval" part, or at least what it implies about extrapolation.

9

[deleted] t1_j9jsgf6 wrote

What is "bounded interval" here?

1

yldedly t1_j9judc7 wrote

Any interval [a; b] where a and b are numbers. In practice, it means that the approximation will be good in the parts of the domain where there is training data. I have a concrete example in a blog post of mine: https://deoxyribose.github.io/No-Shortcuts-to-Knowledge/

8

[deleted] t1_j9jukfu wrote

Interesting but that is valid for us as well. So I am not sure this is true once they learn very general things, like learning itself.

0

GraciousReformer OP t1_j9jfxvh wrote

Yes but is DL the unique mechanism? Why DL?

−10

yldedly t1_j9jpuky wrote

There are two aspects, scalability and inductive bias. DL is scalable because compositions of differentiable functions make backpropagation fast, and those functions being mostly matrix multiplications make GPU acceleration effective. Combine this with stochastic gradients, and you can train on very large datasets very quickly.
Inductive biases make DL effective in practice, not just in theory. While the universal approximation theorem guarantees that an architecture and weight-setting exist that approximate a given function, the bias of DL towards low-dimensional smooth manifolds reflects many real-world datasets, meaning that SGD will easily find a local optimum with these properties (and when it doesn't, for example on tabular data where discontinuities are common, DL performs worse than alternatives, even if with more data it would eventually approximate a discontinuity).

4

GraciousReformer OP t1_j9jr4i4 wrote

"for example on tabular data where discontinuities are common, DL performs worse than alternatives, even if with more data it would eventually approximate a discontinuity." True. Is there references on this issue?

1

yldedly t1_j9jr821 wrote

This one is pretty good: https://arxiv.org/abs/2207.08815

1

GraciousReformer OP t1_j9jrhjd wrote

This is a great point. Thank you. So do you mean that DL work for language models only when they get a large amount of data?

2

GraciousReformer OP t1_j9k1srq wrote

But then what is the difference from the result that NN works better for ImageNet?

1

yldedly t1_j9k3orr wrote

Not sure what you're asking. CNNs have inductive biases suited for images.

3

GraciousReformer OP t1_j9k4974 wrote

So it works for images but not for tabular data?

1

yldedly t1_j9k5n8n wrote

It depends a lot on what you mean by works. You can get a low test error with NNs on tabular data if you have enough of it. For smaller datasets, you'll get a lower test error using tree ensembles. For low out-of-distribution error neither will work.

3