Each float32 is 4 bytes.

Well if you see variety in the top results in google that might give you pause. But you're not getting that from ChatGPT

Well obviously. Search is a tool for information retrieval (mostly). If you have an oracle, it's much more convenient than digging through the source material and doing the research yourself, even when it is presented to you in most relevant first order, which is the most convenient order and what made google successful in the first place.

But yes, anyone reading please don't use ChatGPT instead of google search unless you don't care about the responses being made up.

I mean the more you understand the better obviously. But it's not necessary, it's just context for what we don't do anymore.

If I recall correctly, ViT is a purely transformer based architecture. So you don't need to know RNNs or CNNs, just transformers.

What do you mean by that?

Trees/forests are still state of the art for structured data... So not only did they not give up on them, but traditional methods are seen as better in some domains. Not to mention the ease of use, and the quick training.

Also explainable AI is much more promising with traditional methods, especially trees.

Gradient descent is also about optimization... You can optimize even neural networks with a bunch of different methods other than gradient descent (including evolutionary methods). They don't work as well but you can still do it.

They are not gradient-descent based (so they don't need GPU acceleration as much, but sometimes times still do depending on the problem) but they are definitely ML.

Fine tuning isn’t any different than just training…? You just don’t start with random network, but fine tuning doesn’t really have anything different besides that and the size of the dataset

How do you detect text produced by GPT? Is there like open source code?

Architecturally probably some form of unet is best. It’s the architecture of choice for things like segmentation so I imagine it would be good for IR as well

Having noisy targets is a known augmentation technique, so I don't think it's a problem.

I have the same association as you if I hear classic (ML) methods. But not classic (AI) methods, those I associate with good old fashioned AI, which aren't statistical.

Maybe it's just me, idk. I studied AI in philosophy long before I took an ML class. And I took my first intro to ML class before they were teaching deep learning in intro to ML classes (though i missed this cut-off only by a year or two haha).

Search + hard-coded (expert provided) rules, for example. Deep Blue that beat Kasparov didn't have any statistics in it iirc.

Deductive reasoning (as opposed to inductive which is what statistical/ML methods are), so like reasoning from first principles that are hard coded into the system.

Refers to what’s called symbolic ai that uses logic, and deductions.

Idk what business logic is but maybe. Definitely not neural nets.

I totally buy this. However you said

> Classical methods outperform Machine Learning methods in terms of speed, accuracy, and cost

those classical methods are also machine learning methods. Classic AI methods usually refers to non-statistical methods

Crazy that we are now far enough into AI research that we are comparing chatbots to coworkers.

It's negative data. It's basically contrastive learning, except without backprop. Like you pass a positive example and then a negative example in each forward pass, and update the weights based on how they fired in each pass.

It's a really cool idea, I'm just interested if it's actually biologically plausible.

I might be wrong but inhibitory synaptic connections sounds like a neural connection with weight 0, i.e. it doesn't fire with the other neuron.

I’ve heard that hebbian learning is how brains learn and this doesn’t seem like hebbian learning.

However idk if hebbian learning is even how neuroscientists think we learn in contemporary research

Is this actually biologically plausible? Seems that the idea of negative data is pretty constructed.

I see that Hinton claims it's biologically more plausible, but I don't see any justification for that statement apart from comparing it to other biologically plausible approaches, and more so spending time discussing why backprop is definitely not biologically plausible.

I'm not a neuroscientist so don't have much background on this.

It was just an example. Sure not all sizes of nose are found along the same eigenvector.

It’s not really luck. There is variation in sizes of noses (it’s one of the most varied features of the face) and so that variance is guaranteed to be represented in the eigenvectors.

And beta-VAEs are one of the possible things you can try to get a disentangled latent space yes, although they don’t really work that well in my experience.

It depends on what data you're working with and what you're trying to do. For example for me, I've worked a lot with 3d datasets of meshes of faces and bodies that are in correspondence. And I actually used autoencoders to compress them to the same dimensions as PCA and compared the two.

Basically with the network I'd get less error on reconstruction (especially at lower dimensions). However, the beauty of the PCA reduction was that one dimension was responsible for the size of the nose on the face, another was responsible for how wide or tall the head is, etc.

And you don't get such nice properties from a fancy VAE latent space. Well you can get a nice disentangled latent space but they don't happen for free usually, you often need to add even more complexity to get so nice and disentangled. With PCA, it's there by design.

I’d say that PCA is the most useful method still. The fact that it’s very quick and is a linear transform makes it very easy to use and interpretable