Viewing a single comment thread. View all comments

IntelArtiGen t1_ivg33d5 wrote

Reply to comment by billjames1685 in [D] At what tasks are models better than humans given the same amount of data? by billjames1685

>I’m pretty sure it’s been well established that we can learn after seeing a few images even for things we haven’t seen before

An 18 years old can do that. Ask a 1 y.o. to identify 50 different objects, it won't work, even though this 1 y.o. was trained continuously on thousands of images during his first year of life. Of course you were not talking about training a 1 y.o. but an adult, and that's why you can't really compare. In order to be an adult you need to be a 1 y.o., you need to watch the world during thousands of days before you can have that "pretraining" that makes adults able to handle all these tasks more easily than most models.

>our brains have way more compute than any model

That's not as well-established as many people could think. We would want models to do what an 18 years old could do, yet no deep learning model has been trained with real-world interactions for 18 years.

30

blimpyway t1_ivivcwr wrote

Tesla collected 780M miles of driving till 2016

A human learning to drive for 16h/day at an average speed of 30mph for 18years would have a data set of ~3M miles.

So we can say humans are at least 1000 times more sample efficient than whatever Tesla and any other autonomous driving companies are doing.

−1

The_Real_RM t1_ivizjvu wrote

You are assuming Tesla actually needs all that data to train a competing model, you're also ignoring all of the other training a human has before ever starting to drive. It's not so clear who is more efficient, not at all.

I think a better way to compare is thorough the lense of energy, a human brain runs on about 40w of energy, Tesla's models are trained on MW scale computers, how do they compare in terms of total energy spent to achieve certain performance?

5

IntelArtiGen t1_ivj6nih wrote

Probably not, because a 16 y.o. human has 16 years of interactive navigation pretraining in a real world environment in real time before learning to drive. So it depends on how you include this pretraining.

And it also depends on the accuracy of the model as a function of the size of the dataset. Let's say Tesla is 80% (random number) accurate while driving after training on 780M miles, a human is 75% accurate after 3M miles, and if you train the Tesla model on 3M miles instead of 780M it's 75% accurate, on these metrics alone Tesla would be as efficient as a human.

No comparison is perfect but we can't ignore that during the first years of our lives we train to understand the world while not being very efficient to perform tasks.

1

billjames1685 OP t1_ivg3dy8 wrote

Yeah I addressed that in the second paragraph; we have been pretrained on enough image classification tasks that we probably have some transfer learning-esque reasons leading to our few shot capabilities.

−13

IntelArtiGen t1_ivg4d74 wrote

I think it's not just "transfer learning" or "image classification" it's also just learning without explicitly using "labels". Like contrastive learning / self supervised learning / reinforcement learning etc.

12

billjames1685 OP t1_ivg5bij wrote

Yeah I agree. Not sure if I’m misunderstanding you, but by “transfer learning” I basically mean like all of our pre training (which occurred through a variety of methods as you point out) has allowed us to richly understand images as a whole, so we can apply and generalize well in semi-new tasks/domain.

−5

IntelArtiGen t1_ivg765c wrote

Ok that's one way to say it I also agree. I tend to not use the concept of "transfer learning" for how we learn because I think it's more appropriate for well-defined tasks and we are rarely confronted with tasks that are as well-defined as the ones we give to our models.

And transfer learning implies that you have to re-train a part of the model on a new task, and that's not exactly how I would define what we do. When I worked on reproducing how we learn words I instead implemented the solution as a way to put a new label on a representation we were already able to produce based on our unsupervised pretraining. I don't know which way is the correct one I just know that doing that works and that you can teach new words/labels to a model without retraining it.

3

billjames1685 OP t1_ivh33jr wrote

That’s a fair point; I was kind of just using it as a general term.

2