Viewing a single comment thread. View all comments

blueSGL t1_j20lqfi wrote

Reply to comment by ThePlanckDiver in ChatGPT Could End Open Research in Deep Learning, Says Ex-Google Employee by lambolifeofficial

Google is likely the best positioned for dataset creation.

Text
>google search/cache/AMP links/youtube subtitles

Image
>Image search thumbnails/reCAPTCHA/youtube

Video
>Youtube

They can afford to give away research because I doubt many can match them on shear dataset scale alone.

13

treedmt t1_j218w1d wrote

This is an interesting point. Do you think the dataset that google has is high quality enough to actually train ai? In particular, search queries etc aren’t mapped to specific answers to be useful for supervised learning. Maybe I’m missing something?

1

blueSGL t1_j21ay8x wrote

LLMs where it's a statistical likelihood for next token prediction benefit from more data.

That along with the truism

"You always find things in the last place you look"

can be very powerful tools.

There will be some correlation between search term and result otherwise search would be pointless. That on a large enough scale can sift signal from noise, not only in terms of search results but in delta between individual search terms.

2

treedmt t1_j28o94z wrote

Surely there’s some trade off between qualitative vs quantitative data?

Eg. 50 billion high quality QA pairs may beat 500B random google queries as training data.

1