Viewing a single comment thread. View all comments

Kolinnor t1_iww46to wrote

I agree with the article concerning Galactica, it was utter trash (EDIT : apparently you can still do some nice stuff with it) and excessively arrogant. I'm glad this terrible project just gets shut down.

However, I strongly disagree about the conclusion. It makes no doubt to me that this is the right direction : I've been helped by GPT-3 when studying math (for example today I explained that I wanted to know if a certain type of a function had a name, because I wasn't able to find anything on google, and it correctly understood my vague explanation), or it's just pretty good in general with "well-known" knowledge. The fact that it is really naive helped me to craft some intuition sometimes. Of course, it's still baby steps now, but big potential.

The article kinda downplays how good LLM are in general, kinda dismissing them as nonsense generator. But Gary Marcus being cited in the article is a big red flag for me as well.

19

TopicRepulsive7936 t1_iwwdd1u wrote

Somebody should research his background a little. Why is he so determined and why is he everywhere.

9

Kolinnor t1_iwyuae2 wrote

What do you mean ? I know he's popular for being highly controversial, but I wonder if I don't know all the story about him.

1

TopicRepulsive7936 t1_ix08pz5 wrote

Maybe it's nothing. But for decades he was a small time psychology professor until he bursted into the AI scene and like many he assumed he knew the business. Maybe he just likes attention, bit insecure. But you never know if he gets extra money from somewhere.

4

Veneck t1_ix2b5ot wrote

We seem to be experiencing a moment in time where being critical is being somewhat overly rewarded socially, as opposed to building things.

1

visarga t1_ix0ji8d wrote

> it was utter trash and excessively arrogant

Galactica is a great model for citation retrieval. It has innovations in citation learning and beats all other systems. Finding good citations is a time consuming task when writing papers.

It also has a so called <work> token that triggers additional resources such as a calculator or Python interpreter. This is potentially very powerful, combining neural and symbolic reasoning.

Another interesting finding from this paper is that a smaller, very high quality dataset can replace a much larger, noisy dataset. So there's a trade-off here between quality and quantity, it's not sure which direction has the most payoff.

I'd say the paper was targeted for critique because it comes from Yann LeCunn's AI institute. Yann has some enemies on Twitter since a few years ago. They don't forget or forgive. There's a good video on this topic by Yannic Kilcher.

And by the way, the demo still lives on HuggingFace: https://huggingface.co/spaces/lewtun/galactica-demo

5