GPT-3 scores better than humans on Raven’s Progressive Matrices in a display of emergent analogical reasoning

Submitted by visarga t3_zzpe5h in singularity

So this time the question was - how can language models generalise better? Well, when you are in unfamiliar situations, you usually rely on analogies to make sense of the situations. How good is GPT-3 at analogies?

In order to test that it was necessary to create a task that was never seen before in its training, so it would be really unfamiliar.

> We found that GPT-3 displayed a surprisingly strong capacity for abstract pattern induction, matching or even surpassing human capabilities in most settings. Our results indicate that large language models such as GPT-3 have acquired an emergent ability to find zero-shot solutions to a broad range of analogy problems.

And these analogy problems are very dry. Just meaningless abstract patterns, nothing to relate to. It explains why GPT-3 is so good at coding and symbol manipulation.

[Emergent Analogical Reasoning in Large Language Models] (https://arxiv.org/abs/2212.09196v1)

107

Comments

You must log in or register to comment.

wu_wey t1_j2d2z4z wrote

This is a gem. Thanks for sharing the paper.

14

Lawjarp2 t1_j2d9ocm wrote

If true and it only gets better either through scale or through other novel architectures being added to existing models it indeed may be the beginning of AGI.

32

itachi8258 t1_j2dg3ky wrote

So if this is not the dawn of Artificial General Intelligence, I don't know what is.

12

troll_khan t1_j2e6w40 wrote

Let's see how GPT-4 performs in a few months.

22

Analog_AI t1_j2ed2s8 wrote

Replying to troll_khan (#1,232,032)

They were saying it won’t be larger than GPT-3. They want to focus on finessing it and squeezing more out or customizing and pre training before they increase size any further. It will be much better than it’s predecessor anyway. If this approach yields good harvest they won’t increase the size of GPT-5 either. They will only do so if they hit a wall.

5

Borrowedshorts t1_j2ffld8 wrote

Replying to Analog_AI (#1,234,163)

It seems like most AI companies have been doing this for now. I wonder if they're optimizing a local maxima instead of a global and that the global can only be achieved through further scale.

4

Analog_AI t1_j2fjzjc wrote

Replying to Borrowedshorts (#1,248,372)

They will not stop increasing size. More like taking a breather and squeezing progress from maximizing pre training Also waiting for more cost reduction for scaling. Just a breather. And they also work on tech for beyond the GPT models. More integration of vision recognition and other things will come. Some form of weak AGI will be here in the next 5 years.

3