[R] Nonparametric Masked Language Modeling - MetaAi 2022 - NPM - 500x fewer parameters than GPT-3 while outperforming it on zero-shot tasks

Submitted by Singularian2501 t3_zr2en7 in MachineLearning

Paper: https://arxiv.org/abs/2212.01349

Github: https://github.com/facebookresearch/NPM

Abstract:

>Existing language models (LMs) predict tokens with a softmax over a finite vocabulary, which can make it difficult to predict rare tokens or phrases. We introduce NPM, the first nonparametric masked language model that replaces this softmax with a nonparametric distribution over every phrase in a reference corpus. We show that NPM can be efficiently trained with a contrastive objective and an in-batch approximation to full corpus retrieval. Zero-shot evaluation on 9 closed-set tasks and 7 open-set tasks demonstrates that NPM outperforms significantly larger parametric models, either with or without a retrieve-and-generate approach. It is particularly better on dealing with rare patterns (word senses or facts), and predicting rare or nearly unseen words (e.g., non-Latin script).

https://preview.redd.it/qf2lqrkku47a1.jpg?width=658&format=pjpg&auto=webp&s=7dc7e76f3075b4b4f0916c2de1e442b19b2c0f49

https://preview.redd.it/gqhlbykku47a1.jpg?width=1241&format=pjpg&auto=webp&s=39f63470d18ea6f4a8ed560b371cc46b939b2c6f

https://preview.redd.it/p7bzdukku47a1.jpg?width=883&format=pjpg&auto=webp&s=6a8eb2b66abcb1581abf7280180c1c0e86201232

https://preview.redd.it/z6niwykku47a1.jpg?width=1112&format=pjpg&auto=webp&s=8337a4802db983df1a4b0b11934c0708888641a4

https://preview.redd.it/s8fdhxkku47a1.jpg?width=1361&format=pjpg&auto=webp&s=28b307df857ef2262d3f8348fd1094ebb793a63d

https://preview.redd.it/94t5fwkku47a1.jpg?width=1362&format=pjpg&auto=webp&s=da8bca8fd08ecaf956658c674f5a32a930cdd3a2

271

Comments

You must log in or register to comment.

rjromero t1_j12aza8 wrote

> We use the model architecture and initial weights of RoBERTa large (Liu et al., 2019), consisting of 354M parameters. Training is done for 100,000 steps, using thirty-two 32GB GPUs.

354M parameters? At FP32 that's 1.41gb. It's tiny.

55

Purplekeyboard t1_j12lik7 wrote

Ok, but how does it compare in the real world to GPT-3?

6

master3243 t1_j12nmgc wrote

Replying to Purplekeyboard (#991,894)

There's no way for a paper to just have a table of "real world comparison of GPT-3",

There needs to (for now) be some benchmark created that systematically tests for the things we care about. Which is exactly why I deeply respect researchers dedicated on creating better and more useful benchmarks as their work immensely accelerates the field while they mostly don't get the attention they (IMO) deserve.

31

Purplekeyboard t1_j12uk1s wrote

Replying to master3243 (#992,030)

But what I'm asking is, how do the benchmarks match real world performance? Because I've seen claims that other language models were supposedly close to or equal to GPT-3 in this or that benchmark, but try interacting with them and the difference is striking. It's like the difference between talking to a college grad student and talking to the meth addled homeless guy who shouts at lampposts.

12

valdanylchuk t1_j137hla wrote

Replying to Purplekeyboard (#992,485)

From the paper:

>Extension for generation. It is currently non-trivial to use NPM for generation, since it is the encoder-only model. Future work can explore autoregressive generation as done in Patel et al. (2022) or use NPM for editing (Schick et al., 2022; Gaoet al., 2022).

So, don't expect to talk to it just yet.

7

machinelearner77 t1_j1437x9 wrote

Looks like cool stuff... but if you put a code link in the abstract and publish your paper, it should be a functioning link...

3

blose1 t1_j14q7ul wrote

Replying to RealGrande (#992,750)

Have you actually tried both on same tasks? I mean it seems like a lot of people here read a paper and some blog and make their conclusion without even using the tool, I've used both on the same tasks, compared on hundreds of real world cases and yes it's fine-tuned GPT3 but with human assisted RL and it's doing circles around GPT-3 in question answering, COT and code generation.

2

yaosio t1_j15h0xa wrote

Replying to farmingvillein (#992,327)

They also say there's room for improvement but they didn't explore that in this paper. Just think, one day we'll have the power of the sun GPT-3 in the palm of our hand. Could be really soon, could be far away, but it's coming.

4

gbfar t1_j16478a wrote

I see lots of potential applications for this. I wonder if we could reliably control text generation by tweaking the reference corpus.

1

yaosio t1_j17p2bx wrote

Replying to Purplekeyboard (#992,485)

There was a thread awhile back about one benchmark being filled with spelling errors, grammar errors, and wrong answers. In many cases there were multiple correct answers but one was picked as the correct answer for no particular reason. Creating a benchmark for the subjective task of "is this text good?" seems to be pretty hard. It's even harder when the people creating the benchmark have a poor grasp of language.

If I were to ask a language model "Describe an apple." There are many correct answers, none more correct than the others. Multiple independent humans would have to go over the answers and make subjective decisions on if the LLM answerded well. This becomes much more difficult with better LLMs because the prompts and answers have to become more complex, which makes reviewing the answers harder and more time consuming.

1

drd13 t1_j1h3gvy wrote

Replying to [deleted] (#998,716)

Similarly to T5 (abd Bert) the model is pre-trained by predicting some randomly masked spans of words. However the way these spans of words are predicted is different.

In T5, masked words are generated one-by-one autoregressively (i.e. use a softmax over vocabulary to generate words one by one). Here a set of candidate possible spans, covering your whole trained corpus is preliminarily created and the model looks at all the candidate spans and chooses the one it thinks is the best (using a contrastive loss).

2