>What is the standard modeling approach to these kinds of problems?

The standard approach is reinforcement learning. It works, but it's not very sample-efficient and takes many iterations to train.

LLMs are probably so good at this because of their strong meta-learning abilities; during the process of pretraining they not only learn the task but also learn good strategies for learning new tasks.

This has some really interesting implications. Pretraining seems to drastically improve sample efficiency even if the pretraining was on a very different task. Maybe we could pretrain on a very large amount of synthetic, generated data before doing our real training on our finitely-sized real datasets.

I guess that this is similar to trend that we have seen with “Decision Transformers” and how they have been proposed as a practical alternative for RL.

https://innermonologue.github.io/

[removed]

So cool!

https://arxiv.org/abs/2302.01560

Using language for long term planning in Minecraft.

I wonder if a LLM model would be able to play chess any better than a RL model. I imagine you'd need to serialize the board into a natural language format.

I’m not clever enough to fully understand what is required, but I’d be curious to see how an LM would behave in a DOTA setting. IIRC the OpenAI bot was RL-based and (1) took a lot of training cycles, (2) didn’t generalize all that well to different styles of adversarial play. Some of the qualities suggested by the papers linked by other commenters would seem advantageous…

Anybody else getting 90s text adventure flashbacks?

We can train with the algebraic notation. Feed it all the games and see how it goes. Or make the LLM self-play. ChatGPT can emit the notations, but all it generates are illegal moves.

Edit: after some corrections, it actually proposed the correct moves. Here are the prompts:

ME: The following is a series of chess moves after opening: 1. Nf3 Nf6 2. c4 g6 3. Nc3 Bg7 4. d4 0-0 5. Bf4 d5 What is the best next move?

ChatGPT: The next move depends on the playing style and strategy of both players. However, here are a few options that are often played in this position:

• e3 - This is a flexible move that allows the player to develop their bishop or castle kingside. It can also be useful to control the center.
• dxc5 - Taking the pawn on c5 can lead to a tactical or strategic advantage for White. This move can open up the c-file and create a strong pawn structure for White.
• Be2 - Developing the bishop to e2 can help to control the center and prepare to castle kingside.
• h3 - This is a prophylactic move to prevent Black from playing Bh4, which can be a nuisance for White.

It's important to note that all these moves have potential drawbacks and should be evaluated based on the player's individual preferences, the position of the other pieces on the board, and Black's response.

ME: Some of the moves are illegal. Look at the board again.

ChatGPT: I apologize for the mistake. Here are the corrected moves:

1. Nf3 Nf6 2. c4 g6 3. Nc3 Bg7 4. d4 d5 5. cxd5 Nxd5

In this position, some of the most common moves are:

• e3 - This is a flexible move that allows the player to develop their bishop or castle kingside. It can also be useful to control the center.
• Bd3 - Developing the bishop to d3 can help to control the center and prepare to castle kingside.
• Qc2 - Developing the queen to c2 can help to control the center and prepare to castle kingside.
• 0-0 - Castling kingside is a common move that helps to get the king to a safer square and connect the rooks.

It's important to keep in mind that these moves should be evaluated based on the player's individual preferences, the position of the other pieces on the board, and Black's response.

ME: No. The opening moves are: 1. Nf3 Nf6 2. c4 g6 3. Nc3 Bg7 4. d4 0-0 5. Bf4 d5

ChatGPT: Thank you for correcting me. Here are the moves as given:

1. Nf3 Nf6 2. c4 g6 3. Nc3 Bg7 4. d4 0-0 5. Bf4 d5

In this position, some of the most common moves for White are:

• e3 - This is a flexible move that allows the player to develop their bishop or castle kingside. It can also be useful to control the center.
• Bg5 - Developing the bishop to g5 can put pressure on the Black Knight on f6.
• Qb3 - Developing the queen to b3 can put pressure on the Black pawn on d5 and prepare to castle queenside.
• cxd5 - Taking the pawn on d5 can open up the center and create a strong pawn structure for White.

It's important to keep in mind that these moves should be evaluated based on the player's individual preferences, the position of the other pieces on the board, and Black's response.

Makes me wonder if pretraining makes the model converge on essentially a more efficient architecture that we could be using instead. I'm hoping this thought has already been explored, it would be interesting to read about.

I'm trying to make a decision transformer, but I can't quite figure out how to prompt it. I trained it on a chess dataset of (state, reward, move) but I don't know how to "prompt" it with the reward I would like.

Yes, and this example actually isn't all that impressive. Google demonstrated a LLM can significantly improve decision making for a real world robot, giving it a type of 'common sense'. Check out Palm-SayCan which is a collaboration of two models that can perform real world robotic tasks through the assistance of a language model.

This is probably only the case in which there’s a very low “compression ratio” of model parameters to learned entropy.

Basically, if the model has “too many” parameters it can be distilled but we’ve found that, empirically, until that point is hit, transformers scale extremely well and are generally better than any other known architecture.

Another topic is sparsificafion, which takes a trained model and tries to cut out some percentage of weights that have a minimal output effect, then fine tuning that model. You can check out Neural Magic online and associated works… they can run models on CPUs that normally require GPUs

I'm considering a higher level idea. There's no way that transformers are the end-all-be-all model architecture. By identifying the mechanisms that large models are learning, I'm hoping a better architecture can be found that reduces the total number of multiplications and samples needed for training. It's like feature engineering.

You prompt it by reward. Let's say your top reward is 1.

you predict model(past history, state, 1) -> move

Know any papers related to their work? Magic sounds deceptive...

That's not reasoning. It's spitting out semi-random moves. If you keep giving it more and more chances it increases the probability of getting a set which has some legal moves.

Alpha family of ai was created to be a decision engine.

I would think the monolithic llm wouldn't be as effective as using a delegation to a decision ai model.