rePAN6517 t1_jc4du93 wrote
This will be huge for video games. The ability to run local inferencing on normal gaming hardware will mean every NPC can now be a smart character. I cant wait to be playing GTA6 and come across DAN walking down the streets of Vice City.
dojoteef OP t1_jc4e13h wrote
Not sure we're there yet, but I have some active research in this area right now.
rePAN6517 t1_jc4fq3l wrote
Give every NPC a name and short background description. IE - something like the rules that define ChatGPT or Sydney, but only to give each character a backstory and personality traits. Every time you interact with one of these NPCs, you load this background description into the start of the context window. At the end of each interaction, you save the interaction to their background description so future interactions can reference past interactions. You could keep all the NPC's backgrounds in a hashtable or something with the keys being their names, and the values being their background description. That way you only need one LLM running for all NPCs.
dojoteef OP t1_jc4hwyw wrote
If you actually want the NPCs to meaningfully add to the game rather than merely being mouthpieces then your approach won't work. How do you ensure what they say is consistent with the game world? E.g. what if they make up the location of a hidden treasure, offer to give you an item, etc. All of that needs to be accounted for in the game logic as well, otherwise they'll say things that make no sense in the game world.
It's actually a challenging problem and requires research. As far as I know there a very few people actively researching this area; if they are, then they certainly aren't publishing it. Hopefully my next paper which investigates using LLMs in Disco Elysium will help change that.
generatorman_ai t1_jc5w4m9 wrote
The general problem of generative NPCs seems like a subset of robotics rather than pure language models, so that still seems some way off (but Google made some progress with PaLM-E).
LLMs and Disco Elysium sounds like the coolest paper ever! I would love to follow you on twitter to get notified when you release the preprint.
dojoteef OP t1_jc6om7a wrote
Thanks for the vote of confidence!
Unfortunately, I recently deleted my twitter account 🫣. I was barely active there: a handful of tweets in nearly a decade and a half...
That said, I'll probably post my preprint on this sub when it's ready. I also need to recruit some play testers, so will probably post on r/discoelysium recruiting participants in the next few weeks (to ensure high quality evaluations we need people who have played the game before, rather than using typical crowdsourcing platforms like MTurk).
rePAN6517 t1_jc4jkbt wrote
Honestly I don't care if there's not complete consistency with the game world. Having it would be great, but you could do a "good enough" job with simple backstories getting prepended into the context window.
v_krishna t1_jc4orxw wrote
The consistent with the world type stuff could be built into the prompt engineering (e.g., tell the user about a map you have) and I think you could largely minimize hallucination but still have very realistic conversations
PriestOfFern t1_jc6x37m wrote
Take it from someone who spent a long time working on a davinchi support bot, it’s not that easy. It doesn’t matter how much time you spend working on the prompt, gpt will no matter what, find some way to randomly hallucinate something.
Sure it might get rid of a majority of hallucinating, but not a reasonable amount. Fine tuning might fix this (citation needed), but I haven’t played around with it enough to comfortably tell you.
v_krishna t1_jc7wzmx wrote
I don't doubt it. I've only been using it for workflow aids (copilot style stuff, and using it to generate unit tests to capture error handling conditions etc), and now we are piloting first generative text products but very human in the loop (customer data used to feed into a prompt but the output then feeds into an editor for a human being to proof and update before doing something with it). The amount of totally fake webinars hosted by totally fake people it has hallucinated is wild (the content and agendas and such sound great and are sensible but none of it exists!)
mattrobs t1_jcs3vvo wrote
Have you tried GPT4? It’s been quite resilient in my testing
blueSGL t1_jc5rpta wrote
could even have it regenerate the conversation prior to the vocal synt if the character fails to mention the keyword (e.g. map) in the conversation.
You know, like a percentage chance skill check. (I'm only half joking)
nonotan t1_jc53wlz wrote
"Smart character" would seem to be an awfully generous description for what you could realistically do with this, especially when mentioned alongside games like GTA, which very much do not revolve around text-based interactions. You can't really do a cutscene with an LLM today (you could have it generate a script, but how are you going to translate that to the screen automatically? that's highly non-trivial), nevermind leverage it to have individual characters actually behaving smartly within the game world.
If you're a game developer, do you want to dedicate the bulk of the user's VRAM/GPU time to text inference to... add some mildly dynamic textual descriptions to NPCs you encounter? Or would you rather use those resources to, y'know, actually render the game world?
rePAN6517 t1_jc585bd wrote
> If you're a game developer, do you want to dedicate the bulk of the user's VRAM/GPU time to text inference to... add some mildly dynamic textual descriptions to NPCs you encounter? Or would you rather use those resources to, y'know, actually render the game world?
When you're interacting with an NPC usually you're not moving around much and not paying attention to FPS either. LLM inference would only happen at interaction time and only for a brief second or so per interaction.
Jepacor t1_jc698s6 wrote
You can't just snap your fingers and instantly load and start up a multi GB LLM into VRAM while the game is running though.
zackline t1_jc69d50 wrote
I am not sure about it, but I have heard that it’s at the moment not possible to use CUDA while running a game because supposedly the GPU needs to enter a different mode or something like that.
If that should indeed be the case it might even be a hardware limitation that prevents this use case on current GPUs.
Viewing a single comment thread. View all comments