NPCs will be much more realistic in the future

The simulated village. Source: “Generative Agents: Interactive Simulacra of Human Behavior”, Park et al.

AI assistants like ChatGPT or the new GPT-4 are suitable for more and more tasks, although maybe not for your homework. Now there is the first application that can bring some benefit to video games.

If you’ve always dreamed of characters in a video game world not only having static processes, but reacting to the world and each other almost as dynamically as players in a pen and paper role-playing game, then science is giving you first impressions how that could look like!

Researchers from Stanford University and Google have built their own small village and let the residents go about their daily lives, only controlled by ChatGPT. We took a look at the trade publication and explain to you how it all works and how it turns a single line into an entire Valentine’s Day celebration with invitations, appointments and so on.

ChatGPT as the basis for everything

Behind the interactive village is ChatGPT, i.e. a neural network. The has learned to answer text input from a user by predicting a text word fragment by word fragment. The basic idea of ​​turning this into a village simulation is easy to explain.

As village life progresses and we want to know how a character is behaving, we give ChatGPT a brief summary of the state of our character and the world, and ask it to derive an action from that.

A quick example: Input: Tim is a freelance writer at GameStar and has just discovered a new paper about a simulated village. What is he doing? ChatGPT: Tim might play a round of League of Legends write an article about it.

Like a classic NPC

In order to use it to control NPCs, the authors proceed exactly as they do with passers-by in GTA 5: The environment is perceived (The player draws a weapon), whereupon the condition of the passer-by of stroll down the street on flee from the player will be changed.

The cycle of an NPC. Source: “Generative Agents: Interactive Simulacra of Human Behavior”, Park et al.

An action is then derived from the new state, for example turn away from player, then run. The authors don’t use any fixed rules given by a programmer, but let ChatGPT decide what happens and translate it back into something that the simulation understands.

KI-dew Valley

For their simulation in a web game engine, the researchers built a small village with several residential buildings and infrastructure including a café, bar and grocery store. In addition, all characters have received a character sheet, similar to a pen and paper role-playing game. Properties such as age, character traits and a short description are stored there.

A character arc of the helpful pharmacist. Source: “Generative Agents: Interactive Simulacra of Human Behavior”, Park et al.

Each NPC also has a memory in which current events are stored with a time stamp. For example, that breakfast was eaten in the morning and that the neighbor was met on the way to work.

When the ChatGPT roleplays

Anyone who has ever worked with ChatGPT knows that information is often lost when texts are too long, or that information that is not so important comes to the fore.

In order to avoid this, the authors regularly let ChatGPT draw conclusions from the individual memories, depending on their relevance, topicality and importance (if the NPC is in the kitchen, the memory of one’s own hunger is more relevant than at work).

Rating the importance, for example, also works by telling ChatGPT that it should rate the importance given a context (You are hungry. How important is that on a scale of 1 to 10 when you’re in the kitchen?).

In the end, a time-dependent work plan is created with ChatGPT from the memories, character information and conclusions. For example, that the character gets ready in the morning, works during the day and goes shopping in the evening.

So that the texts are not too long and prone to errors, this large plan is structured into several small, finely divided actions. ChatGPT thus receives all the information from the NPC again and, for example, the task of going shopping from 4 p.m. to 5 p.m. in order to forge a detailed plan that then, again with ChatGPT, leads to a concrete action.

Connection of simulation and data

In order for the simulation and the language model to work together meaningfully with your data, they still have to communicate with each other. When an NPC enters the room, the simulation must report that the characters already standing there now know who is in their line of sight.

Conversely, the actions of the characters must of course also be implemented. If a character decides to go to work or to the bar, the model then makes a corresponding movement of the character model in the game world, i.e. it calculates a route in the classic way and animate the character along this path.

Interaction with the game world

The player has the opportunity to interact with both sides of the world. Either can Thoughts are given directly to the respective character, i.e. the language model is told: The character wants to run for mayorwhich is then written into the character accordingly.

But the game world itself can also be changed, for example a stove can be switched on so that a character then gets the message when entering the room: The stove is onand react accordingly.

Valentine’s Day party

It’s especially exciting when you implant new ideas into the characters and then observe how they play out. For example, a character was told that she wanted to organize a Valentine’s Day party. She talks to other NPCs about it, who then remember this information.

How news about the Valentine’s Day party spreads. Source: “Generative Agents: Interactive Simulacra of Human Behavior”, Park et al.

If these characters meet other characters again, they tell them about it, invite other characters that suit them well to dates, or start decorating the café accordingly. At the time of the celebration, they then run into the café.

Of course it is questionable whether something like this always works particularly well, but just such a form of interaction that in a normal Game requires a lot of scripted sequences between all characters in the code, just so that the sequence then runs the same every time, is still impressive and shows the strengths of ChatGPT in such scenarios.

AI with amnesia and hallucinations

Of course, with ChatGPT as the foundation, the simulated village also inherits the weaknesses of the model, which we describe in more detail here. This leads to characters simply making things up, like telling others about an unplanned appearance by a mayoral candidate.

In addition, neighbors whose names happen to coincide with well-known personalities are often confused with them, and so the neighbor becomes an economist from the 18th century. It gets even weirder when a character remembers what they have to do at an event, but no longer knows that the event even exists.

But it has to be mentioned that GPT-4, with its ability to process longer texts more precisely, should have an advantage here and should be easier to integrate.

If you want to learn more about it, we can highly recommend the authors’ very pleasant to read, albeit English publication, which is in fact largely free of Technical jargon is.

What do you think: are we on the way to sandbox games that also deserve their name with the first steps with AI like ChatGPT? Or will high computing costs, faulty AI and occasionally failing filters for obscene content, for example, thwart the whole thing for a long time to come? We look forward to your opinions, comments and ideas in the comments!