AI plays Minecraft while constantly improving its own code

Bots for games have reached a new level. GPT-4 plays Minecraft for you! (Image: Adobe Stock / JorgeEduardo / OpenAI)

It sounds a bit like the story of a science fiction movie: an AI is created for reason X and has the ability to keep getting smarter. In the mostly dark end-time flicks, such an AI is often the reason for the end of the world.

In reality, however, we are (still) spared from terrorist AIs. Instead, an AI-controlled bot can automatically build its skills in Minecraft.

Minecraft + GPT-4

A team of Nvidia researchers led by Linxi Jim Fan has developed an innovative method to integrate the GPT-4 language model into the popular block game Minecraft.

The from Nvidia employees developed Minecraft bot, called Voyager, uses GPT-4 to solve problems in the game. The language model generates goals that help the bot explore the game and generates code that improves the bot’s capabilities over time.

link to Twitter content

Voyager plays Minecraft not like a human, but uses an APIto capture the current game state. For example, if he has a fishing rod in inventory and sees a river nearby, GPT-4 will suggest that Voyager fish to gain experience.

To achieve this goal, GPT-4 generates the code necessary for the bot to complete the task.

Self-learning wants to be learned

The most innovative part of this project lies in the code generated by GPT-4. He adds new facets to Voyager’s behavior. If the originally proposed code doesn’t work optimally, Voyager will refine it based on error messages, in-game feedback, and a description of the code generated by GPT-4.

Over time, Voyager creates one extensive code libraryto continually complete more complex tasks and further explore the game.

The graph shows the effectiveness of the Voyager bot compared to other bots. (Image: Linxi Fan)

A graph created by the researchers shows that Voyager acquires over three times more items, explores more than twice as far, and creates tools 15 times faster than other AI agents (Reflection, ReAct & AutoGPT).

According to Fan, the system could be further improved in the future by incorporating visual information from the game.

Language models and their bottomless potential

While chatbots like ChatGPT have wowed the world with their eloquent language and obvious knowledge, Voyager shows the enormous potential of language models to perform helpful actions on computers.

Applying language models in this way could potentially be many Automate routine tasks in the officewhich could have a significant economic impact.

The connection between GPT-4 and Minecraft through the Voyager bot illustrates the exciting potential of language models, far beyond just having a conversation. Automating tasks through language models in connection with games like Minecraft could open the door to a variety of new applications.

From helping with office tasks to developing intelligent software assistants that can automate tasks on PCs or phones, the possibilities are endless.

Which tasks would you like to automate? Farm diamonds in Minecraft or do you prefer the tax return? Write it to us in the comments!