[WOW!] Google’s GameNGen: AI breaks new ground by simulating [interactive] Doom without a game engine

[Hamartia Antidote]

https://venturebeat.com/ai/googles-gamengen-ai-breaks-new-ground-by-simulating-doom-without-a-game-engine/

Google researchers have reached a major milestone in AI by creating a neural network that can generate real-time gameplay for the classic shooter Doom—without using a traditional game engine. This system, called GameNGen, marks a significant step forward in AI, producing playable gameplay at 20 frames per second on a single chip, with each frame predicted by a diffusion model.

“We present GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction with a complex environment over long trajectories at high quality,” the researchers state in their paper, published on the preprint server arXiv.

[This is AI generating an interactive Doom game without having had access to any of the source code]

This achievement marks the first time an AI has fully simulated a complex video game with high-quality graphics and interactivity. Running on a single Tensor Processing Unit (TPU)—Google’s custom-built AI accelerator chip—GameNGen handles Doom’s intricate 3D environments and fast-paced action with remarkable efficiency, all without the usual components of a game engine.

AI game engines: A game-changer for the $200 billion gaming industry​

Doom has long been a technological benchmark since its 1993 release, ported to an astonishing array of platforms—from microwaves to digital cameras. However, GameNGen transcends these earlier adaptations. Unlike traditional game engines that rely on painstakingly coded software to manage game states and render visuals, GameNGen autonomously simulates the entire game environment using an AI-driven generative diffusion model.

The transition from traditional game engines to AI-driven systems like GameNGen could transform the $200 billion global gaming industry. By eliminating the need for manually programmed game logic, AI-powered engines have the potential to significantly reduce both development time and costs. This technological shift could democratize game creation, enabling smaller studios and even individual creators to produce complex, interactive experiences that were previously unimaginable.

Beyond cost and time savings, AI-driven game engines could open the door to entirely new genres of games, where the environment, narrative and gameplay mechanics dynamically evolve based on player actions. This innovation could reshape the gaming landscape, moving the industry away from a blockbuster-centric model toward a more diverse and varied ecosystem.

https://venturebeat.com/wp-content/uploads/2024/08/e1m3.mp4

A video from Google’s “GameNGen,” an AI-powered system that simulates the classic first-person shooter “Doom” without a traditional game engine. The video showcases the neural network’s ability to replicate the game’s iconic visuals, demonstrating the potential for AI to generate complex interactive environments in real time. (Credit: Google)

From video games to autonomous vehicles: Broader implications of AI-driven simulations​

The potential applications for GameNGen extend far beyond gaming. Its capabilities suggest transformative possibilities in industries such as virtual reality, autonomous vehicles and smart cities, where real-time simulations are essential for training, testing and operational management.

For instance, autonomous vehicles require the ability to simulate countless driving scenarios to safely navigate complex environments—a task that an AI-driven engine like GameNGen could perform with high fidelity and real-time processing.

In the realm of virtual and augmented reality, AI-driven engines could create fully immersive, interactive worlds that adapt in real time to user inputs. This could revolutionize sectors like education, healthcare, and remote work, where interactive simulations can provide more effective and engaging experiences.

A schematic diagram showing the flow of data from the game environment through various neural network components, including a denoising network and action embedding, showcasing the complex AI processes involved in generating real-time gameplay without a traditional game engine. (Credit: Google)

The future of gaming: When AI dreams of virtual worlds​

While GameNGen represents a significant leap forward, it also presents challenges. Although it can run Doom at interactive speeds, more graphically intensive modern games would likely require much greater computational power.

Additionally, the current system is tailored to a specific game (i.e. Doom), and developing a more general-purpose AI game engine capable of running multiple titles remains a tough challenge.

Nevertheless, GameNGen is a crucial step toward a new era in game engines—one where games are not just played by AI but also created and powered by it.

As AI continues to advance, we may be on the cusp of a future where our favorite games are born not from lines of code, but from the boundless creativity of machines.

This development also opens up exciting possibilities for game creation and interaction. Future games could adapt in real-time to player actions, generating new content on the fly. AI-powered game engines might also dramatically reduce development time and costs, potentially democratizing game creation.

As we stand on the brink of this new era in gaming, one thing is clear: the lines between human creativity and machine intelligence are blurring, promising a future of digital entertainment we can scarcely imagine. With GameNGen, Google researchers have given us an exciting glimpse of that future—a world where the only limit to our virtual experiences is the imagination of AI.

https://twitter.com/x/status/1828650264263323963

 

[Hamartia Antidote]

 

[j_hungary]

Bad news for EA, Rockstar and Epic......

People don't understand how big this achievement was. The entire purpose of a gaming engine is to set the limit and parameter (or the world in that game) Game Engine is very script heavy, which mean a lot of instruction have to follow a standardised and conform with the engine "rule" to be able to create a world that the program interact with.

With this, basically, this means an "open set" which you can build the world however you like and the AI will then simulate the world as if they are guided by the engine, which mean instead of using a string of parameter to limit the world, you tell the AI what you want to do, and the AI create that world for you, and it is most likely be cross engine........(so they can mimic RAGE or Unreal at the same time) if this is going where I think it is going, that mean an "engine" (if we can still call it that) can enjoy the benefit for both RAGE and UR without any of the draw back....

Of course, you are going to need to have a supercomputer to mimic Doom, and you probably need to have Department of Defence Budget to mimic newer gen Game Engine, but damn, would love to have a look at their diffusion model......

@Nilgiri

 

[Hamartia Antidote]

Bad news for EA, Rockstar and Epic......

People don't understand how big this achievement was. The entire purpose of a gaming engine is to set the limit and parameter (or the world in that game) Game Engine is very script heavy, which mean a lot of instruction have to follow a standardised and conform with the engine "rule" to be able to create a world that the program interact with.

With this, basically, this means an "open set" which you can build the world however you like and the AI will then simulate the world as if they are guided by the engine, which mean instead of using a string of parameter to limit the world, you tell the AI what you want to do, and the AI create that world for you, and it is most likely be cross engine........(so they can mimic RAGE or Unreal at the same time) if this is going where I think it is going, that mean an "engine" (if we can still call it that) can enjoy the benefit for both RAGE and UR without any of the draw back....

Of course, you are going to need to have a supercomputer to mimic Doom, and you probably need to have Department of Defence Budget to mimic newer gen Game Engine, but damn, would love to have a look at their diffusion model......

@Nilgiri

What this also means is you could ask the AI to build 1000 games, play each 100 million times to fix all the bugs, and then in the end you hire some people to look at those 1000 games and pick out the ones that seem would sell the best.

 

[j_hungary]

What this also means is you could ask the AI to build 1000 games, play each 100 million times to fix all the bugs, and then in the end you hire some people to look at those 1000 games and pick out the ones that seem would sell the best.

That gonna need a lot of computer power........

 

[firohot]

That gonna need a lot of computer power........

Like the ones Google Microsoft meta has

Ms is already in gaming business

 

[j_hungary]

Like the ones Google Microsoft meta has

Ms is already in gaming business

don't know how fast can their supercomputer handle. Would imagine you need a DOE level supercomputer to run this more than 20 fps.......That diffusion algorithm is going to be a bitach.......

 

[mulj]

Ok. Star track holo deck is not science fiction anymore.

 

[Nilgiri]

Bad news for EA, Rockstar and Epic......

People don't understand how big this achievement was. The entire purpose of a gaming engine is to set the limit and parameter (or the world in that game) Game Engine is very script heavy, which mean a lot of instruction have to follow a standardised and conform with the engine "rule" to be able to create a world that the program interact with.

With this, basically, this means an "open set" which you can build the world however you like and the AI will then simulate the world as if they are guided by the engine, which mean instead of using a string of parameter to limit the world, you tell the AI what you want to do, and the AI create that world for you, and it is most likely be cross engine........(so they can mimic RAGE or Unreal at the same time) if this is going where I think it is going, that mean an "engine" (if we can still call it that) can enjoy the benefit for both RAGE and UR without any of the draw back....

Of course, you are going to need to have a supercomputer to mimic Doom, and you probably need to have Department of Defence Budget to mimic newer gen Game Engine, but damn, would love to have a look at their diffusion model......

@Nilgiri

logistically i want my ground force units to be organically demand led (fuel, ammo, food, maintenance, reserves etc) rather than discrete supply-allotted + constrained heh....same reasoning.

Just opens up possibilities of what they can do so much more.

Also why GOSPLAN was such a disaster compared to free market operation (and limited focused govt model). Red army inheritor and others ofc have their inertia to this model, it plays out again.

In my line of work, we are currently looking at implementing more adaptive AI in similar way for our simulation efficiency. It is why I am increasingly tasked with creating better "more perfect" random number generator, its more important than it was before (as we optimise out whats best on cloud so mainframe can be focused on its time more) ....as AI has now got to that resolution its a chokepoint in our propietary monte carlo sims and the algorithms its improving there

I wish more quality minded people go into STEM, we are short on talent lol...people think AI will replace humans, actually it augments and releases time for us to develop vertically and laterally and we find we need more humans in end...

If society orients to STEM lot more with time, we got tons of work to get through always....that is optimal area for application of human time....compared to many paper degree sectors lot of "higher education" pushes terribly expensively.

Trades is another thing AI wont replace its hands on. But lot of other (paper degree) sectors will be facing huge job losses temporarily as adjustment to AI (and then developing job sectors for humans to augment to and direct that AI). i.e many human jobs will be gone for good if there's no augmentation need around the AI that is desired+driven by society.

 

[AjayGhatak]

Bad news for EA, Rockstar and Epic......

People don't understand how big this achievement was. The entire purpose of a gaming engine is to set the limit and parameter (or the world in that game) Game Engine is very script heavy, which mean a lot of instruction have to follow a standardised and conform with the engine "rule" to be able to create a world that the program interact with.

With this, basically, this means an "open set" which you can build the world however you like and the AI will then simulate the world as if they are guided by the engine, which mean instead of using a string of parameter to limit the world, you tell the AI what you want to do, and the AI create that world for you, and it is most likely be cross engine........(so they can mimic RAGE or Unreal at the same time) if this is going where I think it is going, that mean an "engine" (if we can still call it that) can enjoy the benefit for both RAGE and UR without any of the draw back....

Of course, you are going to need to have a supercomputer to mimic Doom, and you probably need to have Department of Defence Budget to mimic newer gen Game Engine, but damn, would love to have a look at their diffusion model......

@Nilgiri

The problem in this is same as we see in other places in generative models. It breaks every now and then. See in the video, the enemies when shot fade and turn into something else and so on. And remember, this was built with MASSIVE amount of data generated from the real doom engine.

As far as simulating the world goes, that thing has been around for quite some time now. The scenes and trees in game engines were given fractals. This one will end up learning that from data instead of you providing the fractal rules upfront.

One golden rule for "AI"/ML is not to use it problems for which you have very deterministic solutions. Think about inverse kinematics. You will end up getting noisy data where purely noise free data is easy to obtain. It shows here. The rules of the world in doom engine are determinsitic and mostly noise free. You simulate them from information learnt from data, you get noisy rules. Like being shot may not kill an enemy but transform it into something else.

Imagine if FIFA was built around this manner. The ball being hit fades and turns into an opponent player some rare times.

That being said, one can have games that are inherently noisey. Where you EXPECT the rules to not work properly all the times.

 

[AjayGhatak]

Trades is another thing AI wont replace its hands on. But lot of other (paper degree) sectors will be facing huge job losses temporarily as adjustment to AI (and then developing job sectors for humans to augment to and direct that AI). i.e many human jobs will be gone for good if there's no augmentation need around the AI that is desired+driven by society.

One of my friend is working on a legal ML model. A lot of lawyering is purely paper pushing. Especially in immigration and tax. He says that almost ALL of application etc can be replaced with models.

I know someone who actually got their JR decided in favour using purely chatgpt. They used an older JR and rewrote their own using chatgpt. Lawyer charges 10K for this service at minimum.

 

[j_hungary]

The problem in this is same as we see in other places in generative models. It breaks every now and then. See in the video, the enemies when shot fade and turn into something else and so on. And remember, this was built with MASSIVE amount of data generated from the real doom engine.

As far as simulating the world goes, that thing has been around for quite some time now. The scenes and trees in game engines were given fractals. This one will end up learning that from data instead of you providing the fractal rules upfront.

One golden rule for "AI"/ML is not to use it problems for which you have very deterministic solutions. Think about inverse kinematics. You will end up getting noisy data where purely noise free data is easy to obtain. It shows here. The rules of the world in doom engine are determinsitic and mostly noise free. You simulate them from information learnt from data, you get noisy rules. Like being shot may not kill an enemy but transform it into something else.

Imagine if FIFA was built around this manner. The ball being hit fades and turns into an opponent player some rare times.

That being said, one can have games that are inherently noisey. Where you EXPECT the rules to not work properly all the times.

Well, this is going to be like this with diffusion generative model.

It's like this picture of AI rendering of 6 differnet "Cats with an Assault Rifle"



Reality (in real life or inside game engine) is factual, you know what is a cat looks like and you know what's an assault rifle looks like. It follow some kind of script and you process the data thru that pipeline. It work the same in game and in real world.

AI don't know what a cat is, or what is an assault rifle, to them, this is just noise, and they are going to make a series of prediction of those noise based on probability. While it's a lot easier to define a cat, an assault rifle is a whole other issue, hence this picture.

There are always going to be missing data, because for a game engine, things will tumble and fall when it get shot, because that's what the script dictate. It can be out of the scene, it could be anything. But for an AI, it will totally make sense for something to disappear or turn into some other thing after it was shot, because for them it's just a transition, if the probability is right, it would make sense if it transform into nothing. As there are no definition in them.

 

[AjayGhatak]

Reality (in real life or inside game engine) is factual, you know what is a cat looks like and you know what's an assault rifle looks like. It follow some kind of script and you process the data thru that pipeline. It work the same in game and in real world.

AI don't know what a cat is, or what is an assault rifle, to them, this is just noise, and they are going to make a series of prediction of those noise based on probability. While it's a lot easier to define a cat, an assault rifle is a whole other issue, hence this picture.

At its core, all of these models are trying to learn what I call is "shape of data" or more accurately Probablity Density Function of the data, though most learn to approximate it (say by fooling a descriminator/classifier). You want to generate a shape of .. something, you learn PDFs of some transform of that image. This is fine but it requires a lot of data and can never be really correct because at the end of the day it is stochastic.

Its similar to "predicting artificial noise" without knowing the rule of its random number generator. You can predict the statistics accurately (say Power spectral density function) but actual values are not predictable because underlying rule are unknown.

So yes, what you will get with such a model will feel very very similar to actual noise because they have same power spectral density (for instance) but if the actual values mean something, you will see inconsistencies.

Which is really what is happening here.

I once, in matlab, wrote a ML model to learn and simulate ballistic trajectory of an object by fitting a family of kernel function. It was always jagged and never smooth ballistic trajectory but it was close to what newtonian motion will predict. So yes, if you know newtonian laws (the reality) you can make perfect model very computationally cheaply, but "machine learned one" having noisy output is still having same shape with noise.

The fun thing happens when you combine both deterministic programs with these models.

 

[Nilgiri]

One of my friend is working on a legal ML model. A lot of lawyering is purely paper pushing. Especially in immigration and tax. He says that almost ALL of application etc can be replaced with models.

I know someone who actually got their JR decided in favour using purely chatgpt. They used an older JR and rewrote their own using chatgpt. Lawyer charges 10K for this service at minimum.

The whole education model has to be reformed and changed to adapt to what AI will be doing this century moving forward. Humans are throwing too much resource at redundant sectors for themselves still....and throwing large resource at keeping it afloat afterwards too. I think AI will do its role in vast undercutting and resulting forced reform in the entire system. Wont be overnight, but it will work in tiers slowly.

 

[PakFactor]

logistically i want my ground force units to be organically demand led (fuel, ammo, food, maintenance, reserves etc) rather than discrete supply-allotted + constrained heh....same reasoning.

Just opens up possibilities of what they can do so much more.

Also why GOSPLAN was such a disaster compared to free market operation (and limited focused govt model). Red army inheritor and others ofc have their inertia to this model, it plays out again.

In my line of work, we are currently looking at implementing more adaptive AI in similar way for our simulation efficiency. It is why I am increasingly tasked with creating better "more perfect" random number generator, its more important than it was before (as we optimise out whats best on cloud so mainframe can be focused on its time more) ....as AI has now got to that resolution its a chokepoint in our propietary monte carlo sims and the algorithms its improving there

I wish more quality minded people go into STEM, we are short on talent lol...people think AI will replace humans, actually it augments and releases time for us to develop vertically and laterally and we find we need more humans in end...

If society orients to STEM lot more with time, we got tons of work to get through always....that is optimal area for application of human time....compared to many paper degree sectors lot of "higher education" pushes terribly expensively.

Trades is another thing AI wont replace its hands on. But lot of other (paper degree) sectors will be facing huge job losses temporarily as adjustment to AI (and then developing job sectors for humans to augment to and direct that AI). i.e many human jobs will be gone for good if there's no augmentation need around the AI that is desired+driven by society.

From a business standpoint, I don't think AI will replace the human workforce in its totality, as you pointed out, as it's impossible. However, it will replace the menial jobs we do at the office. I'm dealing with a client on the manufacturing side using AI, and it's a give-and-take sort of situation. However, it will complement the work we are currently doing, carrying out some grunt work.

The insurance industry, which is acting as my premium float for other businesses, is seeing rapid AI adoption, and it's paying us dividends in increased profits and margins from rating (actuarial work), underwriting, loss control, adjusting for claims, auditing, etc.