Video games are utilizing artificial intelligence to create increasingly realistic scenarios and interactions, enabled by big increases in processing horsepower and memory, and significantly faster data movement.
GPUs, once confined to graphics rendering, are now also being deployed across a wide range of AI tasks, generating more realistic non-player characters, dynamic worlds, personalized gameplay, as well as level design, content generation, and finely tuned game mechanics. At the same time, these systems are leveraging machine learning tools that perform tasks such as ambient occlusion with less power.
“User interactions with characters in the games used to be very script-based,” said Kristof Beets, vice president of product management at Imagination. “You say this, they say that, and it was all very linear. Now, with AI and all the smarts, you can have proper conversations. Animation also got a lot better with AI. How are the new humanoid robots walking? It’s a neural network. You can apply the same to the physics and the experiences in the game. You can get more dynamics, and more realism by mapping it onto AI, but a lot of that is quite a few years out because the more you map onto AI, the more AI number crunching you need. There’s always a balancing act. There’s definitely a continuous ramp-up on great ideas where you could translate something that was brute force and very expensive, to a fuzzy approximation with a neural network that is good enough, and very convincing.”
Much of this is enabled by the GPUs that are not computing the pixel every time. “Most games do predictive analytics, and they’re basically pre-computing what they’re doing,” said Michal Siwinski, chief marketing officer at Arteris. “That’s a lot of power consumption.”
To counter the high-power consumption of features such as ray tracing, which creates realistic shadows in video games, tools like super resolution are used, which is similar to how an AI network hallucinates answers. “You ask it a question, and something very plausible comes back, but it’s not actually true,” Beets said. “Hallucination is what you don’t want in an AI assistant. You don’t want it to make stuff up. For graphics, it’s almost the opposite. If you think about graphics and filling in detail, that’s exactly what you want the neural network to do. It fills in something that’s plausible. That’s a key thing — rendering lower resolution, not doing all the ray tracing, not doing expensive calculations, and using AI to fill in the gaps.”
For instance, an AI may get things wrong and generate a striped pattern that it’s learned, which the real image may not have. “However, if it looks plausible and if it’s temporarily stable, then it’s good,” said Beets. “One of the hardest things in graphics is temporal stability, so you don’t want it to make up something else on every frame that you see because it would be flickering and changing. The graphics upscaling with AI has been around much longer than the massive boost that we now see in AI search engines and assistants.”
Frame generation is another feature gaining ground. “When gamers play games, we want it to run at 60 FPS,” said Tyrran Ferguson, director of product and strategic partnerships at Imagination. “If it runs at 30 FPS, it may feel like molasses, but it’s still good enough for the human eye. You want it to be at 60 FPS or higher, because then it’s beautiful and smooth. Frame generation interpolates new frames between the real frames. It’s faking frames so that you can go from 30 FPS to 60 FPS, and the hallucination problem can integrate into that. This is a fairly new technology happening on the desktop side. They’re trying to learn how to overcome that and optimize it for games so that you don’t see weird things happening between the frames, like how ray tracing started on the desktop side. It makes gamers comfortable with a feature and an idea, and then they want those features and ideas in the mobile games, or in more difficult places where the GPU is a tiny little block. It’s only a matter of time before we start having to do frame generation on mobile devices.”
Other graphics effects can be achieved with cheaper neural networks, as well. “In the past, a lot of things like depth of field or ambient occlusion were done with very complex shader programs written specifically to do these things,” said Beets. “Now we can teach a much smaller neural network to do the same things with close-enough-quality results, but it’s much cheaper in terms of data flow. This is what NVIDIA calls neural shaders, where you start to teach a neural network.”
Designing chips for gaming tasks
To design GPUs for gaming, engineers need to understand the throughput requirement for the memory at the architectural stage.
“Designers need to know the kind of memory they’re going to use, and the throughput of that memory,” said Matthew Graham, senior group director for verification software product management at Cadence. “We need to make sure the system is designed to fully utilize that throughput. The consumer is spending money to have the fastest memory on their graphics card or in their console. We want to make sure the architecture of the chip is taking advantage of it. Design tools can analyze the very complex algorithms of how we get data from PCI Express to the graphics core, to the memory — whether it’s DDR in the case of graphics, or HBM in the case of AI — and then back into the to the core of the device for the processing, back out to the memory, back to whatever interface, and so on.”
Key to this is making sure the data is functionally correct and coherent from end-to-end, and that it is moving at the appropriate speed. “That’s non-trivial,” said Graham. “It’s not like I put a coin in a bucket, I take the coin out. It’s like I slice the coin into 50 pieces, I put it in 10 different buckets, take them back out of those 10 different buckets, and then make sure I can put that same coin back together. That’s really how these complex systems work. When it comes to a single 4K video frame, which gamers want to have at 120 hertz — 120 times a second — that’s a huge amount of data. So you’re not dealing with all that data in one big chunk. You’re slicing and dicing it and making sure coherency and data integrity are maintained.”
A task like ray tracing can be separated out with its own core within the GPU. “That helps separate the tasks so that when you’re running ray tracing workloads, it goes much quicker through a separate portion of the GPU, so that it’s more efficient,” said Ferguson.
How GPU workloads are assigned depends on the usage case. “Thinking about games, some of it is in order,” said Beets. “First, do the classic graphics rendering, and then you upscale it with AI. That means you go from using nearly 100% of the GPU for doing classic work to wanting to use 100% of the GPU for the AI upscaling. It’s a time-based slicing that you end up doing that fits in naturally with some of the more advanced usage cases, where you get very deep integration. You’re completely mixing up classic and AI-based techniques, and that’s been our focus. How do we make that as efficient as possible? How do we share all that data? How do we keep that data on-chip? The interleaving is where the coolest forward-looking things can be done. You could theoretically try to split the GPU into separate units — with the architecture, you can subdivide and segment some of those approaches — but the most effective way to allocate workloads is to give customers full flexibility. If you want to do 100% AI, you use it for that. If you want to do 100% classic, use it for that.”
Others agree that GPUs are up to both tasks. “A mobile phone SoC can run through your photo library and look for your face, and the next minute it can switch and start running AI for gaming or rendering ray tracing,” said Dave Garrett, vice president of technology and innovation at Synaptics. “The concept of dark silicon plays a big role in that. In the old days, we used to build all these dedicated things to do different tasks, and there were experts in those domains. AI is more about a programmable framework, and the data changes the outcome. But the engine is the same. The mechanisms to train are the same.”
Overall, gaming GPUs require a lot of parallelization and specific, advanced types of instructions. “We help AI accelerator companies optimize how much power they can put into a single GPU chip,” said Daniel Rose, founding engineer at ChipAgents. “When you have more optimal power, when you have less space used, this can help with better gaming chips. There are PPA tradeoffs, depending on the specific chip you’re designing.”
Neural processing units (NPUs) may creep more into gaming to handle specific AI/ML workloads. “We have the GPU, which is extremely flexible and can now deliver a lot more AI performance, but we’ve built in a lot of mechanisms so we can work very closely with the NPU engine,” said Beets. “That usually means putting an amount of SRAM between the NPU and the GPU so they can keep data on-chip and exchange it with each other. That helps with power. It’s still a lot of data movement, but it’s the best you can do if you have two different process units. Latency is really important. You don’t want to go through the whole CPU and all kinds of software stacks to do the job.”
Additionally, custom chips have a lot of issues with access. “It’s very fragmented, and every vendor has a different flavor,” Beets explained. “In gaming, you need an ecosystem. Originally, there were 10 or 12 different vendors in the PC graphics market. They all had different programming models, and they failed. It was a massive shakeout, and there were a couple of big guys that remained because the ecosystem couldn’t sustain more. You couldn’t code for all these different devices. You’ll see the same thing happening with AI. It has to shake out and focus on something, and for gaming, that’s very critical because building a high-end AAA game is very expensive. You can’t afford to replicate that. There’s already a lot of replication, like the PlayStation, Xbox, PC games, and mobile games. A lot of that is merging. You see desktop showing up in mobile. You see mobile guys showing up in desktop. They’re trampling all over each other to grow that ecosystem and that usage case, and AI is a key part of that.”
High-performance gaming on consoles versus handheld devices presents distinct challenges due to differences in hardware, power, and design.
“Consoles prioritize raw power, enabling 4K graphics, high frame rates, and advanced features like ray tracing, but they require robust cooling systems and consume significant power,” said Amol Borkar, director of product management and marketing for Tensilica DSPs in Cadence’s Silicon Solutions Group. “In contrast, handhelds like the Steam Deck or Nintendo Switch must balance performance with portability, facing constraints in battery life, thermal management, and screen size. They often use mobile-optimized chips and dynamic resolution scaling to maintain smooth gameplay.”
Whether mobile or console, power and memory are key. “When it comes to gaming, the challenge is to provide an even more immersive environment for users that increases realism and emotional ties to the gaming world,” said Steven Woo, fellow and distinguished inventor at Rambus. “AI will help to do this, but places increased performance, power, and thermal demands on systems. Memory architectures are critical for good AI and must evolve to support faster access and higher throughput without compromising power budgets.”
XR goggles and puck devices
Gaming is one of the first applications for extended reality (XR), virtual reality (VR), and augmented reality (AR) functions, allowing users to push for more lifelike experiences. As a result, latency is an even bigger concern here than in regular play.
“When you’re doing VR rendering, you’ve effectively got two displays,” said Anand Patel, senior director of product management for Arm’s GPUs in the client line of business. “It might not be two physical displays, but you’re rendering for each eye. You’re doing twice the amount of work you would normally do if you’re rendering to a regular screen. One view will be slightly offset from another to give you a stereoscopic effect. The way the GPU generally handles that is by switching between the two, trying to concurrently render this stuff. You can divide up your GPU into two to sort each panel into two different GPUs. Or you can time slice the GPU so you render one, then move to the other, but do it so quickly it’s transparent to the user.”
The players in the VR space are trying to do different things in different ways. “We’re building very, very small and efficient GPUs, if you do want processing local to the headset,” said Patel. “We’re providing different configurability and configurations, and then our partners can go and innovate.”
Gaming peripherals company Elo powers its XR gaming glasses with an Android processing hub that has RAM storage and a Rock Chip SoC with an Arm CPU and onboard GPU. The glasses feature a Sony OLED 1080p display module in each lens. “It supports well above what we need in terms of resolution, and the processing is way more than what we need for what we’re trying to do,” said Adam Hepburn, founder and CEO of Elo, who noted that Arm CPUs are the reason handheld gaming is taking off. “Previously, the x86 architecture used to take too much energy and wasn’t powerful enough. Now you can play console-level games with a portable device.”
Fig. 1: A gamer using a VR headset and controller. Source: Elo
The notion of the puck device, like Elo’s hub, has been around for some time. “It’s a smartphone without a screen, with more dedicated processing capability,” Arm’s Patel said. “You could have these devices drive VR goggles or AR goggles in a very efficient and low-latency way.”
Coming next is eye tracking. “Your eyes are super, super quick, and you can create it to be extremely accurate with low latency,” said Hepburn. “If you’re playing a game right now, you have to look around a controller or the hub. In the future, you put on the glasses and you can game just by looking around. By that time, the processing would be on board. It would have to be some sort of proprietary, specialized chip.” It also would need to be compatible with multiple devices, through a technology such as Steam’s proton layer.
When pucks are no longer needed to power XR glasses, gamers might use them to carry a personal LLM around in their pocket, like a Tamagotchi digital pet. “You could have a USB-connected dongle or drive, where you are storing your large language model in that drive,” said Gervais Fong, director of product line management for mobile, automotive, and consumer markets at Synopsys. “With the fast connection that USB4 v.2 enables, you can then load the specific model elements that you need into the SoC or into the processing unit and be able to get your generative AI results. That’s a very inexpensive sort of platform where you keep proprietary data local within that area. You don’t have to send it out to the cloud. It keeps it private.”
Agentic AI also will add a new twist to gaming, whether it serves as a teammate or enemy. For example, Intel showed how agentic AI can coach gamers to play better.
Conclusion
The video game industry is growing rapidly, into every corner of the world and every type of device. Chip innovation will continue to enable the latest features demanded by gamers, who want the highest possible fidelity and lowest latency in their experience.
“Gaming is consistently about user experience and increased efficiency of compute, because the more you can get physics and visuals right, the more you can avoid the nausea effect you see in augmented and virtual reality,” said Nandan Nayampally, chief commercial officer at Baya Systems. “Those are the things changing now, and data movement is fundamental to it. What’s really driving all of this is immersive gaming, and that comes down to form factor, which is the stuff going into other things rather than silicon performance. The perfect situation is when any interaction you have becomes more natural and intuitive, rather than mechanical. The fourth wave of augmented reality is where gaming comes to its position. Then you add agentic AI to be your partner or opponent. So there’s plenty of innovation going on for both gaming and agentic AI.”
Related Reading
AR/VR Glasses Taking Shape With New Chips
Smart glasses with augmented reality functions look more natural than VR goggles, but today they are heavily reliant on a phone for compute and next-gen communication.
