AI video game guides are not reliable reveals new study by indie developer

AI is proving an unsuitable substitute for video game guides and one indie developer has highlighted exactly why it doesn’t work.

If you wanted another reason to be wary of AI (or rather large language models, which is all the are) there’s the increasing likelihood of it rendering video game guides redundant and putting guide writers out of work.

Between Microsoft implementing its Copilot software into Minecraft and Google’s own AI overviews, people can now easily find tips and tricks for when they get stuck, so there’s no need to visit dedicated websites… at least in theory.

It has repeatedly been demonstrated that because these AI systems scour and regurgitate information (when they’re not being completely unhinged) they often get things completely wrong and yet state it as unequivocal fact. It’s reportedly got so bad that one indie studio has suggested game developers need to start implementing better tutorials as a countermeasure.

Chances are you’ve already encountered something like this, whether it’s related to video games or not. You’ll be searching for information online, only for Google’s AI overview to give you an answer that you already know is incorrect, like saying Yakuza 0 Director’s Cut, which launched on Switch 2 in June, doesn’t actually exist.

UK indie studio Spilt Milk has decried such AI overviews as ‘frustrating’ while speaking with Game File’s Stephen Totilo, as Google’s Gemini AI has been spreading misinformation about the studio’s management game Trash Goblin since its launch on PC this past May.

Creative director Andrew Smith and programmer Simon Roth have performed rigorous testing of Gemini to see if it can accurately answer questions about their game and, so far, the answer appears to mostly be ‘no.’

For instance, during a test on August 19, the AI stated that trinkets could be damaged during the game’s chiselling phase, which is patently incorrect since while there is a chiselling phase, it’s impossible for trinkets to be damaged.

When we asked it the same question ourselves today, it did answer correctly but there’s no telling when the AI started offering the right answer or whether it will continue to do so consistently.

Rigorous testing saw the AI repeatedly give incorrect information, in some cases drawing it from unrelated sources for different games.

So, what can be done about this? Roth thinks one solution is simply coming up with more in-depth tutorials so players won’t feel like they need to rely on Google, which could be something more developers adopt if such issues become even more widespread.

The team also plans to start checking whether players seeking help have consulted an AI or not since they could wind up trying to look into issues that don’t exist.

Smith, meanwhile, is frustrated that he’s unable to offer any input on Gemini’s answers, but ‘then they have to admit that AI is not able to learn just by scraping – it has to involve a person who’s an expert, and that’s kind of the crux of it, isn’t it?’

This is the real meat of the problem regarding AI serving as a games guide. It doesn’t play the game itself; it simply scrapes the internet and repeats what it finds, meaning it’s very easy to mix stuff up and present incorrect info.

And if guides writers lose work because AI usage becomes more common place, that’s fewer reliable guides to draw from, which creates this awful ouroboros where the AI becomes more and more inaccurate because it’s putting more and more humans out of work.

Email gamecentral@metro.co.uk, leave a comment below, follow us on Twitter.

To submit Inbox letters and Reader’s Features more easily, without the need to send an email, just use our Submit Stuff page here.

For more stories like this, check our Gaming page.

MORE: Next gen Xbox will have special hardware just for AI reveals Microsoft

MORE: Candy Crush developers replaced by the AI tools they helped make

MORE: Horizon Forbidden West voice actor says new AI tech usage ‘scares me’