Microsoft want to record your face and feed it to a generative AI, telling it to go off and design a game from your frown

A new patent from Microsoft shows how they want to record your facial expressions, show them to an AI, and have it redesign the game you’re playing. It’s not only your face they want, but also any words you might be muttering, growls you might be making, or, dare I say, even purrs.

The broader technology described in the patent is a process for game designers to prompt an AI to build out the narrative elements of a game, including objectives, sidequests, NPCs, dialogue, and items. Rather than stop at that first pass, the process would continue when players are in the game, analysing how they’re playing to then generate new content based on their actions and reactions.

A quick peek behind the curtain: I, too, am using this technology and the rest of this article has been rewritten to match exactly the facial expression you are making right… now.

US patent 20260183672A1, pithily titled Generative Narrative Game Experience With Player Feedback, doesn’t describe a technology Microsoft have already made – it is more a case of describing a process of how this tech would work if they were to make it. As ever with patent documents, they are as much a case of laying the groundwork to say “I thought of that first” to a judge when one of your competitors does a thing, as much as they are something you actually intend to do. They also often have the quality of talking to a child describing the spaceship they would make if they were given half the chance – “…and its legs have pipes filled with jelly, so when it lands it can turn the ground into trampolines”. As such, you can’t read a patent document as the company has done a thing so much as the company has thought about doing a thing.

Anyway, onto the thing Microsoft wants everyone to know they have thought about.

The patent describes a few ways the process would work. In one, it’s predominantly on the dev side. The patent suggests that a designer could prompt an AI, using written input, images, and audio files, and the engine would then create characters, narratives, and worlds “at a high level” that the designer would finesse. That finessing would either be by directly editing what the model created or by giving it more prompts to refine what was in front of them. Once the designer was happy with what they had made, they could implement it within a game.

The authors suggest the prompt could be “merely a general theme, idea, or goal for the game or characterize the general nature of the game, or it may include some narrative details, such as main characters, a basic storyline, etc.” The model would then go on to create game content – NPCs, items, dialogue, lore descriptions, and objectives and sidequests. This would then be given back to the designer as “a visual representation of the game […] in the form of text, images, and/or graphs”.

Personally, I find the vagueness here delightful. All I can picture is the creators of Deep Thought, the massive supercomputer in the Hitchhiker’s Guide to the Galaxy, prompting the machine for the answer to the ultimate question: Life! The Universe! Everything! And receiving the answer 42. I can imagine a designer sitting at their desktop, writing ‘Computer, build me a game about the theme of grief’, only to be presented with a flowchart ‘Main character loves person > loved person dies > Main character = sad”. Thank you, computer.

The patent does go on to say that along with the visual representation, the model may also provide code you can stick into a game engine. Alternatively, they say an already existing game, such as Minecraft, could be updated with new content via an api. Well, that all sounds marvellously simple.

Another way the tech would work is for that design role to be taken up by the player – though, not through direct prompting, as with the designer, but with an AI adapting the world based on the player’s actions and “engagement metrics”. In this example, it’s not something that happens on the fly as you’re playing, but either while the game is still in development or between release builds. So, for instance, the players would be QA testers, beta testers, or early access players. Data would be collected from their play and fed into the model and it would spit out content for a designer to work with before officially incorporating it into the game.

Though, they also then go on to describe interactions which are very much on-the-fly calls to a generative AI, similar to the conversations we’ve seen players have with NPCs in games like Where Winds Meet. The patent’s authors say that the model would take what a player asked an NPC, along with contextual information such as where they are in the game, what they’ve been talking about with other NPCs, and the story path they’ve taken in the game, and stuff that into a generative AI to come up with a response. They excitedly write, “the possible natural-language output of the conversational agent are virtually limitless.”

We have spoken to narrative directors in the past and they are less convinced…

Though Microsoft’s engineers aren’t just describing an NPC that spouts newly generated dialogue. They suggest taking the whole thing a little further. “an agent associated with an NPC may create not only conversational output (e.g., suggesting a side quest for the player to pursue), but also programmatic output that creates new game assets, rules, or alterations of the game world associated with the conversational output (e.g., objects or locations associated with the suggested side quest).” A new goal for me in any Microsoft game that allows me to speak directly to NPCs is going to be a two-hour attempt to convince them to turn off the world’s gravity.

Outside of NPCs, the authors suggest that generative agents could operate other objects in the game. They use the example of a box that, instead of being unlocked with a key, is kicked open by the player. “In some instances, the new content can be so involved as to affect even the game narrative, creating new narrative elements, paths through the game, and modes of gameplay. In multi-player games, in particular, the collective interactions of players with each other and with the game can result in complex emergent new game behaviors.”

Perhaps one day, I will kick open a box, a nearby NPC will begin to weep because the box was gift to them from their child. They will generate a sidequest where I go and apologise to their child and, as a reward for my efforts, they will turn the gravity off.

I really have to stress again at this point that there is no evidence that Microsoft have made any of this. They’re not patenting a technology that exists in the world. This is basically a big ‘Wouldn’t it be cool off tech could do this? And, if anyone else does it first, we get the right to sue them for stealing our neat ideas wot we wrote down first.’

Other data Microsoft say they could scrape from a player’s game is how often they return to an area, how long they spend talking to an NPC, whether they complete or abandon a quest. This isn’t much different from how human game developers work, they also monitor how players move through their worlds and then make changes based off what they see. For instance, in Unreal Engine, you can collate movement data from tonnes of playtests and generate a heatmap to see where most players spent their time. It’s just, at the other end of that, a human makes the decision of how to change the game.

The patent starts to get decidedly creepy when Microsoft suggests that “with permission” they could use cameras and microphones to “infer such indicators from the player’s spoken comments, facial expressions, etc. during gameplay”. They also suggest processing “live streams posted online” and feeding that into the AI model to make changes. This does raise the prospect that you could tell a game to delete itself and it would actually go off and do just that.

As you would expect, the dream of the people authoring this patent is that the game-designing AI would make more of what players like and less of what they didn’t. “If a particular part of the game tends to attract players, the model may decide to elaborate on and extend that part of the game to further enhance player engagement. On the other end of the spectrum, game content that is less engaging, whether part of the original game or added on the fly, may be pruned or assigned lower weight.”

Put another way, the AI might see the shocked and saddened faces of players during Aerith’s death in Final Fantasy 7 and decide to delete the sequence, replacing it with something that makes them visibly happier. Or, recognising the emotional impact of the character’s demise, opt to extend it for maximum effect, creating the All Too Well (10 Minute Version) of her final moments.

If you’re now decidedly worried about what this AI generated future of video games looks like, there is some solace in the patent document. The authors provide an example of what an AI might suggest to a designer making a Wizard of Oz game. One suggested quest is “Complete the Wizard’s Tasks: Once Dorothy reaches the Wizard’s castle, they will be given a set of tasks to complete before the Wizard will grant them an audience. These tasks involve finding and collecting various items from around the land of Oz.” That’s hardly earth-shattering stuff. With insights like that, Lies of P developer Neowiz, who are currently hard at work at Oz-set Wonders of O, must be quaking in the boots.

So, in summary, Microsoft haven’t made a thing, but they’ve stated how they would make a thing, and if they do make that thing, then you may want to start wearing a mask when playing Microsoft games out of fear that should you pick your nose while play the game may add a section where you go digging for gold in a cave full of bats.