Video games in MRI scanners reveal how brains predict and adapt in real time

For the better part of a century, neuroscience has operated under a constraint that seemed almost necessary: to understand the brain, you put a person in a scanner, told them to hold very still, and gave them a single, carefully designed task. Press a button when you see a face. Remember these words. Look at this image. The method worked. It revealed genuine truths about how neural tissue organizes itself, how regions communicate, how damage manifests. But it also left out something that happens every waking moment of actual life.

Real cognition is not a series of isolated moments. It is a continuous stream of prediction, action, and adjustment. You see a threat and move. The world responds. You update your understanding and move again. This feedback loop—this constant conversation between mind and environment—has been nearly impossible to study inside a scanner, where the whole point is to eliminate variables and control the experiment down to its smallest detail.

Jörn Diedrichsen and Jody Culham, neuroscientists at Western University, have spent years thinking about how to change that. Diedrichsen developed methods to map brain activity across dozens of different tasks rather than relying on single experiments or resting-state scans where the brain simply does what it does without guidance. Culham's lab studied how brains respond when people actively shape their environment—comparing someone playing Pac-Man to someone watching the game or moving a controller that does nothing. They noticed something: the sensory input and physical movement might be identical, but the player's relationship to the world was fundamentally different. The player was in a closed loop with the environment. The observer was not.

Now they are leading an international research effort, supported by Meta, to merge these two approaches by putting people inside an fMRI scanner and having them play video games. The project is called the Digital Brain Project, and its ambition is large: to create a massive, openly shared dataset of human brain activity that captures how the brain behaves across a much wider range of cognitive states than conventional imaging has ever attempted. Instead of asking a participant to perform one isolated task, researchers will have them play games built around rich three-dimensional environments and first-person perspectives. A core set of tasks—kart racing, action games, crafting games, digital puzzles—will anchor the data for comparison, while new elements will be introduced throughout the project.

To make this work, Western is investing in a 7-tesla MRI scanner, a system far more powerful than conventional machines, housed at the Center for Functional and Metabolic Mapping. The researchers are also developing a new 32-channel head coil designed specifically to capture activity from deep brain structures: the cerebellum, thalamus, and midbrain. This matters particularly to Diedrichsen, whose research focuses on the cerebellum and how it connects to the cortex. For decades, neuroscience thought of the cerebellum as purely a movement and coordination center. It is increasingly clear that it also handles prediction and cognition—exactly the functions that emerge when someone plays a game and must constantly anticipate what comes next.

The upgraded scanner will also have a much larger visual display, allowing researchers to create more immersive environments. The goal is to eventually use 3D displays that make a participant feel as though they are inside the game world, despite lying physically confined inside the machine. When someone plays Pac-Man in this setup, they are not simply reacting to stimuli. They are constantly predicting where the ghost will go, changing direction, moving the controller, watching Pac-Man respond, and immediately updating their plan. It is a continuous loop fundamentally different from looking at an image and pressing a button.

One of the project's larger ambitions reaches beyond the idea of the average brain. Individual brains differ considerably, especially in regions involved in complex actions and cognition. Diedrichsen wants to collect rich data from individual participants across many different activities, mapping how each person's brain is uniquely organized. When researchers average many brains together to create a composite image, they lose the detail—like stacking photographs of trees on top of each other and getting a blur instead of understanding what a tree actually looks like. The goal is to create a large, high-quality dataset that researchers worldwide can access for noncommercial work, asking questions and applying methods that have not yet been imagined.

Artificial intelligence will play a role too. Researchers are developing AI systems that can learn cognitive tasks and play simple games, then examining how those systems learn to navigate, predict, and decide. By studying artificial minds alongside biological ones, scientists can explore possible solutions to problems that brains face, using AI as what Diedrichsen calls an intuition pump for thinking about how complex systems work. The researchers are not suggesting that traditional brain imaging should disappear. Instead, they see the video-game approach as opening a part of the brain's functional landscape that has been difficult to reach with experiments built around isolated questions and tightly controlled responses. They are trying to study the brain as it actually works—engaged with the world, predicting, acting, adapting.