Reading minds with fMRI and machine learning.

Abstract

A major challenge in brain decoding is to reconstruct internally generated visual representations—mental images—from human brain activity. While the Natural Scenes Dataset (NSD) has enabled unprecedented improvements in reconstructing \textit{seen} images from fMRI signals, methods training on it have primarily been limited to decoding only externally presented stimuli. In an analysis of the new NSD-Imagery dataset, I demonstrate that while some modern NSD-trained vision decoders can generalize quite well in reconstructing mental images, some fail, and that state-of-the-art (SOTA) performance on seen image reconstruction is no guarantee of good performance on mental image reconstruction. Motivated by these findings, I developed MIRAGE, a novel method explicitly designed to train on vision datasets and cross-decode mental images from brain activity. MIRAGE employs a simple and robust ridge regression backbone, maps to multi-modal text and image features, and adopts the Stable Cascade diffusion model which accepts multi-modal conditioning and small image embeddings as input. Evaluations on the NSD-Imagery benchmark—supported by human ratings and feature-based metrics—establish MIRAGE as the SOTA method for producing mental image reconstructions. This result indicates that--given the right architecture--existing large-scale datasets using external stimuli are viable training data for decoding mental images, yielding new computationally-driven insights into how mental images are represented in the brain, and offering concrete paths forward for more accurate and flexible mental imagery decoding.