Evidence of compression and probabilistic representation during mental imagery

Abstract

A single kite floating across the sky is easier to imagine than simultaneously imagining hundreds of kites flying independently. So why are some scenes harder to imagine than others? As the human imagination is heavily dependent on the amount of information remembered to successfully generate accurate metal images, complex scenes can be much harder to re-imagine than simpler ones. Here we try to characterize what makes an image hard to imagine by considering several hypotheses: the total number of objects one has to imagine, whether the objects are scattered across the scene, or whether the image can be easily compressed via some computational strategy. We probed human subjects’ imagination of simple, multi-object test images using a same-different task and collected behavior data. We built several models based on perceptual factors and found that the response error rate was more strongly correlated with how fragmented the objects were than the total number of objects the test images contained. We also observed that a model that assumed a compressed representation of these images coupled with information capacity threshold predicted these response errors very well suggesting that compressive effects might be at play during imagery just like in vision. To observe how scenes of different complexities are represented during imagery, we utilized a generative AI model to reconstruct segmentation maps of subjects’ imagination from behavior data. We observed that as scenes get more complex with the addition of newer and fragmented objects, the representation becomes far more volatile compared to a simpler scene. We believe that this can be due to a switch in the imagination strategy: for simple images, where reporting errors are minimal, imagery is generated in a deterministic fashion while for complex images subjects may represent the images in a more probabilistic manner that causes them to be more unstable and drives up the reporting errors.