Computational Exploration of Higher Visual Selectivity in the Human Brain

  A fundamental goal of cognitive neuroscience has been understanding how the  human visual cortex supports perceiving and interpreting visual information in the  world around us. Traditional approaches to mapping the visual cortex have relied  on manually assembled stimulus sets, often employing isolated objects in artificial  contexts with simplified backgrounds. These approaches do not fully capture the  complexity and richness of real-world visual experience, potentially biasing results  and limiting our understanding of visual processing. My thesis introduces a suite  of computational approaches leveraging naturalistic image stimuli to identify and  characterize the high-level organization of visual information in the human brain.  Specifically, I present:  

1. Brain Diffusion for Visual Exploration (BrainDiVE): A method utilizing gradient guidance from a differentiable image-to-fMRI encoder and a pre-trained  image diffusion model to generate naturalistic “most-exciting-inputs” that maximally activate specific brain regions.  
2.  Semantic Captioning Using Brain Alignments (BrainSCUBA): A technique  unifying the embedding spaces of CLIP image and text embeddings with fMRI  encoder weights to drive a vision-language model. This enables the generation  of natural language descriptions of voxel-wise selectivity within the visual  cortex.  
3.  Semantic Attribution and Image Localization (BrainSAIL): An approach employing vision foundation models and dense semantic features to localize activating objects within complex naturalistic images across higher-level visual  areas.  

These computational methods are complemented by human validation experiments  using synthetically generated stimuli. Overall, my thesis work demonstrates the power of combining naturalistic stimuli with advanced computational techniques to reveal  the fine-grained organization of the human visual cortex. In addition to providing  a detailed overview of the computational models I have developed, I outline future  computational and fMRI experiments designed to further validate and extend these  findings. My research paves the way towards a more comprehensive and ecologically  valid understanding of visual processing, with implications for building more accurate  models of the brain and contributing to novel applications in artificial intelligence.