However, the features learned by DNN are not directly optimized by fitting functional magnetic resonance imaging (fMRI) data but are ultimately optimized to serve a certain high-level vision task (e.g., image recognition) in computer vision. Moreover, there is a hierarchical correspondence between the DNN and the human visual system: the features from the lower level net layers can better predict the activity of the early visual cortex, while the higher level net layers can better predict the later visual cortex. Goal-driven visual encoding models can accurately predict the voxel activity in the ventral and dorsal visual areas based on the visual stimuli features extracted from pre-trained DNN. We find that the hierarchy of representations in the ventral stream has a positive effect on improving the performance of the existing model in high-level visual areas. In addition, the proportion of the best-encoded voxels for different models in V4 and LO show that our proposed models have obvious advantages in prediction accuracy. The experimental results show that two categories of hierarchical encoding models effectively improve the encoding performance in V4 and LO. From the feature perspective, the feature space of the first model is extracted to predict the voxel space of the high-level visual area. Then we use the extracted voxel space of the low-level visual area to predict the voxel space of the high-level visual area (V4 or LO) via constructing a voxel-to-voxel model. From the voxel perspective, we first constructed an encoding model for the low-level visual area (V1 or V2) and extracted the voxel space predicted by the model. Under the framework, we propose two categories of hierarchical encoding models from the voxel and the feature perspectives to realize the hierarchical representations. In the present study, we propose a novel visual encoding model framework which uses the hierarchy of representations in the ventral stream to improve the model’s performance in high-level visual areas.

The ventral stream suggests that higher visual areas receive information from lower visual areas, which is not fully reflected in the current encoding models. However, due to the amount of neural data limitation, DNN-based visual encoding models are difficult to fit for high-level visual areas, resulting in insufficient encoding performance.

Visual encoding models based on deep neural networks (DNN) show good performance in predicting brain activity in low-level visual areas.

Visual encoding in psychology