Associative learning of social value Behrens, Timothy E. J; Hunt, Laurence T; Woolrich, Mark W ...
Nature,
11/2008, Letnik:
456, Številka:
7219
Journal Article
Recenzirano
Odprti dostop
Our decisions are guided by information learnt from our environment. This information may come via personal experiences of reward, but also from the behaviour of social partners. Social learning is ...widely held to be distinct from other forms of learning in its mechanism and neural implementation; it is often assumed to compete with simpler mechanisms, such as reward-based associative learning, to drive behaviour. Recently, neural signals have been observed during social exchange reminiscent of signals seen in studies of associative learning. Here we demonstrate that social information may be acquired using the same associative processes assumed to underlie reward-based learning. We find that key computational variables for learning in the social and reward domains are processed in a similar fashion, but in parallel neural processing streams. Two neighbouring divisions of the anterior cingulate cortex were central to learning about social and reward-based information, and for determining the extent to which each source of information guides behaviour. When making a decision, however, the information learnt using these parallel streams was combined within ventromedial prefrontal cortex. These findings suggest that human social valuation can be realized by means of the same associative processes previously established for learning other, simpler, features of the environment.
When choosing between two options, correlates of their value are represented in neural activity throughout the brain. Whether these representations reflect activity that is fundamental to the ...computational process of value comparison, as opposed to other computations covarying with value, is unknown. We investigated activity in a biophysically plausible network model that transforms inputs relating to value into categorical choices. A set of characteristic time-varying signals emerged that reflect value comparison. We tested these model predictions using magnetoencephalography data recorded from human subjects performing value-guided decisions. Parietal and prefrontal signals matched closely with model predictions. These results provide a mechanistic explanation of neural signals recorded during value-guided choice and a means of distinguishing computational roles of different cortical regions whose activity covaries with value.
The hippocampal-entorhinal system is important for spatial and relational memory tasks. We formally link these domains, provide a mechanistic understanding of the hippocampal role in generalization, ...and offer unifying principles underlying many entorhinal and hippocampal cell types. We propose medial entorhinal cells form a basis describing structural knowledge, and hippocampal cells link this basis with sensory representations. Adopting these principles, we introduce the Tolman-Eichenbaum machine (TEM). After learning, TEM entorhinal cells display diverse properties resembling apparently bespoke spatial responses, such as grid, band, border, and object-vector cells. TEM hippocampal cells include place and landmark cells that remap between environments. Crucially, TEM also aligns with empirically recorded representations in complex non-spatial tasks. TEM also generates predictions that hippocampal remapping is not random as previously believed; rather, structural knowledge is preserved across environments. We confirm this structural transfer over remapping in simultaneously recorded place and grid cells.
Display omitted
•Common principles for space and relational memory in the hippocampal formation•Explains hippocampal generalization in both spatial and non-spatial problems•Accounts for many reported hippocampal and entorhinal cell types from such tasks•Predicts how hippocampus remaps in both spatial and non-spatial tasks
The Tolman-Eichenbaum Machine, named in honor of Edward Chace Tolman and Howard Eichenbaum for their contributions to cognitive theory, provides a unifying framework for the hippocampal role in spatial and nonspatial generalization and unifying principles underlying many entorhinal and hippocampal cell types.
Damage to prefrontal cortex (PFC) impairs decision-making, but the underlying value computations that might cause such impairments remain unclear. Here we report that value computations are doubly ...dissociable among PFC neurons. Although many PFC neurons encoded chosen value, they used opponent encoding schemes such that averaging the neuronal population extinguished value coding. However, a special population of neurons in anterior cingulate cortex (ACC), but not in orbitofrontal cortex (OFC), multiplexed chosen value across decision parameters using a unified encoding scheme and encoded reward prediction errors. In contrast, neurons in OFC, but not ACC, encoded chosen value relative to the recent history of choice values. Together, these results suggest complementary valuation processes across PFC areas: OFC neurons dynamically evaluate current choices relative to recent choice values, whereas ACC neurons encode choice predictions and prediction errors using a common valuation currency reflecting the integration of multiple decision parameters.
Prior experience is critical for decision-making. It enables explicit representation of potential outcomes and provides training to valuation mechanisms. However, we can also make choices in the ...absence of prior experience by merely imagining the consequences of a new experience. Using functional magnetic resonance imaging repetition suppression in humans, we examined how neuronal representations of novel rewards can be constructed and evaluated. A likely novel experience was constructed by invoking multiple independent memories in hippocampus and medial prefrontal cortex. This construction persisted for only a short time period, during which new associations were observed between the memories for component items. Together, these findings suggest that, in the absence of direct experience, coactivation of multiple relevant memories can provide a training signal to the valuation system that allows the consequences of new experiences to be imagined and acted on.
Reinforcement learning models that focus on the striatum and dopamine can predict the choices of animals and people. Representations of reward expectation and of reward prediction errors that are ...pertinent to decision making, however, are not confined to these regions but are also found in prefrontal and cingulate cortex. Moreover, decisions are not guided solely by the magnitude of the reward that is expected. Uncertainty in the estimate of the reward expectation, the value of information that might be gained by taking a course of action and the cost of an action all influence the manner in which decisions are made through prefrontal and cingulate cortex.
Although the ventromedial prefrontal cortex (vmPFC) has long been implicated in reward-guided decision making, its exact role in this process has remained an unresolved issue. Here we show that, in ...accordance with models of decision making, vmPFC concentrations of GABA and glutamate in human volunteers predict both behavioral performance and the dynamics of a neural value comparison signal. These data provide evidence for a neural competition mechanism in vmPFC that supports value-guided choice.
Valuation is a key tenet of decision neuroscience, where it is generally assumed that different attributes of competing options are assimilated into unitary values. Such values are central to current ...neural models of choice. By contrast, psychological studies emphasize complex interactions between choice and valuation. Principles of neuronal selection also suggest that competitive inhibition may occur in early valuation stages, before option selection. We found that behavior in multi-attribute choice is best explained by a model involving competition at multiple levels of representation. This hierarchical model also explains neural signals in human brain regions previously linked to valuation, including striatum, parietal and prefrontal cortex, where activity represents within-attribute competition, competition between attributes and option selection. This multi-layered inhibition framework challenges the assumption that option values are computed before choice. Instead, our results suggest a canonical competition mechanism throughout all stages of a processing hierarchy, not simply at a final choice stage.
Knowledge abstracted from previous experiences can be transferred to aid new learning. Here, we asked whether such abstract knowledge immediately guides the replay of new experiences. We first ...trained participants on a rule defining an ordering of objects and then presented a novel set of objects in a scrambled order. Across two studies, we observed that representations of these novel objects were reactivated during a subsequent rest. As in rodents, human “replay” events occurred in sequences accelerated in time, compared to actual experience, and reversed their direction after a reward. Notably, replay did not simply recapitulate visual experience, but followed instead a sequence implied by learned abstract knowledge. Furthermore, each replay contained more than sensory representations of the relevant objects. A sensory code of object representations was preceded 50 ms by a code factorized into sequence position and sequence identity. We argue that this factorized representation facilitates the generalization of a previously learned structure to new objects.
Display omitted
•As in rodents, human replay occurs during rest and reverses direction after reward•As in rodents, human replay coincides with hippocampal sharp-wave ripples•Human replay spontaneously reorganizes experience based on learned structure•Human replay is factorized, allowing fast structural generalization
In the human hippocampus, neural activity known as sequential replay includes both item-specific and generalizable abstract representations and spontaneously reorders sequences to integrate past knowledge with current experiences during rest.
Noninvasive human neuroimaging has yielded many discoveries about the brain. Numerous methodological advances have also occurred, though inertia has slowed their adoption. This paper presents an ...integrated approach to data acquisition, analysis and sharing that builds upon recent advances, particularly from the Human Connectome Project (HCP). The 'HCP-style' paradigm has seven core tenets: (i) collect multimodal imaging data from many subjects; (ii) acquire data at high spatial and temporal resolution; (iii) preprocess data to minimize distortions, blurring and temporal artifacts; (iv) represent data using the natural geometry of cortical and subcortical structures; (v) accurately align corresponding brain areas across subjects and studies; (vi) analyze data using neurobiologically accurate brain parcellations; and (vii) share published data via user-friendly databases. We illustrate the HCP-style paradigm using existing HCP data sets and provide guidance for future research. Widespread adoption of this paradigm should accelerate progress in understanding the brain in health and disease.
PDF
