Attention filters sensory inputs to enhance task-relevant information. It is guided by an “attentional template” that represents the stimulus features that are currently relevant. To understand how the brain learns and uses templates, we trained monkeys to perform a visual search task that required them to repeatedly learn new attentional templates. Neural recordings found that templates were represented across the prefrontal and parietal cortex in a structured manner, such that perceptually neighboring templates had similar neural representations. When the task changed, a new attentional template was learned by incrementally shifting the template toward rewarded features. Finally, we found that attentional templates transformed stimulus features into a common value representation that allowed the same decision-making mechanisms to deploy attention, regardless of the identity of the template. Altogether, our results provide insight into the neural mechanisms by which the brain learns to control attention and how attention can be flexibly deployed across tasks. Display omitted •Monkeys repeatedly learned new attentional templates to select visual stimuli•Neural representations of templates were structured in the prefrontal and parietal cortex•Attentional templates were iteratively updated after each trial based on reward feedback•Sensory inputs were translated into value, allowing attention to generalize across tasks Jahn et al. show that the brain uses reward feedback to learn attentional templates that select task-relevant stimuli. Attended stimulus features were converted into a common value representation, allowing attentional control to generalize across tasks.