Animals require the ability to ignore sensory stimuli that have no consequence yet respond to the same stimuli when they become useful. However, the brain circuits that govern this flexibility in sensory processing are not well understood. Here we show in mouse primary auditory cortex (A1) that daily passive sound exposure causes a long-lasting reduction in representations of the experienced sound by layer 2/3 pyramidal cells. This habituation arises locally in A1 and involves an enhancement in inhibition and selective upregulation in the activity of somatostatin-expressing inhibitory neurons (SOM cells). Furthermore, when mice engage in sound-guided behavior, pyramidal cell excitatory responses to habituated sounds are enhanced, whereas SOM cell responses are diminished. Together, our results demonstrate the bidirectional modulation of A1 sensory representations and suggest that SOM cells gate cortical information flow based on the behavioral relevance of the stimulus. •Passive sound experience causes habituation of sensory representations in A1•Habituation involves an increase in inhibition of layer 2/3 pyramidal cells•Habituation reflects the selective upregulation of SOM interneuron activity•Sound-guided behavior decreases SOM cell activity and rapidly reverses habituation Kato et al. demonstrate that daily sound experience upregulates SOM interneuron activity in A1 and causes “habituation” of sound representations. Sound-guided behavior reverses these effects, indicating that sensory representations are bidirectionally modified based on the behavioral relevance of sensory stimuli.