Much research on Brexit has studied whether the vote to leave the EU is a marker of growing prejudice. In this article, we study instead how the relationship between support for Brexit and prejudice is constructed, negotiated and contested in lay discourse. Our analysis of focus groups (N = 12) conducted prior to and following the EU referendum shows that support for Brexit was predominantly constructed as based on nationalism and anti‐immigration prejudice, especially in Remain‐supporting accounts. This prompted identity management strategies and counter‐arguments by Leave supporters, such as providing alternative constructions of prejudice and racism; relating Brexit with “progressive” values; and presenting it as rational and economically sound. In our discussion, we draw attention to the “ideological creativity” that underpins these accounts and also reflect on the possibilities and constraints of developing alternative political narratives under conditions of political polarisation.