Population intelligence quotients increased throughout the 20th century—a phenomenon known as the Flynn effect—although recent years have seen a slowdown or reversal of this trend in several countries. To distinguish between the large set of proposed explanations, we categorize hypothesized causal factors by whether they accommodate the existence of within-family Flynn effects. Using administrative register data and cognitive ability scores from military conscription data covering three decades of Norwegian birth cohorts (1962–1991), we show that the observed Flynn effect, its turning point, and subsequent decline can all be fully recovered from within-family variation. The analysis controls for all factors shared by siblings and finds no evidence for prominent causal hypotheses of the decline implicating genes and environmental factors that vary between, but not within, families.