Attributing the similarity between individuals to genetic and non-genetic factors is central to genetic analyses. In this paper we use the genomic relationship (Formula: see text) among 417,060 individuals to investigate the phenotypic covariance between pairs of individuals for 32 traits across the spectrum of relatedness, from unrelated pairs through to identical twins. We find linear relationships between phenotypic covariance and Formula: see text that agree with the SNP-based heritability (Formula: see text) in unrelated pairs (Formula: see text), and with pedigree-estimated heritability in close relatives (Formula: see text). The covariance increases faster than Formula: see text in distant relatives (Formula: see text), and we attribute this to imperfect linkage disequilibrium between causal variants and the common variants used to construct Formula: see text. We also examine the effect of assortative mating on heritability estimates from different experimental designs. We find that full-sib identity-by-descent regression estimates for height (0.66 s.e. 0.07) are consistent with estimates from close relatives (0.82 s.e. 0.04) after accounting for the effect of assortative mating.