Additive genetic variance (VA) reflects the potential for evolutionary shifts and can be low for some traits or populations. High VA is critical for the conservation of threatened species under selection to facilitate adaptation. Theory predicts tight associations between population size and VA, but data from some experimental models, and managed and natural populations do not always support this prediction. However, VA comparisons often have low statistical power, are undertaken in highly controlled environments distinct from natural habitats, and focus on traits with limited ecological relevance. Moreover, investigations of VA typically fail to consider rare alleles, genetic load, or linkage disequilibrium, resulting in deleterious effects associated with favored alleles in small populations. Large population size remains essential for ensuring adaptation. The importance of small population size in limiting adaptive potential is reinforced. Estimates of genetic variance across populations tend to be imprecise. Genomic data highlight linkage, load, and rare alleles in population size issues. Long-term conservation maintaining adaptive potential requires large populations.