Evidence for declining populations of both wild and managed bees has raised concern about a potential global pollination crisis. Strategies to mitigate bee loss generally aim to enhance floral resources. However, we do not really know whether loss of preferred floral resources is the key driver of bee decline because accurate assessment of host plant preferences is difficult, particularly for species that have become rare. Here we examine whether population trends of wild bees in The Netherlands can be explained by trends in host plants, and how this relates to other factors such as climate change. We determined host plant preference of bee species using pollen loads on specimens in entomological collections that were collected before the onset of their decline, and used atlas data to quantify population trends of bee species and their host plants. We show that decline of preferred host plant species was one of two main factors associated with bee decline. Bee body size, the other main factor, was negatively related to population trend, which, because larger bee species have larger pollen requirements than smaller species, may also point toward food limitation as a key factor driving wild bee loss. Diet breadth and other potential factors such as length of flight period or climate change sensitivity were not important in explaining twentieth century bee population trends. These results highlight the species-specific nature of wild bee decline and indicate that mitigation strategies will only be effective if they target the specific host plants of declining species. Significance Growing concern about bee declines and associated loss of pollination services has increased the urgency to identify the underlying causes. So far, the identification of the key drivers of decline of bee populations has largely been based on speculation. We assessed the relative importance of a range of proposed factors responsible for wild bee decline and show that loss of preferred host plant species is one of the main factors associated with the decline of bee populations in The Netherlands. Interestingly, species foraging on crop plant families have stable or increasing populations. These results indicate that mitigation strategies for loss of wild bees will only be effective if they target the specific host plants of declining bee species.