Unprecedented advances have been made in cancer treatment with the use of immune checkpoint blockade (ICB). However, responses are limited to a subset of patients, and immune-related adverse events (irAEs) can be problematic, requiring treatment discontinuation. Iterative insights into factors intrinsic and extrinsic to the host that impact ICB response and toxicity are critically needed. Our understanding of the impact of host-intrinsic factors (such as the host genome, epigenome, and immunity) has evolved substantially over the past decade, with greater insights on these factors and on tumor and immune co-evolution. Additionally, we are beginning to understand the impact of acute and cumulative exposures—both internal and external to the host (i.e., the exposome)—on host physiology and response to treatment. Together these represent the current day hallmarks of response, resistance, and toxicity to ICB. Opportunities built on these hallmarks are duly warranted. Immune checkpoint blockade has revolutionized cancer therapeutics, but it doesn’t work for everyone. It may cause unacceptable immune-related adverse events, or tumors may fail to respond or develop resistance. We’re beginning to understand the biological reasons why.