Immune checkpoint inhibitors (ICIs) have provided tremendous clinical benefit in several cancer types. However, systemic activation of the immune system also leads to several immune-related adverse events. Of these, ICI-mediated colitis (IMC) occurs frequently and is the one with the highest absolute fatality. To improve current treatment strategies, it is important to understand the cellular mechanisms that induce this form of colitis. In this review, we discuss important pathways that are altered in IMC in mouse models and in human colon biopsy samples. This reveals a complex interplay between several types of immune cells and the gut microbiome. In addition to a mechanistic understanding, patients at risk should be identifiable before ICI therapy. Here we propose to focus on T-cell subsets that interact with bacteria after inducing epithelial damage. Especially, intestinal resident immune cells are of interest. This may lead to a better understanding of IMC and provides opportunities for prevention and management.