Lineage-tracing approaches, widely used to characterize stem cell populations, rely on the specificity and stability of individual markers for accurate results. We present a method in which genetic labeling in the intestinal epithelium is acquired as a mutation-induced clonal mark during DNA replication. By determining the rate of mutation in vivo and combining this data with the known neutral-drift dynamics that describe intestinal stem cell replacement, we quantify the number of functional stem cells in crypts and adenomas. Contrary to previous reports, we find that significantly lower numbers of “working” stem cells are present in the intestinal epithelium (five to seven per crypt) and in adenomas (nine per gland), and that those stem cells are also replaced at a significantly lower rate. These findings suggest that the bulk of tumor stem cell divisions serve only to replace stem cell loss, with rare clonal victors driving gland repopulation and tumor growth. •Replication-dependent clonal label applied to infer stem cell dynamics•Stem cell numbers and replacement rates are lower than previous estimates•Stem cell dynamics are unchanging with age•Low stem cell numbers but high replacement rates follow oncogenic transformation A replication-dependent clonal-label approach reveals significantly lower than expected numbers of “working” stem cells in the intestinal epithelium and in adenomas and slower rates of stem cell replacement.