Chlorogenic and isochlorogenic acids are naturally occurring antioxidant dietary polyphenolic compounds found in high concentrations in plants, fruits, vegetables, coffee, and coffee by-products. The objective of this review was to assess the potential health risks associated with the oral consumption of coffee by-products containing chlorogenic and isochlorogenic acids, considering both acute and chronic exposure. An electronic literature search was conducted, revealing that 5-caffeoylquinic acid (5-CQA) and 3,5-dicaffeoylquinic acid (3,5-DCQA) are the major chlorogenic acids found in coffee by-products. Toxicological, pharmacokinetic, and clinical data from animal and human studies were available for the assessment, which indicated no significant evidence of toxic or adverse effects following acute oral exposure. The current state of knowledge suggests that long-term exposure to chlorogenic and isochlorogenic acids by daily consumption does not appear to pose a risk to human health when observed at doses within the normal range of dietary exposure. As a result, the intake of CQAs from coffee by-products can be considered reasonably safe.