•Particle size distributions of nucleation mode change remarkably along exhaust system.•Particle number concentration reduces more significantly with higher accumulation mode.•The constructed coagulation model can well catch the measured particle variation.•The coagulation history was provided to explain the particle evolution. In this article, Brownian coagulation of particles in the exhaust system of a gasoline engine were investigated by measuring particle size distribution (PSD) at different sampling points and Monte Carlo simulation. The measurement results show that the particle size distributions of nucleation mode (NM) change remarkably after the exhaust gas passes through the exhaust plenum chamber. The constructed coagulation model based on the classic coagulation theory can well catch the measured PSDs, particle number (PN) concentrations, and count median diameters (CMDs) along the exhaust system under both the in-cylinder direct injection and port fuel injection modes, indicating that the particle process in the exhaust system is dominated by coagulation. The NM PN concentration decreases, and the NM CMD increases in the downstream exhaust gas, and the changes are more significant at higher accumulation mode (AM) PN concentration conditions. The coagulation history provided by modelling shows that coagulation mainly occurs between two NM particles (NM-NM) or one NM and one AM particles (NM-AM). At higher AM PN concentration conditions, the NM-AM coagulation dominates and quickly eliminates NM particles, resulting in more significant NM particle reduction. The PN concentration variation as a function of time indicates that PN concentration decreases quickly at first and then the decreasing rate slows down.