•Negative retention occurs due to intermolecular interactions in the feed solutions.•Presence of gluconic acid induces negative retention of acetic acid.•Formation of salt hydration shells lead to negative HMF retention.•Results obtained for binary mixtures can be applied to complex mixtures. Separation of sugars and aldonic acids from aromatic compounds and short-chain organic acids is a crucial issue in various biorefinery concepts based on wood or agricultural byproducts, which can be resolved by nanofiltration. Model solutions containing up to six compounds, representing major components of biomass hydrolysates, were used to investigate interactions between the individual chemical species, with particular emphasis on the negative retention of 5-hydroxymethylfurfural (HMF) and acetic acid (AcOH). Various factors were found to explain the negative retention of these substances in binary and multicomponent mixtures. The presence of gluconic acid led to the negative retention of AcOH due to an additional electric potential. For HMF retention, the hydration shell formed during the solvation of magnesium sulfate was determined as the main influencing factor. For the first time, the obtained results provide a mechanistic explanation for the negative retention of biomass hydrolysates. In addition, a novel methodology based on a revised concept of retention to quantitatively describe such effects in binary mixtures is provided, which can be qualitatively applied to any biomass hydrolysate. Our results are expected to make nanofiltration for the separation of biomass hydrolysates more predictable, thereby facilitating process development.