•Numerical and syntactical text variations generate different answers distributions.•Statistically significant results confirmed solid findings in mathematics education.•Interesting new quantitative phenomena emerged concerning a classical topic.•Different impacts of the variations on different categories of students emerged.•The same variation affected differently males and females in a significant way. In this paper, we present a methodological approach to the investigation of the impacts of text formulation on students’ answers in mathematical problem solving-based assessment. After a review of the related literature in Mathematics education and a review of the methodologies used until now to investigate this research issue, we describe in depth our quantitative approach, providing motivations and examples of its statistical relevance and its potentiality in making interesting phenomena emerge, to be interpreted with further qualitative methods. We observed statistically significant evidences of different impacts of the variations on different categories of students (males/females; students with high and low performances in the whole test). The methodology and our preliminary results can inform researchers in mathematics education, teachers and experts in the agencies that are responsible for large-scale students learning assessment in several contexts (national and international).