In this work, we investigate how the central stellar metallicity (Z/H) of 1363 galaxies from the SAMI galaxy survey is related to their stellar mass and a proxy for the gravitational potential, \(\Phi\) = log10(M/M*) - log10(\(r_e\)/kpc). In agreement with previous studies, we find that passive and star-forming galaxies occupy different areas of the Z/H-M* plane, with passive galaxies having higher Z/H than star-forming galaxies at fixed mass (a difference of 0.23 dex at log10(M/M*)=10.3). We show for the first time that all galaxies lie on the same relation between Z/H and \(\Phi\), and show that the offset in Z/H between passive and star-forming galaxies at fixed \(\Phi\) is smaller than or equal to the offset in Z/H at fixed mass (an average \(\Delta\)Z/H of 0.11 dex at fixed \(\Phi\) compared to 0.21 dex at fixed mass). We then build a simple model of galaxy evolution to explain and understand our results. By assuming that Z/H traces \(\Phi\) over cosmic time and that the probability that a galaxy quenches depends on both its mass and size, we are able to reproduce these offsets in stellar metallicity with a model containing instantaneous quenching. We therefore conclude that an offset in metallicity at fixed mass cannot by itself be used as evidence of slow quenching processes, in contrast to previous studies. Instead, our model implies that metal-rich galaxies have always been the smallest objects for their mass in a population. Our findings reiterate the need to consider galaxy size when studying stellar populations.