We investigate the quantum phase transition of the anisotropic quantum Rabi model, in which the rotating and counterrotating terms are allowed to have different coupling strengths. The model interpolates between two known limits with distinct universal properties. Through a combination of analytic and numerical approaches, we extract the phase diagram, scaling functions, and critical exponents, which determine the universality class at finite anisotropy (identical to the isotropic limit). We also reveal other interesting features, including a superradiance-induced freezing of the effective mass and discontinuous scaling functions in the Jaynes-Cummings limit. Our findings are extended to the few-body quantum phase transitions with N>1 spins, where we expose the same effective parameters, scaling properties, and phase diagram. Thus, a stronger form of universality is established, valid from N=1 up to the thermodynamic limit.