We study the Fundamental Plane (FP) for a volume- and luminosity-limited sample of 560 early-type galaxies from the SAMI survey. Using r-band sizes and luminosities from new Multi-Gaussian Expansion (MGE) photometric measurements, and treating luminosity as the dependent variable, the FP has coefficients a=1.294\(\pm\)0.039, b= 0.912\(\pm\)0.025, and zero-point c= 7.067\(\pm\)0.078. We leverage the high signal-to-noise of SAMI integral field spectroscopy, to determine how structural and stellar-population observables affect the scatter about the FP. The FP residuals correlate most strongly (8\(\sigma\) significance) with luminosity-weighted simple-stellar-population (SSP) age. In contrast, the structural observables surface mass density, rotation-to-dispersion ratio, Sérsic index and projected shape all show little or no significant correlation. We connect the FP residuals to the empirical relation between age (or stellar mass-to-light ratio \(\Upsilon_\star\)) and surface mass density, the best predictor of SSP age amongst parameters based on FP observables. We show that the FP residuals (anti-)correlate with the residuals of the relation between surface density and \(\Upsilon_\star\). This correlation implies that part of the FP scatter is due to the broad age and \(\Upsilon_\star\) distribution at any given surface mass density. Using virial mass and \(\Upsilon_\star\) we construct a simulated FP and compare it to the observed FP. We find that, while the empirical relations between observed stellar population relations and FP observables are responsible for most (75%) of the FP scatter, on their own they do not explain the observed tilt of the FP away from the virial plane.