We study how half-mass radii, central mass densities (Σ1), and color gradients change as galaxies evolve. We separate ∼7000 galaxies into 16 groups with similar spectral shapes; each group represents a different evolutionary stage. We find that different galaxy types populate different regions of both size–mass and Σ1–mass space. The nine star-forming groups lie along the integrated star-forming Σ1–mass relation. However, these star-forming groups form steep parallel relations in the size–mass plane, with slopes similar to the quiescent size–mass relation. These steep slopes can be explained as a transformation of the star-forming Σ1–mass relation and its scatter. We identify three types of transitional galaxies. Green valley and post-starburst galaxies are similarly compact at z > 1.5; however, their distinct color gradients indicate that the two populations represent different pathways to quenching. Post-starburst galaxies have flat color gradients and compact structures, consistent with a fast quenching pathway that requires structural change and operates primarily at high redshift. Green valley galaxies have negative color gradients, and are both larger and more numerous toward lower redshift. These galaxies are consistent with slow quenching without significant structural change. We find that dusty star-forming galaxies at z ≳ 2 are very compact and may represent the “burst” before post-starburst galaxies; at z ≲ 2, dusty star-forming galaxies are extended and have shallow color gradients consistent with slow quenching. Our results suggest that star-forming galaxies grow gradually up the Σ1–mass relation until (a) they naturally reach the high Σ1 values required for quiescence or (b) a compaction-type event rapidly increases their Σ1.