Dust-enshrouded, starbursting, submillimeter galaxies (SMGs) at z ≥ 3 have been proposed as progenitors of z ≥ 2 compact quiescent galaxies (cQGs). To test this connection, we present a detailed spatially resolved study of the stars, dust, and stellar mass in a sample of six submillimeter-bright starburst galaxies at z ∼ 4.5. The stellar UV emission probed by HST is extended and irregular and shows evidence of multiple components. Informed by HST, we deblend Spitzer/IRAC data at rest-frame optical, finding that the systems are undergoing minor mergers with a typical stellar mass ratio of 1:6.5. The FIR dust continuum emission traced by ALMA locates the bulk of star formation in extremely compact regions (median re = 0.70 0.29 kpc), and it is in all cases associated with the most massive component of the mergers (median ). We compare spatially resolved UV slope (β) maps with the FIR dust continuum to study the infrared excess (IRX = LIR/LUV)-β relation. The SMGs display systematically higher IRX values than expected from the nominal trend, demonstrating that the FIR and UV emissions are spatially disconnected. Finally, we show that the SMGs fall on the mass-size plane at smaller stellar masses and sizes than the cQGs at z = 2. Taking into account the expected evolution in stellar mass and size between z = 4.5 and z = 2 due to the ongoing starburst and mergers with minor companions, this is in agreement with a direct evolutionary connection between the two populations.