Macromolecular complexes exhibit reduced diffusion in biological membranes; however, the physiological consequences of this characteristic of plasma membrane domain organization remain elusive. We report that competition between the galectin lattice and oligomerized caveolin-1 microdomains for epidermal growth factor (EGF) receptor (EGFR) recruitment regulates EGFR signaling in tumor cells. In mammary tumor cells deficient for Golgi β1,6N-acetylglucosaminyltransferase V (Mgat5), a reduction in EGFR binding to the galectin lattice allows an increased association with stable caveolin-1 cell surface microdomains that suppresses EGFR signaling. Depletion of caveolin-1 enhances EGFR diffusion, responsiveness to EGF, and relieves Mgat5 deficiency-imposed restrictions on tumor cell growth. In Mgat5⁺/⁺ tumor cells, EGFR association with the galectin lattice reduces first-order EGFR diffusion rates and promotes receptor interaction with the actin cytoskeleton. Importantly, EGFR association with the lattice opposes sequestration by caveolin-1, overriding its negative regulation of EGFR diffusion and signaling. Therefore, caveolin-1 is a conditional tumor suppressor whose loss is advantageous when β1,6GlcNAc-branched N-glycans are below a threshold for optimal galectin lattice formation.