In clathrin-mediated endocytosis, adapter proteins assemble together with clathrin through interactions with specific lipids on the plasma membrane. However, the precise mechanism of adapter protein assembly at the cell membrane is still unknown. Here, we show that the membrane-proximal domains ENTH of epsin and ANTH of Sla2 form complexes through phosphatidylinositol 4,5-bisphosphate (PIP2) lipid interfaces. Native mass spectrometry reveals how ENTH and ANTH domains form assemblies by sharing PIP2 molecules. Furthermore, crystal structures of epsin Ent2 ENTH domain from S. cerevisiae in complex with PIP2 and Sla2 ANTH domain from C. thermophilum illustrate how allosteric phospholipid binding occurs. A comparison with human ENTH and ANTH domains reveal only the human ENTH domain can form a stable hexameric core in presence of PIP2, which could explain functional differences between fungal and human epsins. We propose a general phospholipid-driven multifaceted assembly mechanism tolerating different adapter protein compositions to induce endocytosis.