Membrane contact sites (MCSs) serve as a zone for nonvesicular lipid transport by oxysterol-binding protein (OSBP)-related proteins (ORPs). ORPs mediate lipid countertransport, in which two distinct lipids are transported counterdirectionally. How such lipid countertransport controls specific biological functions, however, remains elusive. We report that lipid countertransport by ORP10 at ER-endosome MCSs regulates retrograde membrane trafficking. ORP10, together with ORP9 and VAP, formed ER-endosome MCSs in a phosphatidylinositol 4-phosphate (PI4P)-dependent manner. ORP10 exhibited a lipid exchange activity toward its ligands, PI4P and phosphatidylserine (PS), between liposomes in vitro, and between the ER and endosomes in situ. Cell biological analysis demonstrated that ORP10 supplies a pool of PS from the ER, in exchange for PI4P, to endosomes where the PS-binding protein EHD1 is recruited to facilitate endosome fission. Our study highlights a novel lipid exchange at ER-endosome MCSs as a nonenzymatic PI4P-to-PS conversion mechanism that organizes membrane remodeling during retrograde membrane trafficking.