Direct detection of circularly polarized light (CPL) is a challenging task due to limited materials and ambiguous structure–property relationships that lead to low distinguishability of the light helicities. Perovskite ferroelectric semiconductors incorporating chirality provide new opportunities in dealing with this issue. Herein, a pair of 2D chiral perovskite ferroelectrics is reported, which have enhanced CPL detection performance due to interplays among lattice, photon, charge, spin, and orbit. The chirality‐transfer‐induced chiral&polar ferroelectric phase enhances the asymmetric nature of the photoactive sublattice and achieves a switchable self‐powered detection via the bulk photovoltaic effect. The single‐crystal‐based device exhibits a CPL‐sensitive detection performance under 430 nm with an asymmetric factor of 0.20 for left‐ and right‐CPL differentiation, about two times that of the pure chiral counterparts. The enhanced CPL detection performance is ascribed to the Rashba–Dresselhaus effect that originates from the bulk inversion asymmetry and strong spin–orbit coupling, shown with a large Rashba coefficient, which is demonstrated by density functional theory calculation and circularly polarized light excited photoluminescence measurement. These results provide new perspectives on chiral Rashba ferroelectric semiconductors for direct CPL detection and ferroelectrics‐based chiroptics and spintronics. A pair of 2D chiral perovskite Rashba ferroelectric semiconductors is reported. The fabricated single‐crystal device responds to circularly polarized light (CPL) under 430 nm with an anisotropy factor of 0.20 for the left‐ and right‐CPL differentiation, about two times of reported pure chiral counterparts. The enhanced performance is ascribed to the Rashba–Dresselhaus effect with a large Rashba coefficient of 0.93 eV Å.