The Fringe Sensor Unit (FSU) is the central element of the Phase Referenced Imaging and Micro-arcsecond Astrometry (PRIMA) dual-feed facility and provides fringe sensing for all observation modes, comprising off-axis fringe tracking, phase referenced imaging, and high-accuracy narrow-angle astrometry. It is installed at the Very Large Telescope Interferometer (VLTI) and successfully servoed the fringe tracking loop during the initial commissioning phase. Unique among interferometric beam combiners, the FSU uses spatial phase modulation in bulk optics to retrieve real-time estimates of fringe phase after spatial filtering. A R=20 spectrometer across the K-band makes the retrieval of the group delay signal possible. The FSU was integrated and aligned at the VLTI in summer 2008. It yields phase and group delay measurements at sampling rates up to 2 kHz, which are used to drive the fringe tracking control loop. During the first commissioning runs, the FSU was used to track the fringes of stars with K-band magnitudes as faint as m_K=9.0, using two VLTI Auxiliary Telescopes (AT) and baselines of up to 96 m. Fringe tracking using two Very Large Telescope (VLT) Unit Telescopes (UT) was demonstrated. During initial commissioning and combining stellar light with two ATs, the FSU showed its ability to improve the VLTI sensitivity in K-band by more than one magnitude towards fainter objects, which is of fundamental importance to achieve the scientific objectives of PRIMA.