Metro network, as a medium distance transmission system, poses a special challenge of transmission capacity and cost. In this paper, we present a detailed comparison of applying three advanced modulation formats including pulse amplitude modulation-4 (PAM-4), carrierless amplitude and phase modulation-16 QAM (CAP-16), and discrete Fourier transform spread orthogonal frequency division multiplexing (DFT-S OFDM) with low-cost direct detection for 100-Gb/s/λ metro optical transmission systems. These modulation formats are all experimentally demonstrated with similar digital signal processing algorithms. Three kinds of chromatic dispersion (CD) compensation ways named CD precompensation method, single sideband (SSB), and dispersion compensating fiber (DCF) are also compared. Two types of modulators including an IQ modulator and a dual-drive Mach-Zehnder modulator are employed to generate SSB or CD precompensated signals in this experiment and their transmission performance is also evaluated. To the best of our knowledge, this is the first time that the performance of single-wavelength 100-G signal transmission based on direct detection is evaluated over 480 km of standard single mode fiber with PAM-4, CAP-16, and DFT-S OFDM modulation formats.