Considering the growing demand for broadband of access networks, in the present paper we investigate various modulation formats as a way of increasing the performance of optical transmission systems. Non-return-to-zero (NRZ) on-off keying, return-to-zero (RZ) OOK, carrier suppressed RZ (CSRZ) OOK, duobinary (DB), NRZ differential phase shift keying (NRZDPSK), RZ-DPSK and CSRZ-DPSK formats are compared using the maximal achievable reach with bit error rate less than 10 as a criterion. Simulations are performed by using OptSim software tool. It is shown that using the transmission system without dispersion compensation the best results are shown by duobinary and CSRZ-OOK modulation formats, but with the system using dispersion compensating fiber (DCF) the longest transmission distance is achieved by RZ-DPSK modulation format. By investigating the influence of channel spacing for best-performed modulation formats, network reach decrease for transmission systems with DCF fiber has been observed due to channel crosstalk.