Quantum dots (QDs) show a great potential for light-emitting diodes (LEDs) packaging, which still face great challenges compared with the matured phosphor downconversion materials. These are probably caused by the unique scattering and absorption properties of QDs, which are extremely different to the traditional phosphor due to their several nanometers size, while their effect on QDs-converted LEDs (QCLEDs) is barely studied. In this paper, we have experimentally and theoretically investigated the effect of scattering and absorption of CdSe/ZnS QDs on the optical performance for QCLEDs by comparing with the traditional yttrium aluminum garnet phosphor. Results indicate that the strong absorption (reabsorption) of QDs causes low radiant efficacy and stability for QCLEDs; their weak scattering also leads to a low color uniformity. It demonstrates that their unique scattering and absorption properties are key factors leading to low optical performance of QCLEDs compared with the traditional phosphor-converted LEDs. For purpose of gaining the white LED with high efficiency and stability, it is highly suggested to use a low QD concentration to reduce the reabsorption loss and the total internal reflection loss. We believe that this paper can provide a better understanding of improving the optical performance for QCLEDs from the prospective of scattering and absorption. In the future, it is important to use low QD concentration to gain high-performance white LEDs with high downconversion efficiency by optimizing packaging structures.