The top-of-the-atmosphere (TOA) reflectances of the Visible Infrared Imaging Radiometer (VIIRS) M9 channel and the Moderate Resolution Imaging Spectroradiometer (MODIS) 26 channel have been simulated using the libRadtran radiative transfer model and the Cloud-Aerosol Lidar with Orthogonal Polarization (CALIOP) Vertical Feature Mask data. The simulated data were analyzed to quantify the performance differences between the VIIRS M9 and the MODIS 26 channels. Analysis of simulated clear-sky TOA reflectances showed that compared MODIS channel 26, the VIIRS M9 channel always performs better in reducing background reflectance regardless of latitude, season, surface type, vapor content or surface elevation. For mid-latitude, sub-Arctic and tropical regions the VIIRS M9 channel reduce the background reflectance by approximately 66.7%, 52.6% and 41.5%, respectively over the surface type of sandstone, compared to MODIS channel 26. Simulations for cloudy skies showed that both stratus and cumulus clouds contribute less to VIIRS M9 and MODIS band 26 TOA reflectances. Analysis of observed MODIS, VIIRS and CALIOP data was consistent with the simulated results. The VIIRS M9 decreases clear-sky background reflectance by as much as 35.96% and non-cirrus cloud reflectance by 29.86% compared with the MODIS channel 26. The observed reflectances of MODIS and VIIRS cirrus channels for clear-sky, non-cirrus cloud, and cirrus cloud are 0.0133 and 0.0095, 0.020 and 0.015, 0.084 and 0.067 respectively. •The VIIRS M9 channel always performs better than MODIS channel 26.•The VIIRS M9 decreases background reflectance by 24.67% compared with the MODIS.•The VIIRS M9 increase cirrus cloud reflectance by 1.84% compared with the MODIS.