The sky view factor (SVF) is a crucial variable widely used to quantify the characteristics of surface structures and estimate surface radiation budget. Many SVF models based on raster data have been developed but not yet evaluated in a more quantitative and uniform manner. In this paper, four typical SVF models (Dozier‐Frew (D‐F), Manners, Lindberg‐Grimmond (L‐G), and Helbig_h) are evaluated using the SVF derived from simulated fisheye images based on the digital surface model (DSM) and digital elevation model data. The SVF calculated by D‐F method using DSM data has the best accuracy, with a mean bias error of −0.007, root‐mean‐square error of 0.069, and coefficient of determination (R2) of 0.914. For the SVF value derived from digital elevation model data, L‐G method shows good performance, with an mean bias error of 0.013, root‐mean‐square error of 0.032, and R2 of 0.897. The pixels near the edges of buildings, within the valley or along ridgelines, have higher SVF deviations. In addition, the slope angle calculated using DSM data has some artificial defects that make the significant impact on the SVF biases due to their calculation method and the discontinuous surface in urban areas. Thus, L‐G and Helbig_h methods are more applicable for the DSM data due to the difficulty in defining slope and aspect angles. Moreover, the high accordance of SVFs between Helbig_h and L‐G methods implies that the Helbig_h method is an alternative in virtue of its simpler form and lower computation cost than L‐G method. Plain Language Summary Four typical sky view factor (SVF) models are evaluated and compared using simulated SVF data based on two kinds of raster terrain data to estimate models' accuracies and their different spatial characteristics. They show that different calculation accuracies and large SVF bias are mainly near the edges of buildings, along the valley and ridgeline. The slope angle can cause larger SVF biases than the aspect angle, and the influence on the SVF derived from DSM data is more significant than that derived from digital elevation model data. This paper evaluates these methods in a more quantitative and uniform manner, which offers accuracy levels that are important for SVF's application in various fields. The new findings and implications for possible improvements can benefit the estimation of the urban heat island, surface solar radiation, and melting of the polar icecap. Key Points Four different SVF models are evaluated and compared using urban DSM and montane DEM data in a quantitative and uniform manner The slope angle calculated using DSM data has some artificial defects that cause large SVF biases due to the discontinuous surface in urban areas The L‐G and Helbig_h methods are more applicable for urban DSM data because defining slope and aspect angles is difficult for the DSM data