Urban natural surfaces and non-surface human activities are key factors determining the urban heat island (UHI), but their relative importance remains highly controversial and may vary at different spatial scales and focal urban systems. However, systematic studies on the scale-dependency system-specificity remain largely lacking. Here, we selected 32 major Chinese cities as cases and used Landsat 8 images to retrieve land surface temperature (LST) and quantify natural surface variables using point of interest (POI) data as a measure of the human activity variable and using multiple regression and relative weight analysis to study the contribution and relative importance of these factors to LST at a range of grain sizes (0.25–5 km) and spatial extents (20–60 km). We revealed that the contributions and relative importance of natural surfaces and human activities are largely scale-dependent and system-specific. Natural surfaces, especially vegetation cover, are often the most important UHI determinants for a majority of scales, but the importance of non-surface human activities is increasingly pronounced at a coarser spatial scale with respect to both grain and spatial extent. The scaling relations of the UHI determinants and their relative importance were mostly linear-like at the city-collective level, but highly diverse across individual cities, so reducing non-surface heat emissions could be the most effective measure in particular cases, especially at relatively large spatial scales. This study advances the understanding of UHI formation mechanisms and highlights the complexity of the scale issue underpinning the UHI effect.