The installation of green roofs, defined here as rooftops with a shallow soil cover and extensive vegetation, has been proposed as a possible measure to mitigate the loss of green space caused by the steady growth of cities. However, the effectiveness of green roofs in supporting arthropod communities, and the extent to which they facilitate connectivity of these communities within the urban environment is currently largely unknown. We investigated the variation of species community composition (β diversity) of four arthropod groups with contrasting mobility (Carabidae, Araneae, Curculionidae, and Apidae) on 40 green roofs and 40 extensively managed green sites on the ground in the city of Zurich, Switzerland. With redundancy analysis and variation partitioning, we (1) disentangled the relative importance of local environmental conditions, the surrounding land cover composition, and habitat connectivity on species community composition, (2) searched for specific spatial scales of habitat connectivity for the different arthropod groups, and (3) discussed the ecological and functional value of green roofs in cities. Our study revealed that on green roofs community composition of high-mobility arthropod groups (bees and weevils) were mainly shaped by habitat connectivity, while low-mobility arthropod groups (carabids and spiders) were more influenced by local environmental conditions. A similar but less pronounced pattern was found for ground communities. The high importance of habitat connectivity in shaping high-mobility species community composition indicates that these green roof communities are substantially connected by the frequent exchange of individuals among surrounding green roofs. On the other hand, low-mobility species communities on green roofs are more likely connected to ground sites than to other green roofs. The integration of green roofs in urban spatial planning strategies has great potential to enable higher connectivity among green spaces, so that eventually even communities of low-mobility species become connected. Furthermore, improving the design of green roofs (composition and configuration of vegetation and soil types) could enhance the ecological value, particularly for low-mobility species.