Human activities and city routines follow patterns. Transfer learning can help achieve scalable solutions toward the realization of smart cities accounting for similarities between regions, domains, and activities. In this study, we propose a transfer learning-based framework for smart buildings to test this hypothesis in energy-related problems. Our framework has two major components: the network creation and the transferable predictive model. In order to create the network that groups buildings sharing characteristics, we evaluated two strategies: a novel clustering algorithm for mixed data, k -prod, and clustering the image-based representation of time series. Then, a combination of long short term memory and convolutional neural network was trained on the centroids of the clusters for energy consumption prediction. The coefficient of variation of the root mean squared error (CVRMSE) of the predictions in such clusters vary between 3.85% and 58.85%. The obtained parameters were transferred to the rest of the buildings for predictive purposes, finding accurate results in buildings with little data. Our framework deals with insufficient training data since parameters from scenarios with more sensors can be received. It also carries out state-of-the-art performance on three datasets from different sources having in total 533 rooms/buildings and two energy efficiency domains: consumption prediction reducing the CVRMSE in a 21.6%, and air conditioning usage prediction moving from a 4.18% to a 0.28% CVRMSE. Our framework extracts more knowledge from available IoT deployments, so that smartness could be spread between environments at a fewer cost given that less individual effort will be needed.