Recent evidence has shown that buildings designed to be high-ranking, according to the Western architectural decorum, have more impact on the minds of their beholders than low-ranking buildings. Here we investigated whether and how the aesthetic judgment for high- and low-ranking buildings was affected by differences in cultural expertise and by power spectrum differences. A group of Italian and Japanese participants performed aesthetic judgment tasks, with line drawings of high- and low-ranking buildings and with their random-phase versions (an image with the exact power spectrum of the original one but non-recognizable anymore). Irrespective of cultural expertise, high-ranking buildings and their relative random-phase versions received higher aesthetic judgments than low-ranking buildings and their random-phase versions. These findings indicate that high- and low-ranking buildings are differentiated for their aesthetic value and they show that low-level visual processes influence the aesthetic judgment based on differences in the stimuli power spectrum, irrespective of the influence of cultural expertise.