As a circulating commodity, sugar holds heavy histories of violence and exploitation that connect up map points of the former British Empire in ways that place time firmly out of joint. This article suggests how persistent visual traces of these sugar remainders can contribute to contemporary reckonings with the past time of empire. Working in the interval between commemoration and ruination, it shows how Kara Walker's A Subtlety (2014) has informed how contemporary artists and performers in Scotland engage with sugar histories in their practice. Alberta Whittle, Kayus Bankole, and Adura Onashile all figure sugar time as both continual and contemporary, linking histories of empire and post-industrial memories with artistic and activist expressions in the present. Their works respond to the violence of history in what Saidiya Hartman has termed "aesthetic mode", leading to the formation of creative and critical practices that people can enact to untangle themselves from the sedimented pain of imperial pasts.