This article traces the history of slavery and of post-slavery struggles for livelihood and status in the layout, the physical structures, and the social lives of three towns on the southern Swahili Coast: Kilwa, Mikindani, and Lindi. These towns were long surrounded by plantations that relied on the labor of enslaved people. In the first decades of the twentieth century, slave populations dissipated quickly. In towns, the colonial cash crop economy, voluntary rural-urban migration, and the decline of slave-owning elites combined to allow former slaves to assimilate and adopt new urbanite identities. Sufi orders played a central role in affording ex-slaves a respectable presence in town. Nevertheless, former slave owners and former slaves lived in different parts of these towns, former slaves' livelihoods were more precarious, and the imputation of slave origins remains offensive, even today. Indeed, the era of slavery still divides people and still engages the social imagination.