Hsp70s are important cancer chaperones that act upstream of Hsp90 and exhibit independent anti-apoptotic activities. To develop chemical tools for the study of human Hsp70, we developed a homology model that unveils a previously unknown allosteric site located in the nucleotide binding domain of Hsp70. Combining structure-based design and phenotypic testing, we discovered a previously unknown inhibitor of this site, YK5. In cancer cells, this compound is a potent and selective binder of the cytosolic but not the organellar human Hsp70s and has biological activity partly by interfering with the formation of active oncogenic Hsp70/Hsp90/client protein complexes. YK5 is a small molecule inhibitor rationally designed to interact with an allosteric pocket of Hsp70 and represents a previously unknown chemical tool to investigate cellular mechanisms associated with Hsp70. Display omitted •Homology model of human Hsp70 unveils a druggable allosteric site•YK5, a selective binder of the allosteric site, is identified by rational design•YK5 has activity in cancer cells partly by altering the Hsp70/Hsp90 complex•The effect of YK5 on Hsp70/Hsp90 complexes is uncoupled from HSF-1 Hsp70s are important cancer proteins. Rodina et al. unveil an allosteric site located in Hsp70’s nucleotide binding domain, which they use for the rational design of YK5, a selective and potent Hsp70 binder with retained cellular activity. YK5 is a promising chemical tool to investigate Hsp70 associated mechanisms.