Cytoplasmic lipid droplets are important organelles in nearly every eukaryotic and some prokaryotic cells. Storing and providing energy is their main function, but they do not work in isolation. They respond to stimuli initiated either on the cell surface or in the cytoplasm as conditions change. Cellular stresses such as starvation and invasion are internal insults that evoke changes in droplet metabolism and dynamics. This review will first outline lipid droplet assembly and then discuss how droplets respond to stress and in particular nutrient starvation. Finally, the role of droplets in viral and microbial invasion will be presented, where an unresolved issue is whether changes in droplet abundance promote the invader, defend the host, to try to do both. The challenges of stress and infection are often accompanied by changes in physical contacts between droplets and other organelles. How these changes may result in improving cellular physiology, an ongoing focus in the field, is discussed. This review offers a concise summary of our current view of how cytoplasmic lipid droplets respond to cellular stresses and pathogen invasion, and cross‐talk with other organelles in order to maintain energy balance and cellular physiology.