Novel invertebrate‐killing compounds are required in agriculture and medicine to overcome resistance to existing treatments. Because insecticides and anthelmintics are discovered in phenotypic screens, a crucial step in the discovery process is determining the mode of action of hits. Visible whole‐organism symptoms are combined with molecular and physiological data to determine mode of action. However, manual symptomology is laborious and requires symptoms that are strong enough to see by eye. Here, we use high‐throughput imaging and quantitative phenotyping to measure Caenorhabditis elegans behavioral responses to compounds and train a classifier that predicts mode of action with an accuracy of 88% for a set of ten common modes of action. We also classify compounds within each mode of action to discover substructure that is not captured in broad mode‐of‐action labels. High‐throughput imaging and automated phenotyping could therefore accelerate mode‐of‐action discovery in invertebrate‐targeting compound development and help to refine mode‐of‐action categories. Synopsis A combination of imaging and machine learning is used to predict compound mode of action using the unique behavioural responses of the roundworm C. elegans to different pesticides and anthelmintics. Insecticides affect phenotypes in multiple behavioural dimensions. Compounds with the same mode of action have similar effects on behaviour. Combining classifiers by voting enables mode of action prediction. The approach allows mode of action deconvolution within classes. A combination of imaging and machine learning is used to predict compound mode of action using the unique behavioural responses of the roundworm C. elegans to different pesticides and anthelmintics.