Amorphous solids lack long-range order. Therefore identifying structural defects -- akin to dislocations in crystalline solids -- that carry plastic flow in these systems remains a daunting challenge. By comparing many different structural indicators in computational models of glasses, under a variety of conditions we carefully assess which of these indicators are able to robustly identify the structural defects responsible for plastic flow in amorphous solids. We further demonstrate that the density of defects changes as a function of material preparation and strain in a manner that is highly correlated with the macroscopic material response. Our work represents an important step towards predicting how and when an amorphous solid will fail from its microscopic structure.