Vibration-based rolling element bearing diagnostics is a very well-developed field, yet researchers continue to develop new diagnostic algorithms quite frequently. Over the last decade, data from the Case Western Reserve University (CWRU) Bearing Data Center has become a standard reference used to test these algorithms, yet without any recognised benchmark it is difficult to properly assess the performance of any proposed diagnostic methods. There is, then, a clear need to examine the data thoroughly and to categorise it appropriately, and this paper intends to fulfil that objective. To do so, three established diagnostic techniques are applied to the entire CWRU data set, and the diagnostic outcomes are provided and discussed in detail. Recommendations are given as to how the data might best be used, and also on how any future benchmark data should be generated. Though intended primarily as a benchmark to aid in testing new diagnostic algorithms, it is also hoped that much of the discussion will have broader applicability to other bearing diagnostics cases. •We provide a thorough benchmark analysis of the CWRU Bearing Data.•Three established diagnostic techniques are applied.•We provide the diagnostic outcomes for all data sets as a benchmark reference.•We discuss the records at length and identify data problems and anomalies.•We give recommendations as to how best to use the data for algorithm development.