The diagnosis of the key components of rotating machinery systems is essential for the production efficiency and quality of manufacturing processes. The performance of the traditional diagnosis method depends heavily on feature extraction, which relies on the degree of individual's expertise or prior knowledge. Recently, a deep learning (DL) method is applied to automate feature extraction. However, training in the DL method requires a massive amount of sensor data, which is time consuming and poses a challenge for its applications in engineering. In this paper, a new data-driven fault diagnosis method based on compressed sensing (CS) and improved multiscale network (IMSN) is proposed to recognize and classify the faults in rotating machinery. CS is used to reduce the amount of raw data, from which the fault information is discovered. At the same time, it can be used to generate sufficient training samples for the subsequent learning. The one-dimensional compressed signal is converted to two-dimensional image for further learning. An IMSN is established for learning and obtaining deep features. It improves the diagnosis performance of the DL process. The faults of the key components are identified from a softmax model. Experimental analysis is performed to verify effectiveness of the proposed data-driven fault diagnosis method.