Switch rails are weak but essential components in a high-speed rail track system, which have an urgent non-destructive testing requirement due to aging and associated fatigue damage accumulation. They are settled under sophisticated operation environments, which causes them to have unpredictable damages, such as abrasion, exfoliation, and cracks. Our goal is to propose a reliable system to detect structural damages of switch rails. Using ultrasonic guided waves to examine the health status of switch rails makes it possible to continuously evaluate the health status of switch rails when they are in use. Conventional damage detection methods with ultrasonic guided waves such as baseline signal subtraction, independent component analysis-based methods cannot always make reliable detection results. These methods are either lack of powerful abilities to capture the characteristics of damaged signals or time-consuming to be operated in real damage detection tasks. In this paper, a convolutional neural network-based system is proposed to solve both of the above challenges simultaneously. The proposed model employs multiple convolutional layers to extract deep features of ultrasonic guided wave signals. These features are then fed into a classifier to predict whether they are damaged signals or not. To evaluate the proposed model performance, we collected ultrasonic guided wave signals from two different switch rails. The proposed model achieved more than 91% testing accuracy and outperformed other relevant methods. It also demonstrated the proposed model had strong generalization abilities to make it capable in practical switch rail structural damage detection tasks.