Computer-aided diagnosis system is becoming a more and more important tool in clinical treatment, which can provide a verification of the doctors’ decisions. In this paper, we proposed a novel abnormal brain detection method for magnetic resonance image. Firstly, a pre-trained AlexNet was modified with batch normalization layers and trained on our brain images. Then, the last several layers were replaced with an extreme learning machine. A searching method was proposed to find the best number of layers to be replaced. Finally, the extreme learning machine was optimized by chaotic bat algorithm to obtain better classification performance. Experiment results based on 5 × hold-out validation revealed that our method achieved state-of-the-art performance.