Purpose A precise resection of the entire tumor tissue during surgery for brain metastases is essential to reduce local recurrence. Conventional intraoperative imaging techniques all have limitations in detecting tumor remnants. Therefore, there is a need for innovative new imaging methods such as optical coherence tomography (OCT). The purpose of this study is to discriminate brain metastases from healthy brain tissue in an ex vivo setting by applying texture analysis and machine learning algorithms for tissue classification to OCT images. Methods Tumor and healthy tissue samples were collected during resection of brain metastases. Samples were imaged using OCT. Texture features were extracted from B-scans. Then, a machine learning algorithm using principal component analysis (PCA) and support vector machines (SVM) was applied to the OCT scans for classification. As a gold standard, an experienced pathologist examined the tissue samples histologically and determined the percentage of vital tumor, necrosis and healthy tissue of each sample. A total of 14.336 B-scans from 14 tissue samples were included in the classification analysis. Results We were able to discriminate vital tumor from healthy brain tissue with an accuracy of 95.75%. By comparing necrotic tissue and healthy tissue, a classification accuracy of 99.10% was obtained. A generalized classification between brain metastases (vital tumor and necrosis) and healthy tissue was achieved with an accuracy of 96.83%. Conclusions An automated classification of brain metastases and healthy brain tissue is feasible using OCT imaging, extracted texture features and machine learning with PCA and SVM. The established approach can prospectively provide the surgeon with additional information about the tissue, thus optimizing the extent of tumor resection and minimizing the risk of local recurrences.