•The largest publicly dataset of breast cancer pathological images is released.•Dataset diversity alleviates relatively low accuracy of benign images classification.•Richer multilevel features make the image-wise feature fusion more sufficient.•The short-term and long-term correlations between patches are both preserved.•Our hybrid network outperformed other methods in pathological image classification. Even with the rapid advances in medical sciences, histopathological diagnosis is still considered the gold standard in diagnosing cancer. However, the complexity of histopathological images and the dramatic increase in workload make this task time consuming, and the results may be subject to pathologist subjectivity. Therefore, the development of automatic and precise histopathological image analysis methods is essential for the field. In this paper, we propose a new hybrid convolutional and recurrent deep neural network for breast cancer histopathological image classification. Based on the richer multilevel feature representation of the histopathological image patches, our method integrates the advantages of convolutional and recurrent neural networks, and the short-term and long-term spatial correlations between patches are preserved. The experimental results show that our method outperforms the state-of-the-art method with an obtained average accuracy of 91.3% for the 4-class classification task. We also release a dataset with 3771 breast cancer histopathological images to the scientific community that is now publicly available at http://ear.ict.ac.cn/?page_id=1616. Our dataset is not only the largest publicly released dataset for breast cancer histopathological image classification, but it covers as many different subclasses spanning different age groups as possible, thus providing enough data diversity to alleviate the problem of relatively low classification accuracy of benign images.