Facial expression is the main medium of information transmission in human communication, playing an important role in human's daily life. Facial expression recognition is still challenging due to the various obstacle, illumination, and posture. However, most of the existing works focus on deeper or wider network structures and rarely explores the high-level feature statistics. In this paper, we propose a second-order pooling convolution neural network to explore the correlation information between the facial features after deep network learning. At the final stage of the network, we add a new covariance pooling layer to replace the first-order pooling of standard convolution networks. In the pooling layer of covariance, the Newton iteration method is used to approximate the square root instead of EIG or SVD, which makes it more suitable for GPU. Due to the small amount of facial expression data, this paper uses different data augmentation methods to increase the amount of training data and improve the generalization ability of the model. The proposed method, data augmentation and second-order pooling (DASOP), was evaluated on the real-world affective faces database (RAFDB) and the static facial expressions in the wild (SFEW), yielding correct rates of 88.625% and 59.518%, respectively. We achieve state-of-the-art performance superior to existing methods.