During online social networks (OSNs), popularity prediction uncovers the final size of online content based on the observed cascade, which has been the critical technology for online recommendation, viral marketing, and rumor detection. Recently, representation learning could help to infer the mapping between the dynamic cascade and the final popularity efficiently, and has been a new research paradigm for popularity prediction. However, those methods are vulnerable to structure disturbance when lack of fine-grained supervision, as only the dynamic cascade is used. Therefore, we propose a novel trend and cascade based spatiotemporal evolution network (TCSE-Net), which preserves the distinguishable structure pattern while eliminating potential noise, via aligning and fusing the temporal popularity and cascade. To be specific, we first leveraged the Long-Short Term Memory (LSTM) and recurrent graph convolutional network (GCN) to learn the trend representation and the corresponding cascade representation respectively. Meanwhile, we represent node with it’s layer, thereby the hierarchy is preserved in cascade representation through GCN. Then, both trend and cascade representations are aligned in time sequence and selectively assembled by a set of shared parameters for popularity prediction. The extensive experimental results show that our TCSE-Net outperforms state-of-the-art baselines on two real datasets. Related code will be publicly available on https://github.com/TAN-OpenLab/TCSE-Net .