•A lightweight pancreas segmentation model with spatial prior is presented.•A new prior propagation module is built to dynamically explore the spatial prior.•Our method has low computational cost and state-of-the-art performance. Nowadays, pancreas segmentation in CT scans has gained more and more attention for computer-assisted diagnosis of inflammation (pancreatitis) or cancer. Despite the thrilling success of deep convolutional neural networks (DCNNs) in automatic pancreas segmentation, the heavy computational complexity of such networks impedes the deployment in clinical applications. To alleviate this issue, this paper establishes a novel end-to-end DCNN model for pursuing high-accurate automatic pancreas segmentation but with low computational cost. Specifically, built upon a simplified FCN architecture, we propose two novel network modules, named as the scale-transferrable feature fusion module (STFFM) and prior propagation module (PPM), respectively, for pancreas segmentation. Equipped with the scale-transferrable operation, STFFM can learn rich fusion features but with very lightweight network architecture. By dynamically adapting the spatial prior to the input slice data as well as the deep feature maps, PPM enables the network model to explore informative spatial priors for pancreas segmentation. Comprehensive experiments on the NIH dataset and the MSD dataset are conducted to evaluate the proposed approach. The obtained experimental results demonstrate that our approach can effectively reduce the computational cost and simultaneously archive the outperforming performance when compared to the state-of-the-art methods.