Video deblurring is a challenging task because only input blurry sequences are available. To further constrain the optimization process, existing methods explore various additional information, e.g., events, depth, and sharpness prior. However, they consume large computational costs or generate unpleasant visual results due to the insufficient exploitation of spatio-temporal information. In this work, we propose a novel spatio-temporal sharpness map learned by a prior-based generation network implicitly. The proposed generation network blends both spatial and temporal sharpness priors in a blurry sequence, while few extra parameters are added. We show that the proposed map has better spatial continuity and guidance for video deblurring than the previous methods. Furthermore, different from the simply concatenation in the previous work, we allow the sharpness map to accommodate to more effective video deblurring via a dual-stream network. Specifically, the network is decomposed by two branches, namely the inter-frame and intra-frame reconstructions. The inter-frame reconstruction obtains the sharp patches of consecutive frames from the sharpness map to restore textures well. Meanwhile, the intra-frame branch is responsible for recovering structures of the latent frame, where a novel histogram statistical method is developed to quantify and count textures in the features under the modulation of the sharpness map. Quantitative and qualitative experiments successfully validate the effectiveness of our proposed method.