The goal of this study is to demonstrate the feasibility of a novel fully-convolutional volumetric dose prediction neural network (DoseNet) and test its performance on a cohort of prostate stereotactic body radiotherapy (SBRT) patients. DoseNet is suggested as a superior alternative to U-Net and fully connected distance map-based neural networks for non-coplanar SBRT prostate dose prediction. DoseNet utilizes 3D convolutional downsampling with corresponding 3D deconvolutional upsampling to preserve memory while simultaneously increasing the receptive field of the network. DoseNet was implemented on 2 Nvidia 1080 Ti graphics processing units and utilizes a 3 phase learning protocol to help achieve convergence and improve generalization. DoseNet was trained, validated, and tested with 151 patients following Kaggle completion rules. The dosimetric quality of DoseNet was evaluated by comparing the predicted dose distribution with the clinically approved delivered dose distribution in terms of conformity index, heterogeneity index, and various clinically relevant dosimetric parameters. The results indicate that the DoseNet algorithm is a superior alternative to U-Net and fully connected methods for prostate SBRT patients. DoseNet required ~50.1 h to train, and ~0.83 s to make a prediction on a 128  ×  128  ×  64 voxel image. In conclusion, DoseNet is capable of making accurate volumetric dose predictions for non-coplanar SBRT prostate patients, while simultaneously preserving computational efficiency.