In rainy conditions, especially at night with low illumination, the visual of images obtained by outdoor computer vision systems is degraded significantly, leading to a significant negative effect on the work of the outdoor computer vision system. In this paper, we develop a new rainy image model to describe rain scenes at night with low illumination. From this model, we propose a joint deep neural network-based method for single nighttime rainy image enhancement. First, a decom-net based on Retinex theory is employed for image decomposition, and the purpose of this sub-net is to extract the reflection image and the illumination image from the input image. Then, an enhancement net is proposed for illumination adjustment. The goal of this sub-net is to remove the negative effect (low visual) caused by low illumination. Finally, a symmetric sub-net termed multi-stream network-based contextual autoencoder is developed, where rain features are directly learned from the enhanced nighttime rainy images in a recurrent way. The goal of this sub-net is to effectively remove rain streaks from the illumination-enhanced image. The experimental results show the advantage and effectiveness of the proposed method, and evident improvements over existing state-of-the-art methods are obtained with the proposed method.