While previous neural rendering methods have shown promising results in novel view synthesis, they still face difficulties when dealing with challenging street scenes. For instance, certain objects or content may only appear in a limited number of views, which can result in blurry synthesized images with significant artifacts. To address this critical issue, we propose a novel point-based neural rendering framework that facilitates a photo-realistic street simulation environment. Specifically, we introduce the Neural Street Scene Renderer (NSSR) to handle holes resulting from sparse point clouds. We also present a parallel attention module (PAM) and a multi-scale feature fusion and selection module (FFSM) to recalibrate features. Additionally, we use a point cloud filter and Gaussian noise points to remove outliers or floating points in the air and repair sparse point clouds. Our experimental results on four datasets demonstrate the effectiveness of our proposed approach in synthesizing photo-realistic street scenes.