Considering the role of behavioral and environmental factors on road accidents and traffic intensities, the characterization of vehicle use and driver behavior opens new opportunities for safety improvements and energy savings. Thus, the objective of this work was to identify driving behavior patterns for several driving environments (based on street level and weather conditions) from real-world driving data and to analyze how these driving environments influenced driving behavior. The case study for this work was Lisbon, Portugal, where driving data from 47 drivers were collected with on-board data loggers for at least 6 months. The results show that both street level and weather conditions impact driving behavior significantly. However, while for rainy conditions, the results provide evidence that drivers tend to drive more calmly (average speed is 22% lower for heavy rain than without rain, while positive and negative accelerations decrease by 8% and 11%, respectively), when considering the influence of street level more local streets (level 2, 3 and 4 streets) are the ones that present more aggressive driving patterns in terms of acceleration (30–40% increase from level 1 to level 4 streets). This work contribution regards the quantification of the impacts of driving environment on driving behavior, providing evidence that rain conditions significantly affect driving behavior, leading drivers to adjust their driving behavior to the driving environment. However, regarding street level, the differences found in driving behavior seem to be more a consequence of the infrastructure characteristics than an adjustment of driving behavior.