A challenge still to be overcome in the field of visual perception for vehicle and robotic navigation on heavily damaged and unpaved roads is the task of reliable path and obstacle detection. The vast majority of the researches have scenario roads in good condition, from developed countries. These works cope with few situations of variation on the road surface and even fewer situations presenting surface damages. In this paper we present an approach for road detection considering variation in surface types, identifying paved and unpaved surfaces and also detecting damage and other information on other road surfaces that may be relevant to driving safety. Our approach makes use of Convolutional Neural Networks (CNN) to perform semantic segmentation, we use the U-NET architecture with ResNet34, in addition we use the technique known as Transfer Learning, where we first train a CNN model without using weights in the classes as a basis for a second CNN model where we use weights for each class. We also present a new Ground Truth with image segmentation, used in our approach and that allowed us to evaluate our results. Our results show that it is possible to use passive vision for these purposes, even using images captured with low cost cameras.