In building stones that are widely used in the construction of immovable cultural heritage, preservation efforts are necessary due to various weathering processes. The accuracy and effectiveness of the restoration works, specifically for cultural heritage, are important factors. The most basic phase of these works is determining the stone types and preparing geo-lithological maps. However, the time-consuming process and difficult field work may lead to human-induced errors. In this study, a petrographic determination of building stone and a mapping model were developed based on Mask R–CNN in order to prevent human errors. To this end, the city of Konya, Turkey, which is on the UNESCO temporary heritage list, was selected for applying the proposed method. The model was trained with a total of 1800 images collected from nine historic buildings with different ornamental and building stones that constitute the cultural texture of the city. Testing of the model was conducted on the main façade of the Matbah-ı Şerif monument, consisting of 363 building stones with seven different lithologies, for different situations (resolution, shooting distance, and angles). The average precision values for the stone types trained in the model were between 89.10% and 100%, and an accurate lithology determination and map were obtained for each case. These results indicate that the proposed model can provide important bases for restoration works, with its fast and automatic mapping capability as well as its reliable and highly precise lithology determination. Display omitted •The importance of building stone lithology in restoration works.•Determination of building stone lithology via deep learning (DL).•Automatic lithology map via Mask R–CNN.