•Automatic driving stress detection system in physiological records was proposed.•Features were extracted from multimodal analysis.•Efficient feature selection and reduction methods were employed.•Several kernel-based classifiers were adopted and compared.•Our proposed method shows a promising application in intelligent vehicle systems. Monitoring driving status has great potential in helping us decline the occurrence probability of traffic accidents and the aim of this research is to develop a novel system for driving stress detection based on multimodal feature analysis and kernel-based classifiers. Physiological signals such as electrocardiogram, galvanic skin response and respiration were record from fourteen drives executed in a prescribed route at real drive environments. Features were widely extracted from time, spectral and wavelet multi-domains. In order to search for the optimal feature sets, Sparse Bayesian Learning (SBL) and Principal Component Analysis (PCA) were combined and adopted. Kernel-based classifiers were employed to improve the accuracy of stress detection task. Analysis I used features from 10s intervals of data which were recorded during well-defined rest, highway and city driving conditions to discriminate three levels of diving stress achieving an averaging accuracy over 99% at per-drive level and 89% in cross-drive validation. Analysis II made continuous stress evaluation throughout a complete driving test attaining a high coincidence with the true road situation especially at the switching interval of traffic conditions. Experimental results reveal that different levels of driving stress can be characterized by specific set of physiological measures. These physiological measures could be applied to in-vehicle intelligent systems in various approaches to help the drivers better manage their negative driving status. Our design scheme for driving stress detection could also facilitate the development of similar in-vehicle expert systems, such as driver's emotion management, driver's sleeping onset monitoring, and human-computer interaction (HCI).