Recently, in this article, due to pervasive usability, smartphone-based human activity recognition (HAR) has witnessed significant development in smart health. Meanwhile, deep recurrent neural network (DRNN) shows a strong ability to automatically extract features from the time-series data, and therefore, DRNN-based HAR schemes have achieved more effective recognition (i.e., recognition accuracy) than those adopting the traditional machine learning. However, the efficiency in training and recognition (in terms of running time) has not fully taken into account, especially for resource-constraint smartphones. To solve the above issue, we propose the PSDRNN and tri-PSDRNN schemes that employ the explicit feature extraction before DRNN. Specifically, considering that the power spectral density (PSD) feature can capture the frequency characteristics and meanwhile retain the successive time characteristics of data gathered from smartphone accelerometer, PSD feature vectors are, respectively, extracted from linear accelerations and triaxle accelerations and explicitly used as the input to the following DRNN classification model. Thorough experiments based on a real dataset demonstrate that the PSDRNN can achieve the comparable effectiveness as the xyz -DRNN (the most accurate DRNN-based HAR scheme only using acceleration data), and the average recognition and training time were reduced by 56% and 80%, respectively. Moreover, tri-PSDRNN advantages over the xyz -DRNN in terms of recognition accuracy, and the running time is still lower than the xyz -DRNN. Besides, our proposed PSDRNN scheme achieved superiority in the recognition of complex transition activities.