•A deep long short-term memory (LSTM) network is developed for nonlinear structural response modeling.•Two input-output schemes (LSTM-s and LSTM-f) are presented.•The deep learning model is capable of modeling both elastic and inelastic response of buildings.•An unsupervised learning algorithm is used to cluster the seismic inputs for training enhancement.•The approach was successfully verified by both numerical and experimental examples. This paper presents a comprehensive study on developing advanced deep learning approaches for nonlinear structural response modeling and prediction. Two schemes of the long short-term memory (LSTM) network are proposed for data-driven structural seismic response modeling. The proposed deep learning model, trained on available datasets, is capable of accurately predicting both elastic and inelastic response of building structures in a data-driven fashion as opposed to the classical physics-based nonlinear time history analysis using numerical methods. In addition, an unsupervised learning algorithm based on a proposed dynamic K-means clustering approach is established to cluster the seismic inputs in order to (1) generate the least but the most informative datasets for training the LSTM and (2) improve the prediction accuracy and robustness of the model trained with limited data. The performance of the proposed approach is successfully demonstrated through three proof-of-concept studies that include a nonlinear hysteretic system, a real-world building with field sensing data, and a steel moment resisting frame. The results show that the proposed LSTM network is a promising, reliable and computationally efficient approach for nonlinear structural response prediction, and offers significant potential in seismic fragility analysis of buildings for reliability assessment.