The accurate separation between leaf and woody components from terrestrial laser scanning (TLS) data is vital for the estimation of leaf area index (LAI) and wood area index (WAI). Here, we present the application of deep learning time series separation of leaves and wood from TLS point clouds collected from broad-leaved trees. First, we use a multiple radius nearest neighbor approach to obtain a time series of the geometric features. Second, we compare the performance of Fully Convolutional Neural Network (FCN), Long Short-Term Memory Fully Convolutional Neural Network (LSTM-FCN), and Residual Network (ResNet) on leaf and wood classification. We also compare the effect of univariable (UTS) and multivariable (MTS) time series on classification accuracy. Finally, we explore the utilization of a class activation map (CAM) to reduce the black-box effect of deep learning. The average overall accuracy of the MTS method across the training data is 0.96, which is higher than the UTS methods (0.67 to 0.88). Meanwhile, ResNet spent much more time than FCN and LSTM-FCN in model development. When testing our method on an independent dataset, the MTS models based on FCN, LSTM-FCN, and ResNet all demonstrate similar performance. Our method indicates that the CAM can explain the black-box effect of deep learning and suggests that deep learning algorithms coupled with geometric feature time series can accurately separate leaf and woody components from point clouds. This provides a good starting point for future research into estimation of forest structure parameters.