Devising a representation suitable for characterizing human actions on the basis of a sequence of pose estimates generated by an RGBD sensor remains a research challenge. We here provide two insights into this challenge. First, we show that discriminate sequence of poses typically occur over a short time window, and thus we propose a simple-but-effective local descriptor called a trajectorylet to capture the static and kinematic information within this interval. Second, we show that state of the art recognition results can be achieved by encoding each trajectorylet using a discriminative trajectorylet detector set which is selected from a large number of candidate detectors trained through exemplar-SVMs. The action-level representation is obtained by pooling trajectorylet encodings. Evaluating on standard datasets acquired from the Kinect sensor, it is demonstrated that our method obtains superior results over existing approaches under various experimental setups. •We design the trajectorylet that captures the static and dynamic pose information withina short interval.•We obtain discriminative patterns of action instances by encoding each trajectorylet using a discriminative trajectorylet detector set.•This method generalizes well to different parameter settings and datasets.