Group behavior recognition is the task of inferring the collective action of people that have interaction among them in the contextual scenes. The challenge is harder when face with different individual actions. Hierarchical structure model based on deep learning tackles this problem with multi-stages spatio-temporal information modeling. Convolutional neural network (CNN) is designed for extracting the spatial features of scene and person-level. The other component, recurrent neural network (RNN) is aimed to capture temporal feature of the person trajectories in contextual scenes. However, in some prior works, to get the trajectory information in this framework still rely on a third-party tracker that makes the solution not in an end-to-end framework. The exist end-to-end solution incorporates matching strategy in Euclidean space that implicitly tracks the corresponding states as input of RNN unit. In this work, we propose an improved RNN matching strategy by explicitly transform the feature in Euclidean space by distance learning function. Our distance function is based on simple Siamese network with two sub network share the same weights. The network consists of the learned feature based on unsupervised dictionary learning as an intermediate layer between raw input and fully connected layers with non-linear activation and regularization. Our proposed method can yield a little improvement in the applied group behavior recognition framework and yet empirically prove that it can be brought into another task without change the hyper-parameter.