•This paper proposes a lightweight farming behaviors recognition algorithm.•In the plantation of video-based farming behavior recognition, the farming behavior recognition method proposed in this paper has higher recognition accuracy and fewer parameters than current farmer behavior recognition methods.•Promote the development of agricultural intelligence and standardization. In properly managed plantation farms, farming behaviors of workers should be properly recorded. Both the accuracy of the model and the total amount of resources required during processing need to be considered, considering farms only have ordinary computers and they are difficult to have large computing servers. This paper proposes a lightweight method FWNet for recognizing farming behaviors of plantation workers. Firstly, a dataset (FBD) of farming behaviors is established, which includes 1154 video samples involving four types of farming behaviors: spraying pesticides, hoeing, weeding and transplanting seedlings. Secondly, in FWNet, we refer to (2 + 1) D convolution and residual structure, adopt a smaller network input shape and a shallower number of network layers, and use Swish as the activation function. In experiments, FWNet's accuracy, F1-score and mAP are 97.5%, 97.41% and 97.31% which are higher than P3D, R3D, R(2 + 1) D, EPCI-LSTM models. Parameters amount and latency are reduced to 3.20 M and 28.91 ms, far less than baselines. Experiment results indicate that FWNet can recognize farming behaviors with high accuracy and faster.