•A rotor vibration potato-soil separation method is proposed.•A coupling simulation model of rotor vibration potato-soil separation adopted to DEM-MBD is built.•The optimal parameters of the rotor vibration potato-soil separation device are determined via simulation experiment.•The model and optimal parameters of rotor vibration potato-soil separation is verified via field experiment. A rotor vibration potato-soil separation device (RVPSD) is proposed in view of poor potato-soil separation and higher potato damage rate. Separation efficiency between potatoes and soil and the potato damage rate are selected as evaluation indicators, and a coupling simulation model of potato-soil separation based on Discrete Element Method (DEM) and Multibody Dynamics (MBD) is built up according to structure and working principle of the separation device. The optimal combination of working parameters of the RVPSD is obtained via simulation experiment. The results show that the optimal working parameters of vibration point position, conveying speed of potato-soil separation elevating chain, rotor amplitude and rotor vibration frequency are 646.5 mm, 1.08 m/s, 26.7 mm and 5.9 Hz respectively. The field validation experiment is carried out based on the optimal combination parameters. The results show that the potato-soil separation efficiency and potato damage rate of the RVPSD are 97.8 % and 1.16 % respectively, the field experiment results are basically consistent with the simulation results, which proves the correctness of the simulation model. It can provide theoretical reference for rotor vibration potato-soil separation process simulation and device parameter optimization.