Chatter is the main problem that limits the application of industrial robots in the field of machining process. It is critically important to establish an adaptive chatter detection solution for robot machining process and realize the online detection of chatter. However, different from machine tool chatter, the chatter in robotic machining process is more complex to be detected due to the variable stiffness characteristics and weaker stiffness of normal industrial robot, and the existing literature has less research on this problem. This paper presents a comprehensive solution for online chatter detection in robotic machining process. Firstly, in order to detect the chatter in robotic machining process and avoid mode mixing problem in variational mode decomposition (VMD) process, an adaptive variational mode decomposition (AVMD) method based on kurtosis and instantaneous frequency is proposed, which realizes the adaptive selection of the decomposition parameter. Secondly, optimal decomposition parameters are calculated by using genetic algorithm. By optimizing the discrete step length of decomposition parameter, it greatly reduces the optimization time. Last but not least, approximate entropy, energy entropy, and proposed entropy drift coefficient are extracted to distinguish chatter and stable machining state. Simulation and experimental results show that the proposed method can meet the real-time requirements of online detection and detect the occurrence of chatter effectively.