Object tracking is a problem about semi-supervised learning with insufficient data set. In the field of military navigation and security of public life, it is widely used to take the place of human beings. In this paper, we come up with a new algorithm based on Bayesian, CNN and PLK optical flow, which is called COB method, for object tracking problems. With the idea of track-by-detect, we cascade CNN after PLK optical flow and integrate them in a Bayesian method. Most importantly our method is proposed with an adaptive integrating method to reduce the influence of over-fitting. The integrator also introduces the competition mechanism between tracker and detector, so that the algorithm is able to update the classifier with online learning. Besides, the regularization of deep learning is used to solve the blind spots of classifier. The experimental results show that the algorithm is more robust than the previous work.