With the increasing amounts of UAVs usage, the supervision of unmanned aerial vehicles (UAV) has become particularly important, and the demand for detecting and following UAVs has grown rapidly. Compared with ground targets, UAVs are more difficult to track because of the high speed of the target and the interference caused by the shadow of either a target or a tracker. In addition, the problem of how to research the target when the target leaves the camera’s field of view has not received sufficient attention. In this paper, a shadow recognition algorithm and the detection network of a target based on deep learning are combined to eliminate the interference caused by shadows. Fuzzy control is applied in the process of following and the dynamic characteristics of UAV are considered in obstacle avoidance, which ensures the stability of the UAV for tracking. Finally, a spatial probability distribution algorithm based on Bayesian prediction is proposed for re-searching a lost target, which can rediscover a target after that target is lost. For this work, a UAV experimental platform has been built and the algorithm feasibility is verified through both simulation and a physical experiment.