Siamese network based the tracker is a hot topic in the field of visual object tracking. However, Siamese trackers still have a robustness gap compared with state‐of‐the‐art algorithms. Therefore, focusing on the issue, this letter adds Frequency Channel Attention (FCA) and adaptive template feature map to the framework of Siamese neural network. FCA can enhance feature representation of effective channels and improve feature discrimination by modeling the correlation between each channel of the image. In this algorithm, by theoretical analysis and experimental validation, restriction is broken through a simple yet effective FCA network sampling strategy and a Siamese‐FCA tracker with significant performance gain is successfully trained. Meanwhile, in order to better adjust the proportion between target and background, the tracker selects suitable size of the target feature map. Moreover, extensive ablation studies are conducted to demonstrate the effectiveness of the proposed tracker. Fairly, the experimental results of five test benchmarks, including OTB2013, OTB2015, VOT2016, VOT2018 and UAV123 datasets, shows that the proposed algorithm performs outstanding. In particular, the issue of similarity and small target tracking failure is overcome. The average running frame rate reaches 86 frames per second, which can meet the real‐time requirements.