Infrared (IR) small target detection under low signal-to-clutter ratio (SCR) is a fundamental and important problem in some critical missions like IR search and tracking (IRST). Though there exist a lot of works using local contrast measure (LCM) to detect the target, their performance are still unsatisfying due to the imperfect knowledge of target structure. In this letter, a novel IR small target detection method utilizing halo structure prior (HSP)-based LCM (HSPLCM) is proposed, which adequately considers the structure characteristic of the target. Specifically, through weighting the raw IR image via image structure tensor, we put forward a simple but useful image prior (named as HSP) which reflects the unique structural feature of the target to distinguish the real target and other background clutters. Afterward, based on this prior an effective LCM method is constructed to detect the IR small target, which can enhance the target and suppress the background clutters simultaneously. Furthermore, we extend the proposed algorithm to its multiscale version to solve the target scale uncertainty issues. Extensive experimental results have demonstrated that our proposed method favorably outperforms the state-of-the-arts.