The use of a swarm of low-cost, mission-specific drones to form a Flying Ad-hoc Network (FANET) has literally become a ’hotspot’ in the drone community. A number of studies have been conducted on how to achieve a FANET, but few have considered the security perspectives of this subject. FANET’s unique features have made it difficult to strengthen its defense against ever-changing security threats. Today, more and more FANET applications are implemented into civil airspace, but the development of FANET security has remained unsatisfactory. In this paper, we try to address this issue by proposing a new Intrusion Detection System (IDS), an hybrid method based on both spectral traffic analysis and a robust controller / observer for anomaly estimation inside UAV networks. The proposed hybrid method considers, as a preliminary step, a statistical signature of the traffic exchanged in the network. By examining the resulted signatures, the differences are used to select the accurate model for accurate estimation of that abnormal traffic. The proposed IDS design has been successfully applied to some relevant practical problems such as ad hoc networks for aerial vehicles, and the effectiveness is illustrated by using real traffic traces including Distributed Denial of Service (DDoS) attacks. Our first results show promising perspectives for Intrusion Detection System (IDS) in UAV communication networks. Indeed, different types of anomaly have been considered and they are all accurately detected by the intrusion detection process we propose in this paper. Finally, both simulation-based validation and real-time real-world based implementation of our IDS are described in this article.