Crowdsourced indoor localization methods have grasped much attention in recent years as a method of reducing the cost of constructing the fingerprint database. In a crowdsourcing environment, however, the localization system is vulnerable to malicious attacks, which possibly lead to serious localization errors. In this paper, we conclude the potential attacks during fingerprint database updates and online inference phases and propose a secure indoor crowdsourced localization system, BERT-ADLOC, based on BLE fingerprints. Our system consists of two main parts: adversarial sample discriminator BERT-AD and indoor localization model BERT-LOC. Our proposed BERT-AD recognizes fake fingerprints during the database update phase, while BERT-LOC defends against attacks online, in which valid beacons are moved or malicious beacons are deployed. A tailored BERT model is introduced to extract deep hidden features through the self-attention mechanism. Our experiments show that BERT-ADLOC achieves a good localization performance against adversaries both in the fingerprint database update phase and online inference phase.