Learning-based TOA-UWB localization methods have been developed rapidly in recent years and achieve state-of-the-art localization results in complex scenes. However, they still suffer from two drawbacks: 1) biased measurements with large noise are not suppressed effectively, and 2) geometric information which is important for UWB localization is not considered. Thus, we propose two twofold strategies in this paper to overcome these issues: 1) A novel deep attention-based network is proposed. In this network, we introduce the transformer encoder to learn the weights of different ranging measurements, and thus suppress the adverse impact of the biased measurements. Meanwhile, the anchor positions including the geometric information are introduced into the network by an embedding module. 2) We present a novel learning strategy to train the proposed network. This learning strategy both considers the pre-collected ground-truth and the geometric constraints of UWB sensors. Through these two strategies, large measurement noise is further suppressed, while the geometric information and constraints are also developed for the proposed network. Therefore, the localization performance is improved. We build real-world experiments in a narrow and complex indoor scene to demonstrate the advantages of our proposed method compared to the state-of-the-art learning-based method.