Fluorite single crystals, CaF2 are multifunctional crystals that have potential applications in environmental, thermo-luminescent and accidental dosimetry. A comprehensive investigation of the co-located naturally occurring colourless and coloured (yellow, sea-green, purple, blue, black) fluorites (CaF2) doped with bivalent and multivalent cations was carried out for the cationic and anionic sub-lattice defects generated due to natural irradiation from associated actinide containing pyrochlore grains. Detailed spectroscopic analyses have been used to map the various cationic and anionic defect structures such as O3− and O2--Vacancy type defect complexes in yellow fluorites, F-centres and irradiation induced metallic Ca in purple, blue and black fluorites. The confinement of the radiation damage only to the anionic sub-lattice was explained in terms of higher displacement cross-section for F (59 b) as compared to Ca (33 b) as well as lower energy requirement to create F displacements and vacancies. It is observed that the fluorite retained its phase stability under natural radiation exposure and elemental substitutions. Display omitted •Description of the colour variations in the co-located natural fluorite crystals in terms of cationic and anionic sub-lattice defects.•Role of radiation induced metallic calcium colloids on the observed colour variation in the fluorites is explained.•Radiation damage is confined to anionic sub-lattice only owing to higher displacement cross section for F− as compared to Ca2+.•Stable anionic defects in fluorites without radiation exposure are identified.