Olive is considered one of the oldest and the most important cultivated fruit trees in Albania. In the present study, the genetic diversity and structure of Albanian olive germplasm is represented by a set of 194 olive genotypes collected in-situ in their natural ecosystems and in the ex-situ collection. The study was conducted using 26 microsatellite markers (14 genomic SSR and 12 Expressed Sequence Tag microsatellites). The identity analysis revealed 183 unique genotypes. Genetic distance-based and model-based Bayesian analyses were used to investigate the genetic diversity, relatedness, and the partitioning of the genetic variability among the Albanian olive germplasm. The genetic distance-based analysis grouped olives into 12 clusters, with an average similarity of 50.9%. Albanian native olives clustered in one main group separated from introduced foreign cultivars, which was also supported by Principal Coordinate Analysis (PCoA) and model-based methods. A core collection of 57 genotypes representing all allelic richness found in Albanian germplasm was developed for the first time. Herein, we report the first extended genetic characterization and structure of olive germplasm in Albania. The findings suggest that Albanian olive germplasm is a unique gene pool and provides an interesting genetic basis for breeding programs.