The present study focused on investigating the differences in properties between calcined and milled aluminosilicates with different particle size distributions. Two types of clay, i.e., kaolin and kaolinitic claystone, were subjected to calcination at 750 °C, and subsequent milling to obtain different fractions with distinct particle size distributions. These fractions were then combined with a potassium alkaline activator and quartz sand in a 50:50 weight ratio to form a geopolymer composite. The geopolymer binders were then characterized using a mercury intrusion porosimeter (MIP), scanning electron microscopy (SEM), and a rotary rheometer. Mechanical tests were conducted on the geopolymer composites prepared from aluminosilicates with varying particle size distributions. The findings indicated that aluminosilicates with a finer particle size distribution exhibited higher levels of dissolved aluminum (10,000 mg/kg) compared to samples with coarser particle size distributions (1000 mg/kg). Additionally, as the particle size distribution decreased, the dynamic viscosity of the geopolymer binders increased, while the average pore size decreased. Finally, the mechanical properties of the geopolymer composites derived from both tested aluminosilicates demonstrated a decline in performance as the mean particle size increased beyond 10 µm.