This paper presents a new procedure to optimize the geometric parameters of a n-type coaxial HPGe detector. It is based on a statistical technique called “Design of Experiments” (DoE). This technique aims to identify the most influential parameters and to determine the optimal configuration. In this work, The effects of each parameter on the detector responses have been investigated by a fractional factorial design. Only the most influential factors contributing to the detector response have been selected. Precise modeling of these factors was then performed using a full factorial design. Based on the results obtained from this design, the full energy peak efficiencies according to the geometric parameters were modeled by a multiple-linear regression. These models have been statistically validated by analysis of variance (ANOVA). The optimal combination of the geometric parameters has been identified using the desirability function approach, which is a useful tool to optimize multi-response problems. A verification test was performed to validate the results obtained. It was observed that the relative deviation found between experimental and simulated values was less than 5%. •Optimization of geometrical dimensions of an HPGe detector.•Application of Monte Carlo simulation and “Design of Experiments” technique.•Investigating influence of each detector parameter on the FEPE.•Achieving good agreement between the measured and the simulated results.