Purpose This study was a systematic evaluation across different and prominent diffusion MRI models to better understand the ways in which scalar metrics are influenced by experimental factors, including experimental design (diffusion‐weighted imaging DWI sampling) and noise. Methods Four diffusion MRI models—diffusion tensor imaging (DTI), diffusion kurtosis imaging (DKI), mean apparent propagator MRI (MAP‐MRI), and neurite orientation dispersion and density imaging (NODDI)—were evaluated by comparing maps and histogram values of the scalar metrics generated using DWI datasets obtained in fixed mouse brain with different noise levels and DWI sampling complexity. Additionally, models were fit with different input parameters or constraints to examine the consequences of model fitting procedures. Results Experimental factors affected all models and metrics to varying degrees. Model complexity influenced sensitivity to DWI sampling and noise, especially for metrics reporting non‐Gaussian information. DKI metrics were highly susceptible to noise and experimental design. The influence of fixed parameter selection for the NODDI model was found to be considerable, as was the impact of initial tensor fitting in the MAP‐MRI model. Conclusion Across DTI, DKI, MAP‐MRI, and NODDI, a wide range of dependence on experimental factors was observed that elucidate principles and practical implications for advanced diffusion MRI. Magn Reson Med 78:1767–1780, 2017. © 2017 The Authors Magnetic Resonance in Medicine published by Wiley Periodicals, Inc. on behalf of International Society for Magnetic Resonance in Medicine. This is an open access article under the terms of the Creative Commons Attribution‐NonCommercial License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes.