Purpose We assessed the accuracy of quantitative susceptibility mapping in a gadolinium balloon phantom with a large range of susceptibility values and imaging resolutions at 1.5 and 3 Tesla (T). Theory and Methods The phantom contained sources with susceptibility values of 0.4, 0.8, 1.6, and 3.2 ppm and was imaged at isotropic resolutions of 0.7, 0.8, 1.2, and 1.8 mm. Numerical simulations were performed to match the experimental findings. Voxel sensitivity effects were used to explain the susceptibility underestimations. Results Both phantom data and simulation demonstrated that systematic underestimation of the susceptibility values increased with voxel size, field strength, and object susceptibility. Conclusion The underestimation originates from the signal formation in a voxel, which can be described by the voxel sensitivity function. The amount of underestimation is thus affected by imaging resolution, magnitude contrast, image filtering, and details of the susceptibility inclusions such as the susceptibility value and geometry. High‐resolution imaging is therefore needed for accurate reconstruction of QSM values, especially at higher susceptibilities. Magn Reson Med 78:1080–1086, 2017. © 2016 International Society for Magnetic Resonance in Medicine