Purpose Phantom‐based validation of diffusion‐weighted image processing techniques is an important key to innovation in the field and is widely used. Openly available and user friendly tools for the flexible generation of tailor‐made datasets for the specific tasks at hand can greatly facilitate the work of researchers around the world. Methods We present an open‐source framework, Fiberfox, that enables (1) the intuitive definition of arbitrary artificial white matter fiber tracts, (2) signal generation from those fibers by means of the most recent multi‐compartment modeling techniques, and (3) simulation of the actual MR acquisition that allows for the introduction of realistic MRI‐related effects into the final image. Results We show that real acquisitions can be closely approximated by simulating the acquisition of the well‐known FiberCup phantom. We further demonstrate the advantages of our framework by evaluating the effects of imaging artifacts and acquisition settings on the outcome of 12 tractography algorithms. Conclusion Our findings suggest that experiments on a realistic software phantom might change the conclusions drawn from earlier hardware phantom experiments. Fiberfox may find application in validating and further developing methods such as tractography, super‐resolution, diffusion modeling or artifact correction. Magn Reson Med 72:1460–1470, 2014. © 2013 Wiley Periodicals, Inc.