Imaging parameters of photoacoustic breast imaging systems such as the spatial resolution and imaging depth are often characterized with phantoms. These objects usually contain simple structures in homogeneous media such as absorbing wires or spherical objects in scattering gels. While these kinds of basic phantoms are uncluttered and useful, they do not challenge the system as much as a breast does, and can thereby overestimate the system’s performance. The female breast is a complex collection of tissue types, and the acoustic and optical attenuation of these tissues limit the imaging depth, the resolution and the ability to extract quantitative information. For testing and challenging photoacoustic breast imaging systems to the full extent before moving to in vivo studies, a complex breast phantom which simulates the breast’s most prevalent tissues is required. In this work we present the first three dimensional multi-layered semi-anthropomorphic photoacoustic breast phantom. The phantom aims to simulate skin, fat, fibroglandular tissue and blood vessels. The latter three are made from custom polyvinyl chloride plastisol (PVCP) formulations and are appropriately doped with additives to obtain tissue realistic acoustic and optical properties. Two tumors are embedded, which are modeled as clusters of small blood vessels. The PVCP materials are surrounded by a silicon layer mimicking the skin. The tissue mimicking materials were cast into the shapes and sizes expected in the breast using 3D-printed moulds developed from a magnetic resonance imaging segmented numerical breast model. The various structures and layers were assembled to obtain a realistic breast morphology. We demonstrate the phantom’s appearance in both ultrasound imaging as photoacoustic tomography and make a comparison with a photoacoustic image of a real breast. A good correspondence is observed, which confirms the phantom’s usefulness.