The emerging metasurfaces with the exceptional capability of manipulating an arbitrary wavefront have revived the holography with unprecedented prospects. However, most of the reported metaholograms suffer from limited polarization controls for a restrained bandwidth in addition to their complicated meta-atom designs with spatially variant dimensions. Here, we demonstrate a new concept of vectorial holography based on diatomic metasurfaces consisting of metamolecules formed by two orthogonal meta-atoms. On the basis of a simply linear relationship between phase and polarization modulations with displacements and orientations of identical meta-atoms, active diffraction of multiple polarization states and reconstruction of holographic images are simultaneously achieved, which is robust against both incident angles and wavelengths. Leveraging this appealing feature, broadband vectorial holographic images with spatially varying polarization states and dual-way polarization switching functionalities have been demonstrated, suggesting a new route to achromatic diffractive elements, polarization optics, and ultrasecure anticounterfeiting.