The article presents a systematic study of Sb-doped Znsub.1−xMgsub.xO layers, with various concentrations of Mg, that were successfully grown by plasma-assisted MBE on polar a- and c-oriented and non-polar r-oriented sapphire substrates. X-ray diffraction confirmed the polar c-orientation of alloys grown on c-and a-oriented sapphire and non-polar structures grown on r-oriented substrates. A uniform depth distribution of the Sb dopant at level of 2 × 10sup.20 cmsup.−3 was determined by SIMS measurements. Raman spectroscopy revealed the presence of Sb-related modes in all samples. It also showed that Mg alloying reduces the compressive strain associated with Sb doping in ZnO. XPS analysis indicates that the chemical state of Sb atoms in ZnMgO is 3+, suggesting a substitutional position of Sbsub.Zn, probably associated with two Vsub.Zn vacancies. Luminescence and transmission spectra were measured to determine the band gaps of the Znsub.1−xMgsub.xO layers. The band gap energies extracted from the transmittance measurements differ slightly for the a, c, and r substrate orientations, and the differences increase with increasing Mg content, despite identical growth conditions. The differences between the energy gaps, determined from transmission and PL peaks, are closely correlated with the Stokes shift and increase with the Mg content in the analyzed series of ZnMgO layers.