•Kesterite and stannite phases of Ag2MgSn(S/Se)4 have been studied for the first time.•Kesterite and stannite phases of Ag2MgSnS4 is more stable than Ag2MgSnSe4.•Ag2MgSnS4 (Sta) phase has bandgap of 1.41 eV with highest value of absorption coefficient.•Optical properties of Ag2MgSnS4 (Sta) exhibit similar characteristic as Ag2MgSnSe4 (Sta). In this work, generalized gradient approximation (GGA) and the Heyd-Scuseria-Ernzerhof (HSE) hybrid functional within the framework of density functional theory (DFT) are used to investigate the structural, electronic and optical properties of Ag2MgSn(S/Se)4 quaternary chalcogenides in kesterite and stannite phases. The kesterite and stannite phases of Ag2MgSn(S/Se)4 are optimized and the values of lattice constants are calculated. The predicted energy bandgaps are found to be direct in nature along the Γ direction, with highest bandgap energy of less than 1.64 eV for Ag2MgSnS4 in kesterite phase. The total and partial density of states are analysed in detail. The optical properties such as imaginary and real parts of dielectric function, refractive index, reflectivity and extinction coefficient are also discussed. Further, in order to understand the light trapping capacity, we examine the absorption coefficients of Ag2MgSnS4 and Ag2MgSnSe4 materials and explore their applicability as solar cell absorber layers. The predicted values of all parameters will help researchers to carry out further investigations on these materials for photovoltaic applications.