Two nonperturbative methods such as Monte-Carlo simulation (MC) and Transfer-Matrix Finite-Size-Scaling calculations (TMFSS) have been used to study the phase transition of the spin-3/2 ​Blume–Emery–Griffiths model (BEG) with quadrupolar and antiferromagnetic next-nearest-neighbor exchange interactions. Ground state and finite temperature phase diagrams are obtained by means of these two methods. New degenerate phases are found and only second order phase transitions occur for all values of the parameter interactions. No sign of the intermediate phase is found from both methods. Critical exponents are also obtained from TMFSS calculations. Ising criticality and nonuniversal behaviors are observed depending on the strength of the second neighbor interaction. •A spin-3/2 Blume–Emery–Griffiths model with second neighbor exchange is proposed.•Monte-Carlo simulations and Transfer-matrix Finite-size-scaling methodology are used.•New phases appear in this model and second order transitions are found.•The critical behavior has been analyzed by calculating the exponent nu.•Ising critical universality and nonuniversal behavior are found.