Different types and ratios of surfactant, co-surfactant, and oil phase, have a greater impact on nanoemulsion preparation. The presence of surfactants in the nanoemulsion can reduce surface tension and characteristic stability. In this study, four groups of oil-in-water (O/W) nanoemulsions (NEs) with different ratios of surfactant and co-surfactant, and two oils were formulated as carriers of Rhodiola rosea. The variable optimization was investigated and then indicated as optimization group A (Opt A) with the formula of 10% of transcutol, 16.63% of tween 80, Opt B with 10% of tween 80, 29.87% of span 80, Opt C with 28.42% of transcutol, 30% of labrasol, and Opt D with 30% of transcutol, 30% of tween 80. Labrafac and soybean oil were used as the oil phase. The optimized formula using the response surface method (RSM) by design expert software showed the ideal conditions with a higher desirability score. Desirability score are 0.72% (Opt A), 0.81% (Opt B), 0.76% (Opt C) and 0.98% (Opt D), the desirability rating close to 1 indicates a high possibility that the projected values would closely match the experimental results for the optimum formula. All of the optimized formulation were also checked for the characteristics of nanoemulsion including particle size, polydispersity index (PDI), zeta potential, viscosity, encapsulation efficiency, transmission electron microscope (TEM), antioxidant activity, skin irritation test and stability studies. Our study provides a promising combination of surfactant-co-surfactant and oil phases to produce a stable nanoemulsion that can be used in pharmaceuticals and cosmetics in the future. Display omitted •Polysaccharides and phenolic derivatives as primary constituents in R. rosea extract.•The surfactant exerts a discernible influence on the physical properties of R.rosea NE.•R.rosea NE have demonstrated commendable stability throughout storage.•All of the optimization formulations exhibited notable antioxidant activity.