•Using oxide nanoparticles of cobalt in deionized water to heat transfer analysis of boiling process in a nucleate regime on a copper surface.•Improvement of the heat transfer coefficient of pool boiling under a magnetic field for low volume fractions.•Maximum heat transfer of cobalt oxide-deionized water is 58% compared to the deionized water.•This nanofluid can be applied to increase the heat transfer rate at low concentrations. In this paper, the analysis of the boiling process is studied in a nucleate regime on a copper surface through using oxide nanoparticles of cobalt (Co3O4) in deionized water at volume fractions of 0.005, 0.01, 0.05, 0.1, and 0.2% by means of a nucleate boiling device. The nanoparticles’ average size is about 45 nm for cobalt oxide. As the nanofluid concentration increases, the deposition and the boiling surface heat resistance increases, the nucleation points and coefficient of the boiling heat transfer decrease due to deposition. Despite the magnetic field positive effect on dilute liquids, sediment has a more negative effect than the magnetic field on concentrated nanofluids. The results reveal the improvement of the heat transfer coefficient of pool boiling under a magnetic field for low volume fractions. Maximum heat transfer of Co3O4/ DI-water is 58% compared to the DI-water observed at the concentration of 0.005%. This nanofluid can be applied to increase the heat transfer rate at low concentrations.