•Heat transfer performance of nanofluids in a plate heat exchanger is investigated.•The maximum enhancement in average Nu is 22.6% for 1.0 wt.% Fe3O4-water nanofluid.•The optimum concentration for thermal enhancement is 0.5 wt.% for CuO nanofluid.•Empirical formulas of experimental Nu are derived based on experimental data.•Fe3O4-water nanofluid is a promising heat transfer medium for solar energy systems. In this paper, a corrugated plate heat exchanger in solar energy systems is used to investigate heat transfer and fluid flow characteristics of various nanofluids. By adding various nanoparticles (Al2O3-30 nm, SiC-40 nm, CuO-30 nm and Fe3O4-25 nm) into the base fluid, effects of nanofluid types and particle concentrations (0.05 wt.%, 0.1 wt.%, 0.5 wt.% and 1.0 wt.%) on the thermal performance of the plate heat exchanger are analyzed at flow rates in the range of 3–9 L/min. Results indicate that both heat transfer enhancement and pressure drop for nanofluids show significant increases compared to the base fluid. The Fe3O4-water and CuO-water nanofluids show the best and the worst thermal performances of the plate heat exchanger, respectively. When 1.0 wt.% Fe3O4-water nanofluid is used as the working fluid, compared to DI-water, the convective heat transfer coefficient is increased by 21.9%. However, an increase of 10.1% in pressure drop is obtained for the 1.0 wt.% Fe3O4-water nanofluid. Finally, empirical formulas of experimental Nusselt number are obtained based on the experimental data. A new way to predict the thermal performance for various nanofluids in heat transfer systems is provided.