•Gold nanofluid and butanol self-rewetting fluid represent a new and innovative class of heat transfer fluids for advanced cooling.•An experimental analysis is performed on the capillary evaporator section of capillary heat pipe.•Thermal performance is investigated by measuring the thermal resistance of the evaporator.•Gold nanofluid, with 1% Cv, showed a decreasing in the thermal resistance of the system by 13%.•A mixture of self-rewetting gold nanofluid showed the highest thermal performance by decreasing the thermal resistance by 22%.•The phase change is visualized by infrared camera.•This improvement has been explained due to the thermo-physical properties of nanofluid and self-rewetting fluid. This paper presents an experimental analysis of the phenomenon of phase change inside a porous medium using different types of working fluids. It represents the impact of these fluids on improving the characteristics of heat and mass transfer in a Capillary Heat Pipe (CHP). In this study, gold nanoparticles (5 nm in diameter with 1% Cv), a self-rewetting binary solution (butanol with 3% Cv), and a mixture of self-rewetting butanol and gold nanofluid are considered to be the operating fluids within the CHP. The experiments are carried out after designing and developing the capillary heat pipe section. It consists of a water tank with a pump, an evaporator attached to a copper porous medium on which thermocouples and power supplies are placed. The experimental results showed the positive influence of gold nanoparticles on the thermal system's performance by reducing the thermal resistance by 13% compared to pure water as the base working fluid. In addition, a self-rewetting butanol solution showed improvement in the performance of the capillary evaporator by decreasing its casing temperature. While a mixture of self-rewetting butanol solution (3 % Cv) and gold nanofluid (1% Cv) exhibited the best performance of heat and mass transfer performance by reducing the thermal resistance of the system by approximately 22 %. To explain the mechanism for improving heat transfer, the phase change phenomenon was visualized by an infrared camera for the three working fluids. It is shown that as the applied power increases, the shape of the vapor pocket developed in the wick also increases, for pure water, until it reaches a stable form. Whereas, with respect to nanofluid and self-rewetting fluid, the shape of the vapor pockets was smaller than that of pure water allowing more efficient mass and heat transfer. The thermophysical properties of these fluids such as thermal conductivity, stability, surface tension, Marangoni, wettability, and capillary forces were presented to ensure and validate the decrease in the vapor pocket as well as the enhancement of the CHP thermal system.