Display omitted For clean upgrading of heavy oil, catalytic treatment in supercritical water (SC-H2O) is known as an attractive green methodology. To gain quantitative and qualitative insights of heavy oil upgrading in SC-H2O, optimization of operating conditions was performed where higher heating value (HHV), low sulfur contents as well as low rate of coke formation are desirable. A series of experiments for upgrading the vacuum residue (VR) feed in SC-H2O were designed with response surface methodology (RSM) and performed in presence of formic acid and hematite iron oxide nanoparticles as hydrogen donor and catalyst, respectively. The key process parameters including temperature, water, formic acid, catalyst contents and reaction time were optimized by the mentioned statistical methodology. The proposed optimization model displayed an acceptable correlation between the predicted and experimental results. From the experimental results it was apparent the incorporation of hematite nanoparticles with SC-H2O and formic acid exhibited higher degree of light fraction product and low coke formation derived from oxygen vacancy and better catalyst reducibility in SC-H2O.