Cavity optomechanical (COM) sensors, as powerful tools for measuring ultraweak forces or searching for dark matter, have been implemented to date mainly using linear COM couplings. Here, quantum force sensing is explored by using a quadratic COM system which is free of bistability and allows accurate measurement of mechanical energy. We find that this system can be optimized to achieve a force sensitivity \(7\) orders of magnitude higher than any conventional linear COM sensor, with experimentally accessible parameters. Further integrating a quadratic COM system with a squeezing medium can lead to another \(3\) orders enhancement, well below the standard quantum limit and reaching the zeptonewton level. This opens new prospects of making and using quantum nonlinear COM sensors in fundamental physics experiments and in a wide range of applications requiring extreme sensitivity.