Abstract Measuring the flatness error of large precision workpieces quickly and accurately is a difficult problem. A new method for preprocessing flatness measurement data based on MSE (mean squared error) is proposed. A mathematical model of a new data preprocessing method was established, and the mathematical formula for model solving was derived in detail. The data were measured by digital level on the plane of the granite base with dimensions of 2340 m×1540 mm. The new method and SmartLevel (basic measurement system of the level computer) were used to calculate and process the data. The flatness errors after diagonal evaluation were 4.07 μm and 3.90 μm, respectively. The relative error of the two was 4.36%, which confirmed the reliability and accuracy of the new method. The data results show that this method can be effectively used for the engineering measurement of the flatness of large precision workpieces.