With the continuous application and development of marine navigation technology, the accuracy requirements of navigation information for ships are becoming higher and higher. The Inertial Navigation System (INS) has been widely used due to their advantages in accuracy and stability. Star sensor, which can provide high-precision carrier attitude quaternion information, has gradually gained more and more applications. With the continuous application and development of various integrated navigation systems, based on the above two sensors, the Star sensor/INS combined attitude positioning system has the advantages in high-precision, anti-interference and autonomy. However, sensor errors and system initial errors will affect the performance of this integrated navigation system significantly. To compensate the above errors is a common way to improved the accuracy of the integrated navigation system, and the error calibration is the premise of error compensation. Different from the traditional Star sensor/INS integrated navigation system which carries out data fusion on the computer, this study is based on the actual installation in which the Star sensor is installed directly and fixedly on the INS. In this case, not only the error sources within the INS, but also the installation error between the INS and the Star sensor should be considered. Thus, on the basis of considering the traditional error terms, introduced the new installation error term and established the extended INS error model, the errors can be estimated and compensated in real time by utilizing the state estimation method, improving the accuracy of the Star sensor/INS integrated navigation system greatly. Simulations are carried out and the results verify the effectiveness and availability of the improved on-line calibration method proposed in this manuscript.