Abstract Vision-based precision measurement is limited by the optical resolution. Although various super-resolution algorithms have been developed, measurement precision and accuracy are difficult to guarantee. To achieve nanoscale resolution measurement, a super-resolution microstructure concept is proposed which is based on the idea of a strong mathematical mapping relationship that may exist between microstructure surface topography features and the corresponding image pixel intensities. In this work, a series of microgrooves are ultra-precision machined and their surface topographies and images are measured. A mapping relationship model is established to analyze the effect of the microgroove surface topography on the imaging mechanism. The results show that the surface roughness and surface defects of the microgroove have significant effects on predicting the imaging mechanism. The optimized machining parameters are determined afterward. This paper demonstrates a feasible and valuable work to support the design and manufacture super-resolution microstructure which has essential applications in precision positioning measurement.