One of the significant challenges for robotic construction with dimensional lumber and other construction materials is the accumulation of material imperfections and manufacturing inaccuracies, resulting in significant deviations between the as-built structure and its digital twin. This paper presents and evaluates methods for addressing these challenges to enable a multi-robot construction process that adaptively updates future fabrication steps to accommodate for perceived inaccuracies, improving build quality. We demonstrate through a physical stacking case study experiment that our methods can decrease fabrication deviations due to setup and calibration errors by utilizing robot perception and adaptive processes. Overall, this research advances current toolpath and task optimization strategies to help shape a comprehensive system for working with tolerance-aware robotic construction.