We test recent claims that the polar field at the end of Cycle 23 was weakened by a small number of large, abnormally oriented regions, and investigate what this means for solar cycle prediction. We isolate the contribution of individual regions from magnetograms for Cycles 21, 22, and 23 using a 2D surface flux transport model, and find that although the top ∼10% of contributors tend to define sudden large variations in the axial dipole moment, the cumulative contribution of many weaker regions cannot be ignored. To recreate the axial dipole moment to a reasonable degree, many more regions are required in Cycle 23 than in Cycles 21 and 22 when ordered by contribution. We suggest that the negative contribution of the most significant regions of Cycle 23 could indeed be a cause of the weak polar field at the following cycle minimum and the low-amplitude Cycle 24. We also examine the relationship between a region's axial dipole moment contribution and its emergence latitude, flux, and initial axial dipole moment. We find that once the initial dipole moment of a given region has been measured, we can predict the long-term dipole moment contribution using emergence latitude alone.