We develop a general framework for quantifying the many different contributions to the noise budget of an image made with an array of dishes or aperture array stations. Each noise contribution to the visibility data is associated with a relevant correlation timescale and frequency bandwidth so that the net impact on a complete observation can be assessed when a particular effect is not captured in the instrumental calibration. All quantities are parameterised as function of observing frequency and the visibility baseline length. We apply the resulting noise budget analysis to a wide range of existing and planned telescope systems that will operate between about 100 MHz and 5 GHz to ascertain the magnitude of the calibration challenges that they must overcome to achieve thermal noise limited performance. We conclude that calibration challenges are increased in several respects by small dimensions of the dishes or aperture array stations. It will be more challenging to achieve thermal noise limited performance using 15 m class dishes rather than the 25 m dishes of current arrays. Some of the performance risks are mitigated by the deployment of phased array feeds and more with the choice of an (alt,az,pol) mount, although a larger dish diameter offers the best prospects for risk mitigation. Many improvements to imaging performance can be anticipated at the expense of greater complexity in calibration algorithms. However, a fundamental limitation is ultimately imposed by an insufficient number of data constraints relative to calibration variables. The upcoming aperture array systems will be operating in a regime that has never previously been addressed, where a wide range of effects are expected to exceed the thermal noise by two to three orders of magnitude. Achieving routine thermal noise limited imaging performance with these systems presents an extreme challenge. The magnitude of that challenge is inversely related to the aperture array station diameter.