Simultaneous Swift and Fermi observations of gamma-ray bursts (GRBs) offer a unique broadband view of their afterglow emission, spanning more than 10 decades in energy. We present the sample of X-ray flares observed by both Swift and Fermi during the first three years of Fermi operations. While bright in the X-ray band, X-ray flares are often undetected at lower (optical), and higher (MeV to GeV) energies. We show that this disfavors synchrotron self-Compton processes as the origin of the observed X-ray emission. We compare the broadband properties of X-ray flares with the standard late internal shock model, and find that in this scenario, X-ray flares can be produced by a late-time relativistic (Gamma > 50) outflow at radii R ~ 10 super(13)-10 super(14) cm. This conclusion holds only if the variability timescale is significantly shorter than the observed flare duration, and implies that X-ray flares can directly probe the activity of the GRB central engine.