All-optical control of plasmons can enable optical switches with high speeds, small footprints and high on/off ratios. Here we demonstrate ultrafast plasmon modulation in the near-infrared (NIR) to mid-infrared (MIR) range by intraband pumping of indium tin oxide nanorod arrays (ITO-NRAs). We observe redshifts of localized surface plasmon resonances arising from a change of the plasma frequency of ITO, which is governed by the conduction band non-parabolicity. We generalize the plasma frequency for non-parabolic bands, quantitatively model the fluence-dependent plasma frequency shifts, and show that different from noble metals, the lower electron density in ITO enables a remarkable change of electron distributions, yielding a significant plasma frequency modulation and concomitant large transient bleaches and induced absorptions, which can be tuned spectrally by tailoring the ITO-NRA geometry. The low electron heat capacity explains the sub-picosecond kinetics that is much faster than noble metals. Our work demonstrates a new scheme to control infrared plasmons for optical switching, telecommunications and sensing.