Ultrafast laser physics continues to advance at a rapid pace, driven primarily by the development of more powerful and sophisticated diode-pumping sources, the development of new laser materials, and new laser and amplification approaches such as optical parametric chirped-pulse amplification. The rapid development of high average power cryogenic laser sources seems likely to play a crucial role in realizing the long-sought goal of powerful ultrafast sources that offer concomitant high peak and average powers. In this paper, we review the optical, thermal, thermo-optic and laser parameters important to cryogenic laser technology, recently achieved laser and laser materials progress, the progression of cryogenic laser technology, discuss the importance of cryogenic laser technology in ultrafast laser science, and what advances are likely to be achieved in the near-future.