Based on high-frequency data for more than fifty commodities, currencies, equity indices, and fixed-income instruments spanning more than two decades, we document strong similarities in realized volatility patterns within and across asset classes. Exploiting these similarities through panel-based estimation of new realized volatility models results in superior out-of-sample risk forecasts, compared to forecasts from existing models and conventional procedures that do not incorporate the similarities in volatilities. We develop a utility-based framework for evaluating risk models that shows significant economic gains from our new risk model. Lastly, we evaluate the effects of transaction costs and trading speed in implementing different risk models.