Buildings consume about 41.1% of primary energy and 74% of the electricity in the U.S. Better or even optimal building energy control and operation strategies provide great opportunities to reduce building energy consumption. Moreover, it is estimated by the National Energy Technology Laboratory that more than one-fourth of the 713GW of U.S. electricity demand in 2010 could be dispatchable if only buildings could respond to that dispatch through advanced building energy control and operation strategies and smart grid infrastructure. Energy forecasting models for building energy systems are essential to building energy control and operation. Three general categories of building energy forecasting models have been reported in the literature which include white-box (physics-based), black-box (data-driven), and gray-box (combination of physics based and data-driven) modeling approaches. This paper summarizes the existing efforts in this area as well as other critical areas related to building energy modeling, such as short-term weather forecasting. An up-to-date overview of research on application of building energy modeling methods in optimal control for single building and multiple buildings is also summarized in this paper. Different model-based and model-free optimization methods for building energy system operation are reviewed and compared in this paper. Agent based modeling, as a new modeling strategy, has made a remarkable progress in distributed energy systems control and optimization in the past years. The research literature on application of agent based model in building energy system control and operation is also identified and discussed in this paper.