Wireless communication with unmanned aerial vehicles (UAVs) is a promising technology for future communication systems. In this paper, assuming that the UAV flies horizontally with a fixed altitude, we study energy-efficient UAV communication with a ground terminal via optimizing the UAV's trajectory, a new design paradigm that jointly considers both the communication throughput and the UAV's energy consumption. To this end, we first derive a theoretical model on the propulsion energy consumption of fixed-wing UAVs as a function of the UAV's flying speed, direction, and acceleration. Based on the derived model and by ignoring the radiation and signal processing energy consumption, the energy efficiency of UAV communication is defined as the total information bits communicated normalized by the UAV propulsion energy consumed for a finite time horizon. For the case of unconstrained trajectory optimization, we show that both the rate-maximization and energy-minimization designs lead to vanishing energy efficiency and thus are energy-inefficient in general. Next, we introduce a simple circular UAV trajectory, under which the UAV's flight radius and speed are jointly optimized to maximize the energy efficiency. Furthermore, an efficient design is proposed for maximizing the UAV's energy efficiency with general constraints on the trajectory, including its initial/final locations and velocities, as well as minimum/maximum speed and acceleration. Numerical results show that the proposed designs achieve significantly higher energy efficiency for UAV communication as compared with other benchmark schemes.