Mechanical space heating and cooling are responsible for over one-third of the greenhouse gases released by building operations globally. As a result, heating and cooling load reductions are high priorities in climate change mitigation efforts. Direct solar heating, natural ventilation, and shading are often able to condition indoor spaces “passively” using only climatic resources, but their performance is limited by the lack of effective and affordable controls for their operable elements: rule-based control strategies cannot anticipate changes in weather or adapt to seasonal changes, while model-based strategies require significant investment into the creation of customized thermal models. Here, we design and validate a model-free data-driven reinforcement learning approach by comparing tabular Q-learning and policy-gradient (REINFORCE) algorithms for passive heating and cooling. These algorithms are trained on a residential building simulated in EnergyPlus in Albany NY and evaluated on the basis of unmet heating and cooling loads in both the training climate and six others. We find that the learned operation of shading, night insulation, and window aperture opening, driven by indoor and outdoor air temperatures, window surface heat flux, and weather forecasts, reduces total loads by 47-76% compared to operation without passive systems. Additionally, the REINFORCE policy reduces loads by 13-64% over conventional rule-based control, with one exception. Together, these results show that reinforcement learning can improve passive heating and cooling performance substantially, ultimately reducing space heating and cooling energy requirements.