Rice is mainly grown under rainfed conditions in West Africa. Unpredictable and variable rainfall, poor soil quality, and suboptimal crop management practices are the main determinants of low productivity. We assessed the effects of soil water availability and fertilizer application, and their interaction on the yield of rainfed rice in Glazoué, Department of Zou-Collines, central Benin between 2010 and 2013. On-farm fertilizer management trials and field surveys were conducted in 13–39 farmers’ fields per year. Field water conditions were visually assessed three times per week during the rice-growing season and flood and drought indices were calculated on the basis of number of days with ponded water and dry surface soil relative to the total number of days for the vegetative, the reproductive and whole rice-growing period. Variations in flood and drought indices were related to the sand content of the soil. While nitrogen was the most limiting nutrient, average response to N fertilizer application was low with an agronomic N use efficiency of only 7–9 kg grain per kg of N applied. Year-to-year variation in rainfall and spatial variation in field water status affected both rice yield and response to N fertilizer. Some 47% of the observed yield variation was explained by field water status and the amounts of N fertilizer applied, with rice response to N fertilizer being less when water was limited. We conclude that the prevailing blanket fertilizer recommendations are unlikely to contribute to yield increases in rainfed systems of West Africa. There is a need for field-specific recommendations that consider soil texture and the spatial–temporal dynamics of water availability.