Following the accident at the Fukushima Nuclear Power Plant in March 2011, both experts and the national government provided insufficient information on radiation, leading to widespread distrust in the community. This study aimed at clarifying the association between anxiety about radiation and health literacy among residents in evacuation and non-evacuation areas in Fukushima. A questionnaire survey was sent to randomly sampled residents between August and October 2016, and data from 777 responses (38.9% valid response) were analyzed. The questionnaire assessed current radiation anxiety and discrimination and prejudice based on radiation exposure through seven items and communicative and critical health literacy through five items. Multiple regression analysis of the association between radiation anxiety and health literacy showed that the level of health literacy was significantly negatively associated with radiation anxiety in the evacuation areas (marginally in the non-evacuation areas) and marginally negatively associated with discrimination and prejudice in the evacuation areas but not in the non-evacuation areas. Therefore, improving health literacy could alleviate radiation anxiety.