Research on stress related to the COVID-19 pandemic has been dominated by the cases of healthcare workers, students, patients, and their stress during the COVID-19 pandemic. This study examined the relationship between the amount of stress change under the COVID-19 pandemic and demographic factors (age, sex, occupation, etc.) in residents of a large city and a rural area of Japan. A total of 1331 valid responses were received in June 2020 from residents of Tokyo, Osaka, and Nagano registered with a private research firm. We were able to identify 15 statistically significant variables out of 36 explanatory variables, which explained the significant increase in stress compared to the pre-pandemic period. Multiple-factor analysis showed that the relationship with people is a more significant explanatory variable for the level of increase in stress than the difference in environment between big cities (Tokyo, Osaka) and rural areas (Nagano), the type of housing, and the decrease in income compared to the pre-pandemic period.