Online surveys have the advantages of affordability and speedy collection; however, there have been some concerns about their response quality. Previous studies have measured response quality using nonresponse rate and biased choice behavior. However, it is difficult to detect defective respondents who are randomly selected without careful consideration by using those indicators. Therefore, this study measured the accuracy rate as an index of response quality using a questionnaire with questions for which possible answers were provided. Questionnaire length was found to have a negative effect on response rate but no significant effect on the accuracy rate. In addition, considering the response device, surveys from smartphone users have a lower accuracy rate than surveys from personal computer users. Lastly, respondents who have a response rate faster than 10 seconds per question have a lower accuracy rate. It is important to understand the factors that affect response quality and to design surveys accordingly.