Vaccination is a major strategy to prevent the coronavirus disease 2019 (COVID-19). However, information about factors associated with men and women intention to be vaccinated are scarce. To determine COVID-19 vaccine acceptance and identify factors associated vaccine hesitancy according to sex, we performed a cross-sectional population-based random survey in Salvador, Brazil between Nov/2020-Jan/2021. Participants were interviewed to obtain data on intention to receive and pay for a COVID-19 vaccine, as well as on demographics, comorbidities, influenza vaccination history, previous diagnosis of COVID-19, and exposures and perception of COVID-19 risk. Among 2,521 participants, 2,053 (81.4%) reported willingness to use a COVID-19 vaccine and 468 (18.6%) hesitated to take it. Among those intending to get vaccinated, 1,400 (68.2%) would pay for the vaccine if necessary. Sex-stratified multivariable analysis found that men who were working and who had comorbidities were less likely to hesitate about using the vaccine. Among women, higher educational level and high perception of COVID-19 risk were associated with less vaccine hesitancy. In both groups, reporting influenza vaccination in 2020 reduced the chance of COVID-19 vaccine hesitancy. COVID-19 vaccine campaigns targeting to reduce vaccine hesitancy are urgently needed. These campaigns should consider gender differences in order to be successful.