The control of magnetism with an electric field is a challenging area with the potential to affect fields related to magnetic data storage, sensors and magnetic random access memory. Although there are some successful examples of such control based on the use of magnetic metals and semiconductors, energy loss caused by current flow is a problem that needs to be addressed. In particular, the repeatable control of magnetization with an electric field can be disturbed by joule heat loss. In this regard, non-centrosymmetric insulating magnets are good candidates for controlling magnetization without energy loss, in which the linear magnetoelectric effect has an essential role. Moreover, such magnets exhibit an unconventional magneto-optical effect, which allows the time-resolved detection of the magnetization direction. Here, we show a periodic oscillation of the magnetization direction by ±20 in a non-centrosymmetric soft magnet (Cu,Ni)B2O4, which is induced by an a.c. electric field of 2 kHz. The present study provides a strategy for identifying materials in which the magnetization direction can be modulated at high speed with an electric field.