•The optical fiber coil with spool (OFCS) is a crucial tool for precise physics measurements.•To improve the seismic resistance of OFCSs in space applications, the vibration modes are thoroughly investigated through dynamic analysis, simulations, and experiments.•We establish a dynamic model of the OFCS based on the vibration transmission mechanism to characterize its vibration modes.•The vibration eigenfrequency and shapes are calculated under various conditions.•The experimental results validate the feasibility of the dynamic model and simulation, meanwhile, the discrepancies between the simulations and experiments are analyzed.•This study is the first of its kind to develop a comprehensive dynamic model for an OFCS, with simulations and experiments confirming its effectiveness. The optical fiber coil with spool (OFCS) is a crucial tool for precise physics measurements. To improve the seismic resistance of OFCSs in space applications, the vibration modes are thoroughly investigated through dynamic analysis, simulations, and experiments. We establish a dynamic model of the OFCS based on the vibration transmission mechanism to characterize its vibration modes. The vibration eigenfrequency and shapes are calculated and simulated under various conditions. The experimental results validate the feasibility of the dynamic model and simulation, meanwhile, the discrepancies between the simulations and experiments are analyzed. This study is the first of its kind to develop a comprehensive dynamic model for an OFCS, with simulations and experiments confirming its effectiveness.