Pillar structure surfaces and surface vibrations are considered to be two individual techniques to enhance nucleate boiling. In this work, a volume of fluid (VOF) based numerical model is developed to study bubble growth promotion during nucleate pool boiling on a pillar structure surface under the aid of surface vibration. Three vibration modes of the pillar structure, including the horizontal vibration mode (HVM), vertical vibration mode (VVM) and angular vibration mode (AVM), are considered, and their effects on the bubble dynamics and heat transfer are discussed with various vibration amplitudes and frequencies. It is found that the pillar structure surface is favorable for bubble detachment as it constrains the bubble base radius and forces the bubble to grow more vertically compared to the plain surface. Although the HVM and AVM exert no or less vertical force component on the bubble for its detachment than the VVM does, but their horizontal force component significantly alters the bubble shape profile, breaks the force balance, and greatly increases the bubble departure frequency and heat transfer. The largest increment of liquid-vapor phase change rate is found to be 30.9% for the HVM. The bubble departure frequency and the vapor volume flux quantity increase with the pillar vibration amplitude and frequency. This work will deepen our understanding of fundamental bubble growth mechanisms as altered by vibration on the pillar structure surface for better heat transfer applications.