•Conduct a non-contact 3D Scanning Laser Doppler Vibrometer (SLDV) test on a three-bladed wind turbine assembly.•Develop a finite element model of the assembly including the composite blade modeling.•Correlate modal parameters and dynamic response between the SLDV test and the finite element model.•Investigate high-order complex curvature mode shapes and mode coupling of the wind turbine blade through both experimental and numerical approach. Experimental and numerical modal analysis on wind turbine blades has been previously studied, considering mainly low order bending modes. However, high-order modes are also critical modes for understanding blade dynamics. The mode coupling is essential because a better understanding of the high-frequency blade dynamics can support advances in model validation, blade aeroelastic simulations, blade design, and structural health monitoring. However, these high-order modes and the associated mode couplings of wind turbine blades have not been studied. This work presents a comprehensive experimental and numerical study based on three modal tests and a correlated finite element simulation to study the complex curvature mode shapes and mode coupling dynamics for a three-bladed wind turbine assembly. Three tests are conducted: Test 1, ten accelerometers are deployed on the whole assembly under impact excitation; Test 2, nine accelerometers are deployed on a single blade under impact excitation; and Test 3, a non-contact 3D Scanning Laser Doppler Vibrometer (SLDV) test is performed on a single blade under shaker excitation. This is the first work to use a 3D SLDV for an experimental modal test on the wind turbine blade. With 300–400 points measured with the 3D SLDV, experimental mode shapes having a high spatial resolution with 3D response are used to characterize the coupling for the low-order and high-order modes with complex curvatures. A reliable finite element model of the three-bladed assembly, including the composite blade modeling, is also developed and is well correlated with Test 2 and Test 3. With the high-fidelity 3D SLDV test and well-correlated finite element model, this is also the first work of using experimental and numerical approaches to investigate the high-order mode shape with complex curvatures and mode coupling of bending and torsional behavior that is present in the wind turbine blade for these high-order modes.