Highly Automated Vehicles (HAVs) must undergo strict safety testing before being released to the public. Mileage-based on-road testing suffers from unaffordable time costs and high safety risks. Simulated scenario-based testing has been found to be a trustworthy alternative for testing HAVs' built-in algorithms and functionalities. Test automation is typically used to generate target scenarios. This approach facilitates customized testing and avoids wasting time on simple and redundant scenarios. This study aims to review test automation methods and discuss how to accentuate their strengths rather than be trapped in their weaknesses under certain applicable conditions. According to their main purposes, we classify test automation methods into coverage-oriented, unsafe-scenario-oriented, and naturalistic-assessment-oriented categories. To further demonstrate the differences of these methods, we then design numerical experiment to compare the capabilities of seven test automation methods. Finally, we compile our observations to form a comprehensive guide for selecting test automation methods with different test requirements in mind.