The importance of evaluating performance of cloud systems has been increasing with the rapid growing market demands for cloud computing. However, the performance testers often have to go through the hassle of tedious manual operations when interacting with the cloud. A cloud performance evaluation framework is designed for both broad cloud support and good workload extensibility, which provides an automatic interface to monitor the capability and scalability of Infrastructure-as-a-Service cloud systems. Cloud API modules are implemented for Amazon EC2 service and OpenStack. It can achieve flexible control workflows for multiple of different workloads and user customization to test scenarios. With several built-in workloads and metric aggregation methods, a series of tests is performed on our private clouds to compare the performance and scalability from multiple aspects. A methodology is also proposed to build a cost-performance model to better understand and analyze the efficiency of different types of cloud systems. Based on the results of the experiments, the model indicates a polynomial relation between performance per instance and the overall cost.