This paper contributes two issues for enhancing the accuracy and speed of a pedestrian detection system. First, it introduces a feature description using variant-scale block based Histograms of Oriented Gradients (HOG) features. By non-restricted block sizes, an extensive feature space that allows high-discriminated features to be selected for classification can be obtained. Second, a classification method based on a hybrid cascade boosting technique and a Support vector machine (SVM) is described. The SVM is known as one of the most efficient learning models for classification. On the other hand, one advantage of cascade boosting structure is to quickly reject most negative examples in the early layers, while retains almost all positive examples for speed up of the system. Because the performance of boosting depends on the kernel of weak classifier, the hybrid algorithms using the proposed feature descriptor is helpful for constructing an efficient classification with low computational time. In addition, an “integral image” method is utilized to support fast computation of the feature. The experimental results showed that performance of the proposed method is higher than the SVM using standard HOG features about 5% and the AdaBoost using variant-scale based HOG features about 4% detection rates, at 1% false alarm rates. The speed of classification using a cascade boosting approach is doubled comparing to that of the non-cascade one.