Privacy protection is considered as an important problem in learning-based systems. Recently, various works based on differential privacy have been proposed to protect an individual's privacy in the machine learning and deep learning contexts. One of the state-of-the-art approaches is Private Aggregation of Teacher Ensembles (PATE), a generic framework which can be successfully applied to many different learning algorithms. In PATE, we need to split the private dataset into many disjoint subsets and train an ensemble of teachers on these subsets. Then, we transfer noisy predictions from the ensemble of teachers to a student model. In this paper, we show that for complex datasets and tasks, such as nature image classification, the training set allocated for one teacher may be too small with respect to the corresponding task to achieve an ideal performance. To alleviate this problem, we propose the TrPATE framework which extends PATE with transfer learning. Based on PATE, we transfer and share the knowledge extracted from a publicly available non-private dataset to the teachers. The extensive experiments are conducted on various datasets, and the empirical results demonstrate the effectiveness of our method.