Building models with limited data is one of the key steps towards the application of deep learning models in realistic scenarios. Under the framework of representation learning, we propose several novel algorithms for tackling challenging tasks in building models with limited data, including standard few-shot learning, incremental few-shot learning, and unsupervised few-shot learning. In the first part of this thesis, we propose an approach to learn low-rank representation that generalizes well to a new task using just a few training samples. High-quality representation can be found by averaging the weights of neural networks during the pre-training phase. Our approach achieves strong performance on both few-shot classification and regression benchmarks. We then consider incremental few-shot learning, in which the model incrementally learns new tasks from few-shot samples without forgetting old ones. We propose an approach to harmonize old knowledge preserving and new knowledge adaptation through quantized vectors of the learned representation. Prediction is made in a nonparametric way using similarity to learned reference vectors, which circumvents biased weights in a parametric classification layer during incremental few-shot learning. Experimental results demonstrate that the proposed method outperforms other state-of-the-art methods in incremental learning. In addition, we develop deep Laplacian eigenmaps to learn representation from unlabeled image data for downstream few-shot learning tasks. Our method learns representation by grouping similar images together and can be intuitively interpreted by random walks on augmented training data. The proposed method significantly closes the performance gap between supervised and unsupervised few-shot learning. To get us closer to general unsupervised representation learning across different data types, we present a domain-agnostic self-supervised learning method, which learns representation from unlabeled data without domain-specific data augmentations. The proposed method is adversarial perturbation based latent reconstruction (APLR), which is closely related to multi-dimensional Hirschfeld-Gebelein-Renyi maximal correlation and has theoretical guarantees on the linear probe error. APLR not only outperforms existing domain-agnostic self-supervised learning methods, but also closes the performance gap to domain-specific self-supervised learning methods on various domains, such as tabular data, images, and audio.