Quantifying full left ventricular (LV) metrics including cavity area, myocardium area, cavity dimensions and wall thicknesses from cardiac magnetic resonance (MR) images, and then assessing regional and global cardiac function plays a crucial role in clinical practice. However, due to highly variable cardiac structures across different subjects, it is challenging to obtain an accurate estimation of LV metrics. In this paper, we propose a novel deep learning framework, called cascaded segmentation and regression network (CSRNet), to improve the quantification results. The CSRNet consists of two components: a segmentation component and a regression component. The segmentation component yields myocardial contours of the left ventricle from the input cardiac MR images, and then the regression component learns hierarchical representations from the segmented images and estimates the desired LV metrics. By introducing the myocardial contours, the regression component can pay more attention to the left ventricle, which contributes to more accurate quantification results, although the cardiac structures are variable. The extensive experiments on a dataset of 145 subjects demonstrate that our framework outperforms the state-of-the-art methods.