This article proposes an efficient and adaptive statistical superpixel merging approach with edge penalty for polarimetric synthetic aperture radar (PolSAR) image segmentation. Based on the initial superpixel over-segmentation result obtained by our previously proposed adaptive polarimetric superpixel generation algorithm (Pol-ASLIC), this work achieves efficient and accurate PolSAR image segmentation by merging superpixels using the statistical region merging (SRM) framework. This article proposes to define a new dissimilarity measure between superpixels, which takes the edge penalty into consideration, leading to a reasonable and accurate merging order for superpixel pairs. With regard to the merging predicate of superpixels, a polarimetric homogeneity measurement (HoM) is used to define the merging threshold, making the merging predicate and merging threshold adaptive to the PolSAR image content. Experimental results on three airborne and one spaceborne PolSAR data sets demonstrate that the proposed approach can effectively improve the computation efficiency and segmentation accuracy in comparison with state-of-the-art merging-based methods for PolSAR data. More importantly, the proposed approach is free of parameters and easy to use.