Permanent scatterer interferometry is a multitemporal interferometric synthetic aperture radar technique that produces high-accuracy ground deformation measurement. A high density of permanent scatterer (PS) is required to provide accurate results. In natural environments with low PS density, distributed scatterers (DSs) could serve as additional coherent observations. This paper introduces a polarimetric scattering property-based adaptive filtering method that preserves PS candidates and filters DS candidates. To further increase the coherence estimate of DS candidates, the technique includes a complex coherence decomposition that adaptively selects the most stable scattering mechanisms, thus improving pixel coherence estimation. The proposed method was evaluated on 11 quad-polarized ALOS PALSAR images and 21 dual-polarized Sentinel-1 images acquired over San Fernando Valley, CA, USA, and Groningen, The Netherlands, respectively. The application of this method increased the number of coherent pixels by almost a factor of eight compared with a single-polarization channel. This paper concludes that a coherence estimate can be significantly improved by applying scattering property-based adaptive filtering and coherence matrix decomposition and accurate displacement measurements can be achieved.