Fraud is an aggravating problem in the health insurance system, causing a substantial increase in the cost of medical services. Many models have been developed using data mining or machine learning techniques to lessen the impact of fraud on healthcare system. However, achieving good accuracy is still challenging as the claims data is multivariate with multiple class overlappings. In this paper, we propose a novel approach of unsupervised multivariate analysis for healthcare claims submitted by the providers. Our proposed model analyzes multivariate categorical data and continuous data in two stages to observe providers’ behaviors. The first stage constructs Weighted MultiTree (WMT) for categorical data to analyze similarity among provider profiles, the relation among profiles, and rendered services to identify false services. The second stage detects false claims by developing a univariate fraud detection model using different Density Based Clustering (DBC) techniques on continuous data of claims such as service counts and service charges. The performance of our proposed WMTDBC is measured by conducting experiments on the claims within various medical specialties or provider types of CMS part B program. Our empirical results evidence that the detection performance is enhanced with our WMTDBC approach when compared with the state-of-the-art models. •WMTDBC model does not require any labeled data of healthcare claims.•WMTDBC can analyze both categorical and continuous types of data.•WMT has no class overlaps and intermediate results while grouping similar claims.•WMT can be constructed with reduced number of nodes and reduce complexity.