Abstract Background Sorafenib is a kinase inhibitor that is used as a first-line therapy in advanced hepatocellular carcinoma (HCC) patients. However, the existence of sorafenib resistance has limited its therapeutic effect. Through RNA sequencing, we demonstrated that miR-138-1-3p was downregulated in sorafenib resistant HCC cell lines. This study aimed to investigate the role of miR-138-1-3p in sorafenib resistance of HCC. Methods In this study, quantitative real-time PCR (qPCR) and Western Blot were utilized to detect the levels of PAK5 in sorafenib-resistant HCC cells and parental cells. The biological functions of miR-138-1-3p and PAK5 in sorafenib-resistant cells and their parental cells were explored by cell viability assays and flow cytometric analyses. The mechanisms for the involvement of PAK5 were examined via co-immunoprecipitation (co-IP), immunofluorescence, dual luciferase reporter assay and chromatin immunoprecipitation (ChIP). The effects of miR-138-1-3p and PAK5 on HCC sorafenib resistant characteristics were investigated by a xenotransplantation model. Results We detected significant down-regulation of miR-138-1-3p and up-regulation of PAK5 in sorafenib-resistance HCC cell lines. Mechanistic studies revealed that miR-138-1-3p reduced the protein expression of PAK5 by directly targeting the 3′-UTR of PAK5 mRNA. In addition, we verified that PAK5 enhanced the phosphorylation and nuclear translocation of β-catenin that increased the transcriptional activity of a multidrug resistance protein ABCB1. Conclusions PAK5 contributed to the sorafenib resistant characteristics of HCC via β-catenin/ABCB1 signaling pathway. Our findings identified the correlation between miR-138-1-3p and PAK5 and the molecular mechanisms of PAK5-mediated sorafenib resistance in HCC, which provided a potential therapeutic target in advanced HCC patients.