Peptide receptor activation therapy (PRRT) is a promising treatment option for metastatic neuroendocrine tumors (NETs). However, predicting tumor shrinkage before treatment is challenging. We analyzed the shrinkage rate of each metastatic tumor lesion to identify predictive factors related to shrinkage. Patients with metastatic NET who underwent PRRT were included in this retrospective study. For each patient, between one to five metastatic lesions were selected in descending order of size, and the change in the maximum tumor diameter after treatment was defined as the shrinkage rate per lesion (L-SR). We analyzed the relationship between pretreatment clinicopathological factors and L-SR. The median L-SR of all 75 lesions in 20 patients was 20% (95% CI: 4.8−26.1%). While previous treatment with cytotoxic agents (34.4%, p < 0.05) and primary tumor of the pancreas (27.8%, p < 0.05) were significantly favorable factors, a primary tumor of the rectum was significantly more resistant to shrinkage (−20.5%, p < 0.001). Therefore, lesion-based analysis of PRRT for NETs showed that pancreatic NET and previous treatment with cytotoxic agents were favorable factors for tumor shrinkage; however, rectal NET was a factor associated with resistance to shrinkage.