Rectal neuroendocrine tumors (NETs) are currently divided into L-cell and non-L-cell types. In the World Health Organization 2010 classification, L-cell tumors are defined as borderline, whereas non-L-cell tumors are considered to represent malignancies. To establish differential diagnostic criteria and therapeutic strategy, we investigated the pathologic features of rectal NETs associated with lymph node metastasis and the clinicopathologic significance of the L-cell phenotype. We analyzed 284 patients with rectal NETs. Factors, including T stage, mitosis, histologic pattern, lymphatic invasion, tumor border, and lymph node metastasis, were retrospectively evaluated. We also evaluated tumor immunoreactivity for L-cell markers, including glucagon-like peptide 1, pancreatic peptide, and peptide YY, in 240 cases. L-cell immunoreactivity was detected in 189 of 240 NETs (79%). Of the factors evaluated, only age and the frequency of lymphatic invasion were significantly different between patients with L-cell and non-L-cell tumors. Of the 284 patients, 18 (6.3%) had lymph node metastases. Lymphatic invasion and T stage were independent risk factors for lymph node metastasis. Subgroup analysis based on tumor size showed lymph node metastasis in 0%, 4%, 24%, and 100% of patients with NETs with a size of <5, 5 to 9, 10 to 14, and ≥ 15 mm, respectively. Depth of tumor invasion, lymphatic invasion, and mitosis were correlated with tumor size (P<0.0001). In conclusion, L-cell phenotype alone does not guarantee favorable biological characteristics. The clinical management of rectal NETs should depend on tumor size. Careful pathologic examination of lymphatic invasion is necessary.