Distant metastasis remains the major cause of treatment failure in esophageal cancer, though there have been few large-scale studies of the patterns of distant metastasis in different histological types. We investigated the patterns of distant metastasis in esophageal adenocarcinoma (AC) and squamous cell carcinoma (SCC) using a population-based approach. Patients with stage IV esophageal cancer at diagnosis were identified using the Surveillance, Epidemiology, and End Results database. Multivariable logistic regression was performed to identify potential risk factors for site-specific distant metastasis to the distant lymph nodes, bone, liver, brain, and lung at diagnosis. We identified 1,470 patients with complete data for analysis including 1,096 (74.6%) patients with AC and 374 (25.4%) patients with SCC. A total of 2,243 sites of distant metastasis were observed, the liver was the most common site of distant metastasis (727, 32.4%), followed by the distant lymph nodes (637, 28.4%), lung (459, 20.5%), bone (344, 15.3%), and brain (76, 3.4%). Multivariable logistic regression showed that compared to patients with SCC, patients with AC were more likely to have metastasis to the brain (odds ratio OR 3.026, 95% confidence interval CI 1.441-6.357, = 0.003) and liver (OR 1.848, 95% CI 1.394-2.451, < 0.001), and less likely to have metastasis to the lung (OR 0.404, 95% CI 0.316-0.516, < 0.001). Histological type had no effect on metastasis to the distant lymph nodes or bone. Patients with esophageal AC are more likely to present with liver and brain metastases, and less likely to present with lung metastasis than patients with esophageal SCC.