Although it has been previously reported that patients with inflammatory breast cancer (IBC) experience worse survival than patients with other breast cancer (BC) types, the socioeconomic and ethnic factors leading to this survival difference are not fully understood. The association between county-level percent of persons below the poverty level and BC-specific (BCS) survival for cases diagnosed from 1990 to 2008 in the Surveillance, Epidemiology, and End Results (SEER) database linked to census derived county attributes was examined. A sub-analysis of cases from 2000 to 2008 also examined BCS survival by an index combining percent below poverty and less than high school graduates as well as metropolitan versus non-metropolitan county of residence. The Kaplan–Meier estimator was used to construct survival curves by stage, inflammatory status, and county-level socioeconomic position (SEP). Stage and inflammatory status stratified proportional hazards models, adjusted for age, race/ethnicity, tumor and treatment characteristics were used to determine the hazard of BCS death by county-level SEP. Kaplan–Meier survival curves indicated IBC has worse survival than stage matched non-IBC, (stage III IBC median survival = 4.75 years vs. non-IBC = 13.4 years, p  < 0.0001). Residing in a lower SEP, non-metro county significantly worsens BCS survival for non-IBC in multivariate proportional hazards models. African American cases appear to have worse survival than non-Hispanic Whites regardless of inflammatory status, stage, county-level SEP, tumor, or treatment characteristics. This is the first study to examine IBC survival by SEP in a nation-wide population-based tumor registry. As this analysis found generally poorer survival for IBC, regardless of SEP or race/ethnicity, it is important that interventions that help educate women on IBC symptoms target women in various SEP and race/ethnicity groups.