E-cadherin is a tumor suppressor gene in invasive lobular breast cancer. However, a proportion of high-grade ductal carcinoma shows reduced/loss of E-cadherin. In this study, we assessed the underlying mechanisms and molecular implications of E-cadherin loss in invasive ductal carcinoma. This study used large, well-characterized cohorts of early-stage breast cancer-evaluated E-cadherin expression via various platforms including immunohistochemistry, microarray analysis using Illumina HT-12 v3, copy number analysis using Affymetrix SNP 6.0 arrays, and next-generation sequencing for differential gene expression. Our results showed 27% of high-grade invasive ductal carcinoma showed reduced/loss of E-cadherin membranous expression. CDH1 copy number loss was in 21% of invasive ductal carcinoma, which also showed low CDH1 mRNA expression (p = 0.003). CDH1 copy number was associated with copy number loss of TP53, ATM, BRCA1, and BRCA2 (p < 0.001). Seventy-nine percent of invasive ductal carcinoma with reduced CDH1 mRNA expression showed elevated expression of E-cadherin transcription suppressors TWIST2, ZEB2, NFKB1, LLGL2, CTNNB1 (p < 0.01). Reduced/loss E-cadherin expression was associated with differential expression of 2143 genes including those regulating Wnt (FZD2, GNG5, HLTF, WNT2, and CER1) and PIK3-AKT (FGFR2, GNF5, GNGT1, IFNA17, and IGF1) signaling pathways. Interestingly, key genes differentially expressed between invasive lobular carcinoma and invasive ductal tumors did not show association with E-cadherin loss in invasive ductal carcinoma. We conclude that E-cadherin loss in invasive ductal carcinoma is likely a consequence of genomic instability occurring during carcinogenesis. Potential novel regulators controlling E-cadherin expression in invasive ductal carcinoma warrant further investigation.