•A multilabel strategy for breast and the tumor simultaneous segmentation combined with the self-attention module.•Five separate streams for each phase of DCE-MRI to disentangle information, avoiding early stage feature fusion.•Designing a time-signal intensity map as a new MRI modality to reflect pixelwise dynamic variations of DCE-MRI.•Comprehensive evaluation including segmentation results, experiments on different machines, and the clinical influence. Accurate breast and tumor segmentations from dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) is vital in breast disease diagnosis. Here, we propose a novel attention-guided joint-phase-learning network for multilabel segmentation including the breast and tumors simultaneously and automatically. Instead of common multichannel inputs, our novel network consists of five separated streams designed for extracting comprehensive features for each DCE-MRI phase to fully use the dynamic intensity of enhanced images. A new time-signal intensity map was designed based on the DCE-MRI pixel-by-pixel values and added as an additional stream to reflect breast tumor dynamic variations. The multiple streams were fused in a fully connected layer to integrate the comprehensive tumor information. Weighted-loss was applied to the multilabel strategy to highlight breast tumor segmentation. In addition, the net applies the self-attention module with grid-based attention coefficients based on a global feature vector to emphasize breast regions and suppress irrelevant non-breast tissue features. We trained our method on 144 DCE-MRI datasets acquired from Philips and achieved mean Dice coefficients of 0.92 and 0.86 for breast and tumor segmentations that were superior to common networks with multichannel structures. The model was extended to an independent test set with 59 cases from two different MRI machines and achieved a Dice coefficient of 0.83 for breast tumor segmentation, which illustrates the robustness of our framework. The automatically generated masks can improve the accuracy and time of diagnosis of malignant and benign breast tumors. Qualitative comparisons illustrate that the proposed method has high precision and generalizability.