The intelligent recognition of epileptic electro-encephalogram (EEG) signals is a valuable tool for the epileptic seizure detection. Recent deep learning models fail to fully consider both spectral and temporal domain representations simultaneously, which may lead to omitting the nonstationary or nonlinear property in epileptic EEGs and further produce a suboptimal recognition performance consequently. In this paper, an end-to-end EEG seizure detection framework is proposed by using a novel channel-embedding spectral-temporal squeeze-and-excitation network (CE-stSENet) with a maximum mean discrepancy-based information maximizing loss. Specifically, the CE-stSENet firstly integrates both multi-level spectral and multi-scale temporal analysis simultaneously. Hierarchical multi-domain representations are then captured in a unified manner with a variant of squeeze-and-excitation block. The classification net is finally implemented for epileptic EEG recognition based on features extracted in previous subnetworks. Particularly, to address the fact that the scarcity of seizure events results in finite data distribution and the severe overfitting problem in seizure detection, the CE-stSENet is coordinated with a maximum mean discrepancy-based information maximizing loss for mitigating the overfitting problem. Competitive experimental results on three EEG datasets against the state-of-the-art methods demonstrate the effectiveness of the proposed framework in recognizing epileptic EEGs, indicating its powerful capability in the automatic seizure detection.