The spectral signatures of asymmetry in Type Ia Supernova (SN Ia) explosions are investigated, using a sample of late-time nebular spectra. First, a kinematical model is constructed for SN Ia 2003hv, which can account for the main features in its optical, Near-Infrared (NIR), and Mid-Infrared (Mid-IR) late-time spectra. It is found that an asymmetric off-center model can explain the observed characteristics of SN 2003hv. This model includes a relatively high-density, Fe-rich region which displays a large velocity off-set, and a relatively low density, extended 56Ni-rich region which is more spherically distributed. The high-density region consists of the inner stable Fe-Ni region and outer 56Ni-rich region. Such a distribution may be the result of a delayed-detonation explosion, in which the first deflagration produces the global asymmetry in the innermost ejecta, while the subsequent detonation can lead to the bulk spherical symmetry. This configuration, if viewed from the direction of the off-set, can consistently explain the blueshift in some of the emission lines and virtually no observed shift in other lines in SN 2003hv. For this model, we then explore the effects of different viewing angles and the implications for SNe Ia in general. The model predicts that a variation of the central wavelength, depending on the viewing angle, should be seen in some lines (e.g., Ni II Delta *l7378), while the strongest lines (e.g., Fe III blend at ~4700 A) will not show this effect. By examining optical nebular spectra of 12 SNe Ia, we have found that such a variation indeed exists. We suggest that the global asymmetry in the innermost ejecta, as likely imprint of the deflagration flame propagation, is a generic feature of SNe Ia. It is also shown that various forbidden lines in the NIR and Mid-IR regimes provide strong diagnostics to further constrain the explosion geometry and thus the explosion mechanism.