Human T cell leukemia virus type 1 (HTLV-1), the etiological agent of adult T cell leukemia/lymphoma (ATLL) and of HTLV-1-associated myelopathy/tropical spastic paraparesis (HAM/TSP), was identified a few years before Human Immunodeficiency Virus (HIV). However, forty years later, our comprehension of HTLV-1 immune detection and the host immune responses to HTLV-1 is far more limited than for HIV. In addition to innate and adaptive immune responses that rely on specialized cells of the immune system, host cells may also express a range of antiviral factors that inhibit viral replication at different stages of the cycle, in a cell-autonomous manner. Multiple antiviral factors allowing such an intrinsic immunity have been primarily and extensively described in the context HIV infection. Here, we provide an overview of whether known HIV restriction factors might act on HTLV-1 replication. Interestingly, many of them do not exert any antiviral activity against HTLV-1, and we discuss viral replication cycle specificities that could account for these differences. Finally, we highlight future research directions that could help to identify antiviral factors specific to HTLV-1.