There is mixed evidence on whether living arrangements and social interactions are associated with poorer health outcomes after hip fracture repair. Distinct social profiles among male and female older adults may explain some of the differences. However, prior studies did not evaluate these differences by sex. This article aims to assess if the associations between living alone, social interaction, and physical performance differ by sex among hip fracture survivors. This prospective cohort study is part of the Baltimore Hip Studies seventh cohort, with 168 male and 171 female hip fracture patients assessed at baseline (≤22 days after hospitalization) and at 2, 6, and 12 months post admission. Living arrangements and interaction with children or siblings and others in the past 2 weeks were collected at all visits. Physical performance was measured in the follow-up visits with the Short Physical Performance Battery (SPPB). Linear mixed models tested associations of living alone and social interaction with SPPB over time adjusted for age, education, comorbidities, physical functioning pre-fracture, cognitive function, self-rated health, and time. For men only, living alone was associated with worse performance (0.7 points lower SPPB scores, p = 0.05). Higher social interaction was associated with 0.8 and 1.2 point higher SPPB scores for men and women, respectively (p < 0.05). Visiting with friends was significantly associated with better function among males, while visiting with children or siblings was associated with worse SPPB among females. Living arrangements and types of social interaction are differentially associated with physical function for older men and women. Screening for social isolation/integration and including interventions that promote social interaction and participation should be considered in healthcare programs for hip fracture survivors.