While obesity is recognisably associated with changes in heart rate variability (HRV), the association between skeletal muscle mass and HRV is less clear. In this cross sectional study, we analysed the association of body fat (four parameters) and muscle mass (five parameters) with indicators of HRV activity. Assessment of body composition and HRV was performed in n = 180 young-to-middle age healthy men exposed to high occupational physical activity, using the multi-frequency bioelectrical impedance device and the PPG-StressFlow® HRV photoplethysmography device, respectively. Mean values of parameters of fat tissue were above normal/reference values. Muscle tissue indicators were higher or within the reference ranges. Fat tissue parameters were significantly higher in participants with lower parasympathetic nervous system (PNS) indicators. Weight-adjusted skeletal muscle index (wSMI) was significantly lower in men with reduced PNS parameters. Fat tissue parameters were negatively correlated with PNS parameters, while wSMI was positively correlated with PNS parameters. Participants with higher fat mass and lower muscle mass had poorer parasympathetic activity. Since mean values of HRV parameters indicated mild parasympathetic dominance, we conclude that physical activity and consequently good muscle mass potentially compensated for the negative interaction between fat tissue and HRV.