To solve object detection issues in infrared images, such as a low recognition rate and a high false alarm rate caused by long distances, weak energy, and low resolution, we propose a region-free object detector named YOLO-FIR for infrared (IR) images with YOLOv5 core by compressing channels, optimizing parameters, etc. An improved infrared image object detection network, YOLO-FIRI, is further developed. Specifically, while designing the feature extraction network, the cross-stage-partial-connections (CSP) module in the shallow layer is expanded and iterated to maximize the use of shallow features. In addition, an improved attention module is introduced in residual blocks to focus on objects and suppress background. Moreover, multiscale detection is added to improve small object detection accuracy. Experimental results on the KAIST and FLIR datasets show that YOLO-FIRI demonstrates a qualitative improvement compared with the state-of-the-art detectors. Compared with YOLOv4, the mean average precision (mAP50) of YOLO-FIRI is increased by 21% on the KAIST dataset, the speed is reduced by 62%, the parameters are decreased by 89%, the weight size is reduced by more than 94%, and the computational costs are reduced by 84%. Compared with YOLO-FIR, YOLO-FIRI has an approximately 5% to 20% improvement in AP, AR (average recall), mAP50, F1, and mAP50:75. Furthermore, due to the shortcomings of high noise and weak features, image fusion can be applied to image preprocessing as a data enhancement method by fusing visible and infrared images based on a convolutional neural network.