The object 1E 161348-5055 (1E 1613) is a pointlike, soft X-ray source shining at the center of the 2000 yr old supernova remnant (SNR) RCW 103. It features a puzzling 6.67 hr periodicity and dramatic variability over a timescale of a few years. This, coupled with a young age and the lack of an obvious optical counterpart, makes 1E 1613 a unique source among all compact objects associated with SNRs. It could either be the first low-mass X-ray binary system discovered inside a SNR or a peculiar isolated magnetar with an extremely slow spin period. Analysis of archival VLT ISAAC and HST NICMOS infrared observations unveils a very crowded field. A few sources are positionally consistent with the refined X-ray error region that we derived from the analysis of 13 Chandra observations. To shed light on the nature of 1E 1613, we have performed deep IR observations of the field with the NACO instrument at the VLT, searching for variability. None of the candidates show clear modulation at 6.67 hr or have significant long-term variability. Moreover, none of the candidates stand out for peculiar colors with respect to the bulk of the field sources. We find no compelling reasons to associate any of the candidates with 1E 1613. On one hand, it is very unlikely that one of the candidates is a low-mass companion star to 1E 1613. On the other hand, if the X-ray source is an isolated magnetar surrounded by a fallback disk, we cannot exclude that the IR counterpart is hidden among the candidates. If none of the potential counterparts are linked to the X-ray source, 1E 1613 will remain undetected in the IR down to image, which will make its interpretation as an accreting binary system rather problematic.