Between 1997 and 2004 several observing runs were conducted, mainly with the CTIO 0.9 m, to image International Celestial Reference Frame (ICRF) counterparts (mostly QSOs) in order to determine accurate optical positions. Contemporary to these deepCCDimages, the same fields were observed with theU.S.Naval Observatory astrograph in the same bandpass. They provide accurate positions on the Hipparcos/Tycho-2 system for stars in the 10-16 mag range used as reference stars for the deep CCD imaging data. Here we present final optical position results of 413 sources based on reference stars obtained by dedicated astrograph observations that were reduced following two different procedures. These optical positions are compared to radio very long baseline interferometry positions. The current optical system is not perfectly aligned to the ICRF radio system with rigid body rotation angles of 3-5 mas (= 3sigma level) found between them for all three axes. Furthermore, statistically, the optical-radio position differences are found to exceed the total, combined, known errors in the observations. Systematic errors in the optical reference star positions and physical offsets between the centers of optical and radio emissions are both identified as likely causes. A detrimental, astrophysical, random noise component is postulated to be on about the 10 mas level. If confirmed by future observations, this could severely limit the Gaia to ICRF reference frame alignment accuracy to an error of about 0.5 mas per coordinate axis with the current number of sources envisioned to provide the link. A list of 36 ICRF sources without the detection of an optical counterpart to a limiting magnitude of about R = 22 is provided as well.