Birefringence offers an intrinsic contrast mechanism related to the microstructure and arrangement of fibrillary tissue components. Here we present a reconstruction strategy to recover not only the scalar amount of birefringence but also its optic axis orientation as a function of depth in tissue from measurements with catheter-based polarization sensitive optical coherence tomography. A polarization symmetry constraint, intrinsic to imaging in the backscatter direction, facilitates the required compensation for wavelength-dependent transmission through system elements, the rotating catheter, and overlying tissue layers. Applied to intravascular imaging of coronary atherosclerosis in human patients, the optic axis affords refined interpretation of plaque architecture.