Accurate self-assessment is an important but underdeveloped skill in medicine that, in the past, has received little formal attention from educators. Following an orthopedic rotation, twenty-five orthopedic surgery residents performed a self-assessment task for ten skills using a new relative ranking method, in which an individual's skills are ranked relative to each other rather than being compared to the individual's peers. Supervising faculty assessed residents using the same instrument. Faculty inter-rater reliability was measured and comparisons were made between each resident's self-assessment and the faculty assessments using Spearman rank order correlation coefficients. The mean correlation between faculty rating the same resident was 0.27 (sd = 0.49). The mean correlation between resident and faculty rankings was 0.20 (sd = 0.38), but was higher for junior residents (0.33) than for senior residents (0.12), apparently because senior residents do not alter their self-assessments while faculty change their assessments of senior residents. Consistent with the literature in other fields, we find that self-assessment is poor among surgical trainees when they are asked to assess their own performance over an extended time period.