Enabling robots to provide effective assistance yet still accommodating the operator’s commands for telemanipulation of an object is very challenging because robot’s assistance is not always intuitive for human operators and human behaviors and preferences are sometimes ambiguous for the robot to interpret. Due to the difference in hand structures, some motion assistance from the robot may surprise the operator with counter-intuitive movements, which could introduce more burden to the human to correct the actions and/or reduce the operator’s sense of system control. To address these problems, we developed a novel preference-aware assistance knowledge learning approach. An assistance preference model learns what assistance is preferred by a human, and a stage-wise model updating method ensures the learning stability while dealing with the ambiguity of human preference data. Such a preference-aware assistance knowledge enables a teleoperated robot hand to provide more active yet preferred assistance toward manipulation success. We also developed knowledge transfer methods to transfer the preference knowledge across different robot hand structures to avoid extensive robot-specific training. Experiments to telemanipulate a 3-finger hand and 2-finger hand, respectively, to use, move, and hand over a cup have been conducted. Results demonstrated that the methods enabled the robots to effectively learn the preference knowledge and allowed knowledge transfer between robots with less training effort.