Time delay in bilateral control system seriously deteriorates performance and stability. Acceleration-based bilateral control (ABC) is hybrid of position and force control in the acceleration dimension based on a disturbance observer. It can be divided into two modal spaces: common and differential modal spaces. The sum of master force and slave force is controlled to be zero in common modal space to realize the law of action-reaction. The difference of master position and slave position is controlled to be zero in differential modal space for position tracking. This paper analyzes the stability of each modal space under time delay. Based on modal space analysis, this paper proposes a novel "frequency-domain damping design" using high-pass filter for ABC. The validity of the proposed control system is confirmed by some experimental results.