Under the constraint of limited training samples, achieving robust tracking of hyperspectral data with various target characteristics using a single network is challenging. To address this issue, firstly, we propose a Multi-Band hyperspectral object tracking algorithm based on Spectral Scale-Aware representation (SSAMB). This algorithm can adaptively perceive and capture object characteristics in hyperspectral data across different spectral scales. Secondly, inspired by the "prompt learning" paradigm, we propose a Hyperspectral Object Tracking network based on Spectral Information Prompts (SIPHOT). It aims to fully leverage the powerful tracking capability of the RGB-based model to alleviate the issue of limited training samples affecting the model's generalization performance. Meanwhile, it can effectively utilize the "spectral dimension advantage" of hyperspectral data to enhance object tracking accuracy. The experimental results on the HOT2023 dataset demonstrate that the proposed SSAMB and SIPHOT trackers perform excellently on different spectral data, validating the effectiveness of our methods.