Performance degradation assessment (PDA) of hydraulic pumps is of great importance to preserve operational reliability and ensure safety of hydraulic systems. PDA of hydraulic pumps relies heavily on degradation monitoring data such as vibration data, acoustic emission data and oil data. However, besides the inherent degradation of pumps, the time-varying load conditions (e.g. output pressure) have a significant influence on the behavior of degradation data. This makes PDA more difficult under varying conditions. To address this issue, this paper proposes a conditional factor variational auto-encoder (CFVAE) model whereby variational theory is firstly applied to degradation data decoupling and degradation characteristics extraction. In the training phase, degradation data decoupling is realized by punishing the total correlation term of the model with a hyper-parameter γ and the degradation characteristics can be extracted from latent code units of the model. However, the punishment can result in large reconstruction losses of the model which is not conducive to degradation assessment. Then the degradation label information is integrated into the decoder of the model to decrease reconstruction loss caused by data decoupling. Moreover, a new metric based on inter-class distance and intra-class divergence is introduced to optimize the hyper-parameter and select the best latent code units for degradation characteristics extraction and description. Using these techniques, the CFVAE models corresponding to all degradation states can be well trained. In the testing phase, degradation data from unknown degradation states is fed into all of the trained CFVAE models respectively and minimum distance criterion is introduced to predict the actual degradation state. Taking return oil flow as degradation data, the proposed CFVAE model along with several advanced methodologies is applied to degradation assessment for hydraulic pumps under varying conditions. The assessment results of ten experimental cases verify that the proposed CFVAE model can adapt better to change of conditions and has higher assessment accuracy than other methodologies. The proposed model also shows good robustness in multiple cases, making it likely to apply this model to other mechanical components. •Firstly apply the variational theory to extract degradation characteristics of hydraulic pumps.•Construct a novel CFVAE model to assess performance degradation of hydraulic pumps.•A new metric is introduced for hyper-parameter optimization.•An effective assessment strategy based on minimum distance criteria is proposed.•The proposed degradation assessment method has high accuracy and good robustness.