•Define a new tensor unfolding to unfold an N-way tensor into a three-way tensor.•Propose a novel tensor rank for N-way tensors based on the new tensor unfolding.•Establish a convex relaxation for efficiently minimizing the proposed tensor rank.•Apply the proposed relaxation to tensor recovery problems with ADMM-based solver. The recent popular tensor tubal rank, defined based on tensor singular value decomposition (t-SVD), yields promising results. However, its framework is applicable only to three-way tensors and lacks the flexibility necessary tohandle different correlations along different modes. To tackle these two issues, we define a new tensor unfolding operator, named mode-k1k2 tensor unfolding, as the process of lexicographically stacking all mode-k1k2 slices of an N-way tensor into a three-way tensor, which is a three-way extension of the well-known mode-k tensor matricization. On this basis, we define a novel tensor rank, named the tensor N-tubal rank, as a vector consisting of the tubal ranks of all mode-k1k2 unfolding tensors, to depict the correlations along different modes. To efficiently minimize the proposed N-tubal rank, we establish its convex relaxation: the weighted sum of the tensor nuclear norm (WSTNN). Then, we apply the WSTNN to low-rank tensor completion (LRTC) and tensor robust principal component analysis (TRPCA). The corresponding WSTNN-based LRTC and TRPCA models are proposed, and two efficient alternating direction method of multipliers (ADMM)-based algorithms are developed to solve the proposed models. Numerical experiments demonstrate that the proposed models significantly outperform the compared ones.