This paper investigates the joint transceiver design for full-duplex (FD) multiple-input multiple-output relay systems where the direct link (DL) and nonlinear successive-interference-cancellation (SIC) detection are considered. The objective is to jointly optimize the source precoder, relay precoder, and receiver such that the resultant symbol-vector error rate (SVER) is minimized. The joint design is challenging since the coexistence of DL and FD relaying will introduce the inter symbol-vector interference. To address this problem, we first propose a novel QR-SIC receiver structure by performing the dual-filtering (DuF) and channel stacking before SIC. Then, the source and relay precoders are optimized for the proposed DuF-based QR-SIC receiver. Compared with the source precoder, optimizing the relay precoder is much more involved due to the complicated objective function. Hence, we transform the original optimization problem into a trace-minimization one, enabling a closed-form solution for the relay precoder. Furthermore, we theoretically show that the proposed design framework can be extended to the system with the DuF-based minimum mean-squared-error SIC receiver. Simulations show that our transceivers are indeed able to significantly improve the overall SVER performance.