We report the transverse single-spin asymmetries of J/ψ production at forward and backward rapidity, 1.2<|y|<2.2, as a function of J/ψ transverse momentum (pT) and Feynman-x (xF). The data analyzed were recorded by the PHENIX experiment at the Relativistic Heavy Ion Collider in 2015 from p+p, p+Al, and p+Au collisions with transversely polarized proton beams at sNN=200  GeV. At this collision energy, single-spin asymmetries for heavy-flavor particle production of p+p collisions provide access to the spin-dependent gluon distribution and higher-twist correlation functions inside the nucleon, such as the gluon Qiu-Sterman and trigluon correlation functions. Proton+nucleus collisions offer an excellent opportunity to study nuclear effects on the correlation functions. The data indicate a positive asymmetry at the two-standard-deviation level in the p+p data for 2  GeV/c<pT<10  GeV/c at backward rapidity and negative asymmetries at the two-standard-deviation level in the p+Au data for pT<2  GeV/c at both forward and backward rapidity, while in p+Al collisions the asymmetries are consistent with zero within the range of experimental uncertainties.