We study the Langevin dynamics of a physical system with manifold structure M⊂Rp based on collected sample points {xi}i=1n⊂M that probe the unknown manifold M. Through the diffusion map, we first learn the reaction coordinates {yi}i=1n⊂N corresponding to {xi}i=1n, where N is a manifold diffeomorphic to M and isometrically embedded in Rℓ with ℓ≪p. The induced Langevin dynamics on N in terms of the reaction coordinates captures the slow time scale dynamics such as conformational changes in biochemical reactions. To construct an efficient and stable approximation for the Langevin dynamics on N, we leverage the corresponding Fokker-Planck equation on the manifold N in terms of the reaction coordinates y. We propose an implementable, unconditionally stable, data-driven finite volume scheme for this Fokker-Planck equation, which automatically incorporates the manifold structure of N. Furthermore, we provide a weighted L2 convergence analysis of the finite volume scheme to the Fokker-Planck equation on N. The proposed finite volume scheme leads to a Markov chain on {yi}i=1n with an approximated transition probability and jump rate between the nearest neighbor points. After an unconditionally stable explicit time discretization, the data-driven finite volume scheme gives an approximated Markov process for the Langevin dynamics on N and the approximated Markov process enjoys detailed balance, ergodicity, and other good properties.