It is unavoidable for a soft manipulator to interact with environments during some tasks. These interactions may affect the soft manipulator and make the kinematic model different from the one in free space, e.g., the soft manipulator's effective length and the target positions might change. In order to apply the soft manipulator to constrained environments, an adaptive visual servo controller based on piecewise-constant curvature kinematic, without knowing the true values of the manipulator's length and the target positions, is proposed in this paper. Experimental results in the free space, constrained environment, and the gravity-influenced environment, demonstrate the convergence of the image errors under the proposed controller.