•Present an automated framework for assessing dynamic characteristics of building structures with a conjunction of marker-free vision approach.•Formulate an algorithm for deriving lateral stiffness from displacement data by reconstructing the equation of motion.•Employ image convex hull optimization method to realize marker-free vision-based displacement sensor (MVDS).•Validate the propose framework in numerical and experimental ways.•Show distinct aspects of the proposed framework monitoring dynamic characteristics exceeding the Nyquist frequency of sensors. Vision-based displacement sensors (VDSs) are drawing attention as next-generation methods for monitoring buildings due to easy installation, however, there is a further need to develop integrated techniques both to obtain displacement at desired positions and to assess dynamic characteristics with high-precision. This study aimed to develop an automated framework to assess the dynamic characteristics of buildings through the derivation of lateral stiffness using a marker-free vision-based displacement sensor (MVDS). The MVDS utilizes image convex hull optimization to measure displacement at user-defined positions without ancillary markers. Then, the dynamic characteristics were estimated from eigenvalue analysis by reconstructing the equation of motion based on lateral stiffness derived through linear regression in load-displacement curves using the measured displacement data. From numerical simulation and shake table test, the results showed that the proposed framework enables monitoring of the dynamic characteristics where frequency domain exceeded the Nyquist frequency of the sensor compared to FFT-based analysis.