Muscular dystrophies (MD) are a group of genetic neuromuscular disorders that cause progressive weakness and loss of muscles over time, influencing 1 in 3500-5000 children worldwide. New and exciting ...treatment options have led to a critical need for a clinical post-marketing surveillance tool to confirm the efficacy and safety of these treatments after individuals receive them in a commercial setting. For MDs, functional gait assessment is a common approach to evaluate the efficacy of the treatments because muscle weakness is reflected in individuals' walking patterns. However, there is little incentive for the family to continue to travel for such assessments due to the lack of access to specialty centers. While various existing sensing devices, such as cameras, force plates, and wearables can assess gait at home, they are limited by privacy concerns, area of coverage, and discomfort in carrying devices, which is not practical for long-term, continuous monitoring in daily settings. In this study, we introduce a novel functional gait assessment system using ambient floor vibrations, which is non-invasive and scalable, requiring only low-cost and sparsely deployed geophone sensors attached to the floor surface, suitable for in-home usage. Our system captures floor vibrations generated by footsteps from patients while they walk around and analyzes such vibrations to extract essential gait health information. To enhance interpretability and reliability under various sensing scenarios, we translate the signal patterns of floor vibration to pathological gait patterns related to MD, and develop a hierarchical learning algorithm that aggregates insights from individual footsteps to estimate a person's overall gait performance. When evaluated through real-world experiments with 36 subjects (including 15 patients with MD), our floor vibration sensing system achieves a 94.8% accuracy in predicting functional gait stages for patients with MD. Our approach enables accurate, accessible, and scalable functional gait assessment, bringing MD progressive tracking into real life.
Person identification is important in providing personalized services in smart buildings. Many existing studies focus on closed-world person identification, which only identifies a fixed group of ...people who have training data; however, they assume everyone has pre-collected data, which is not practical in real-world scenarios when newcomers are present. To overcome this drawback, open-world person identification recognizes both newcomers and registered people, which opens up new opportunities for smart building applications that involve newcomers, such as smart visitor management, customized retail, personalized health monitoring, and public emergency assistance. To achieve this, structural vibration sensing has various advantages when compared with the existing sensing modalities (e.g., cameras, wearables, and pressure sensors) because it only needs sparsely deployed sensors mounted on the floor, does not require people to carry devices, and is perceived as more privacy-friendly. However, one fundamental challenge in analyzing footstep-induced structural vibration data is its high variability due to the structural heterogeneity and the footstep variations. Therefore, it is difficult to distinguish different people given this high variability within each person, and it is more challenging to recognize a new person as that data is unobserved before.
In this paper, we characterize the variability in footstep-induced structural vibration to develop an open-world person identification framework. Specifically, we address three variability challenges in developing our method. First, the high variability within each person comes from multiple sources that are entangled in the vibration signals, and thus is difficult to be decomposed and reduced. Secondly, the distribution of features extracted from the vibration signals is irregularly shaped, and therefore is difficult to model. Moreover, the identity of the next person is correlated with the previous observations, which makes the identification process more complicated. To overcome these challenges, we first characterize multiple variability sources and design a transformation function that results in signal features that are less variable within one person and more separable between different people. We then develop a modified Chinese Restaurant Process (mCRP) for nonparametric Bayesian modeling to capture the irregularly shaped feature patterns both from local and global perspectives. Finally, we design an adaptive hyperparameter α that represents the prior probability of newcomers at each observation, which keeps updating depending on the time, location, and previous predictions. We evaluate our approach through walking experiments with 20 people across 2 different structures. With only 1 pre-recorded person at each structure, our method achieves up to 92.3% average accuracy with randomly appearing newcomers.
•We localize occupants using footstep-induced structural vibrations.•We mitigate dispersion-induced signal distortions through signal decomposition.•We locate footsteps through adaptive ...multilateration in heterogeneous floor structures.•The results show an 0.34 m localization error (6X improvements from baseline).
In this paper, we present an occupant localization approach through sensing footstep-induced floor vibrations. Occupant location information is an important part of many smart building applications, such as energy and space management in a personal home or patient tracking in a hospital room. Adoption of current occupant location sensing approaches in smart buildings (e.g., camera, radio frequency (RF), mobile devices, etc.) is often limited due to the maintenance, installment, and calibration requirements of these sensing systems. To overcome these limitations, we introduce a new approach to use footstep-induced structural vibration for step-level occupant localization. The intuition behind this approach is that footsteps induce floor vibrations which are received in different vibration sensor locations at different times. This paper focuses on localizing a single occupant within each sensing range. To localize the footsteps, we utilize the time differences of arrival (TDoA) of the footstep-induced vibrations. However, this approach involves two main challenges: (1) the vibration wave propagation in the floor is of dispersive nature (i.e., different frequency components travel at different velocities) and (2) due to floor heterogeneity, these wave propagation velocities vary in different structures as well as in different locations in a structure. These issues lead to large localization inaccuracies or calibration requirements. To address dispersion challenge, we present a decomposition-based dispersion mitigation technique which extracts specific components (which have similar propagation characteristics) for localization. To address velocity variations, we introduce an adaptive multilateration approach that employs heuristics to constrain the search space and robustly locate the footsteps when the propagation velocity is unknown. Constraining the search space overcomes the additional complexity which is resulted from adding an unknown variable (propagation velocity). We evaluated our approach using the field experiments in 3 different types of buildings (both commercial and residential) with human participants. The results show an average localization error of 0.34 m, which corresponds to a 6X reduction in error compared to a baseline method. Furthermore, our approach resulted in standard deviation of as low as 0.18 m, which corresponds to a 8.7X improvement in precision compared to the baseline approach.
AbstractThis paper presents a floor-vibration-based step-level occupant-detection approach that enables detection across different structures through model transfer. Detecting the occupants through ...detecting their footsteps (i.e., step-level occupant detection) is useful in various smart building applications such as senior/healthcare and energy management. Current sensing approaches (e.g., vision-based, pressure-based, radio frequency–based, and mobile-based) for step-level occupant detection are limited due to installation and maintenance requirements such as dense deployment and requiring the occupants to carry a device. To overcome these requirements, previous research used ambient structural vibration sensing for footstep modeling and step-level occupant detection together with supervised learning to train a footstep model to distinguish footsteps from nonfootsteps using a set of labeled data. However, floor-vibration-based footstep models are influenced by the structural properties, which may vary from structure to structure. Consequently, a footstep model in one structure does not accurately capture the responses in another structure, which leads to high detection errors and the costly need for acquiring labeled data in every structure. To address this challenge, the effect of the structure on the footstep-induced floor vibration responses is here characterized to develop a physics-driven model transfer approach that enables step-level occupant detection across structures. Specifically, the proposed model transfer approach projects the data into a feature space in which the structural effects are minimized. By minimizing the structure effect in this projected feature space, the footstep models mainly represent the differences in the excitation types and therefore are transferable across structures. To this end, it is analytically shown that the structural effects are correlated to the maximum-mean-discrepancy (MMD) distance between the source and target marginal data distributions. Therefore, to reduce the structural effect, the MMD between the distributions in the source and target structures is minimized. The robustness of the proposed approach was evaluated through field experiments in three types of structures. The evaluation consists of training a footstep model in a set of structures and testing it in a different structure. Across the three structures, the evaluation results show footstep detection F1 score of up to 99% for the proposed approach, corresponding to a 29-fold improvement compared to the baseline approach, which do not transfer the model.
•We characterize the effects of obstructions on footstep-induced floor vibrations.•This characterization enables obstruction-invariant indoor occupant localization while reducing the sensing ...requirements.•We first detect and estimate the obstruction mass by characterizing the attenuation.•Then, we find the propagation velocities by modeling the velocity-mass relationship.•Results shows 1.6X improvement compared to the baseline approach.
In this paper, we characterize the effects of obstructions on footstep-induced floor vibrations to enable obstruction-invariant indoor occupant localization. Occupant localization is important in smart building applications such as smart healthcare and energy management. Maintenance and installment requirements limit the application of current sensing approaches (e.g., mobile-based, RF-based, and pressure-based sensing) in real-life applications. To overcome these limitations, prior work has utilized footstep-induced structural vibrations for occupant localization. The main intuition behind these approaches is that the footstep-induced floor vibration waves take different amounts of time to arrive at different sensors. These Time-Differences-of-Arrival (TDoA) can then be leveraged to locate the footstep by assuming similar velocities between the footstep and various sensor locations. This assumption makes these approaches suitable for open areas; however, real buildings have various types of obstructions (e.g., walls, furniture, etc.) which affect wave propagation velocities and hence significantly reduce localization accuracy. Therefore, the prior work requires unobstructed paths between footsteps and sensors for accurate occupant localization, which increases the sensing density requirement and thus, instrumentation and maintenance costs. We have observed that the obstruction mass is one of the key factors in affecting the wave propagation velocity and reducing the localization accuracy. Therefore, to overcome the obstruction challenge, we localize footsteps by considering different velocities between the footsteps and sensors depending on the existence and mass of obstruction on the wave path. Specifically, we (1) detect and estimate the mass of the obstruction by characterizing the wave attenuation rate, (2) use this estimated mass to find the propagation velocities for localization by modeling the velocity-mass relationship through the lamb wave characteristics, and (3) introduce a non-isotropic multilateration approach which robustly leverages these propagation velocities to locate the footsteps (and the occupants). In field experiments, we achieved average localization error of 0.61 meters, which is (1) the same as the average localization error when there is no obstruction and (2) 1.6X improvement compared to the baseline approach.
AbstractThis paper presents a structure- and sampling-adaptive approach for analyzing human footstep-induced structural floor vibrations to estimate footstep ground reaction forces (GRFs) and gait ...balance symmetry. Balance symmetry and footstep GRFs are critical indicators of overall gait health and elderly fall risks. Prior works, including direct observation by trained medical personnel, computer vision-, pressure sensor-, and wearable-based sensing, are limited due to operational restrictions. We introduce a nonintrusive balance symmetry monitoring approach, which utilizes sparse structural vibration sensing. The intuition is that footstep-induced floor vibration responses are proportional to footstep GRFs, and balance symmetry can be defined using consecutive GRF pairs. However, GRF-vibration relationships are also influenced by spatially-varying structural properties and gait sampling bias, introducing errors to real-world estimations. We address these challenges first by extracting structural regions to overcome spatially-varying vibration behavior and then by developing a kernel-based robust regression model to overcome biased training data and enable robust GRF and balance symmetry modeling. We evaluate our approach through real-world experiments, achieving a balance symmetry index estimation accuracy as high as 96.5%.
The objective of this research is to model, characterize, and analyze structural vibrations generated from multiple concurrent persons to achieve robust human gait health and activity monitoring in ...indoor settings. Human activity monitoring is a critical aspect of smart and connected infrastructure. Identifying, tracking, and monitoring the condition and activity level of occupants provides information about their health status, well-being, and can reduce the occurrence of injury and illness. Prior works in this area leverage direct observation as well as sensing-based approaches including ones using vision, radio frequency/WiFi, pressure, and wearables, but are often limited at building/large-scale due to operational restrictions such as requiring specially trained staff, direct line-of-sight, dense sensor deployment, and/or requiring individual persons to wear or carry a device at all times. To address these limitations, this research uses structural vibration sensing for passive, sparse, and non-intrusive monitoring of human gait health and activities in indoor environments. The primary intuition behind this work is that, when humans move or interact with the structure(s) surrounding them, they excite those structures, which results in structural vibrations. By modeling, characterizing, and analyzing these multiple-person vibrations, the system presented in this work is able to extract information about that human interaction (e.g., footstep/gait information or the specific activity a person is involved in).These structural vibration signals, however, are not only influenced by the activities of interest, but also by human- and structure-based factors. These factors constitute the primary system challenges addressed through this dissertation work, and can be summarized as the: 1) multiple person challenge; 2) structural variation challenge; and 3) human behavior challenge.First, this dissertation models vibration responses from multiple concurrent persons as a sparse combination of individual walkers' responses, and then decomposes them into their individualized components. The presence of multiple concurrent persons in the sensing area results in mixed/overlapping signals from each person's gait and/or activities, making it difficult to extract individualized information. Further, the recorded vibration signals in multiple walker scenarios are influenced not only by the persons themselves (i.e., the "sources"), but also by the dynamic properties of the structure itself. Specifically, in the case of multiple concurrent walkers, this codependency on the structure response characteristics makes separation of overlapping and mixed signals difficult using traditional approaches (e.g., blind source separation). This challenge is addressed through a sequential sparse representation approach based on cosine distance which models multiple occupant vibration signals to separate and identify each walker. We evaluate our approach by conducting real-world walking experiments with 3 concurrent walkers and achieve an average F1 score for identifying all persons of 0.89 (2.9X baseline error reduction), and then with a 10 person hybrid dataset (real-world footstep responses with simulated overlap), we observe trace-level accuracies of 100%, 93.3%, and 73% for 2, 3, and 4 person combinations.Second, this dissertation characterizes the relationship between footstep-induced vibration responses and human gait health parameters (e.g., footstep ground reaction forces (GRFs), balance symmetry, etc.). This relationship is affected by spatial variations in the structural characteristics (e.g., damping, stiffness, mass, etc.) of the underlying structure. To overcome this challenge, a structure- and sampling-adaptive approach for analyzing human footstep-induced structural floor vibrations to estimate footstep ground reaction forces (GRFs) and gait balance symmetry is presented. Real-world experimental evaluation demonstrates that our approach realizes an overall force estimation accuracy of up to 92% (1.5X improvement over the baseline approach), and a trace-level balance SI estimation accuracy as high as 96.5% (2.2X baseline improvement) across varying structural conditions, walking speed/balance conditions, and different individuals (i.e., walking styles).Third, this dissertation characterizes the unique components of similar structural vibration responses that are generated by different human behavior. Despite mitigating the effects of multiple walkers and spatial variations in the underlying structure, differences in human behavior can result in similar vibration responses that are difficult to distinguish from one another. These differences in human behavior are most prominent in how each individual performs various daily activities (e.g., hand washing). The primary research challenge is that vibration responses are similar for different activities - a person walking toward the sink area has a similar vibration responses as a person using a soap dispenser (i.e., an impulsive response), while a person rinsing their hands in running water has a very similar response as the water itself (without the presence of hands). This challenge is addressed through a cepstrum-based hierarchical classification approach for hand washing activity monitoring where the vibration responses are transformed to a new space where the periodicity of the signal is more prominent, and similar signals are more easily distinguished. We evaluate our approach using real-world hand washing data across 4 different structures, and achieve an average F1 score for hand washing activities of 0.95, which represents as much as a 10.2X error reduction over naive baseline approaches.By modeling, characterizing, and analyzing multiple person footstep- and activity-induced structural vibration responses, this dissertation provides a framework for sparse, passive, and robust monitoring human gait health and activities in indoor settings.
In this paper, we present a multiple concurrent occupant identification approach through footstep-induced floor vibration sensing. Identification of human occupants is useful in a variety of indoor ...smart structure scenarios, with applications in building security, space allocation, and healthcare. Existing approaches leverage sensing modalities such as vision, acoustic, RF, and wearables, but are limited due to deployment constraints such as line-of-sight requirements, sensitivity to noise, dense sensor deployment, and requiring each walker to wear/carry a device. To overcome these restrictions, we use footstep-induced structural vibration sensing. Footstep-induced signals contain information about the occupants' unique gait characteristics, and propagate through the structural medium, which enables sparse and passive identification of indoor occupants. The primary research challenge is that multiple-person footstep-induced vibration responses are a mixture of structurally-codependent overlapping individual responses with unknown timing, spectral content, and mixing ratios. As such, it is difficult to determine which part of the signal corresponds to each occupant. We overcome this challenge through a recursive sparse representation approach based on cosine distance that identifies each occupant in a footstep event in the order that their signals are generated, reconstructs their portion of the signal, and removes it from the mixed response. By leveraging sparse representation, our approach can simultaneously identify and separate mixed/overlapping responses, and the use of the cosine distance error function reduces the influence of structural codependency on the multiple walkers' signals. In this way, we isolate and identify each of the multiple occupants' footstep responses. We evaluate our approach by conducting real-world walking experiments with three concurrent walkers and achieve an average F1 score for identifying all persons of 0.89 (1.3x baseline improvement), and with a 10-person "hybrid" dataset (simulated combination of single-walker real-world data), we identify 2, 3, and 4 concurrent walkers with a trace-level accuracy of 100%, 93%, and 73%, respectively, and observe as much as a 2.9x error reduction over a naive baseline approach.
This paper presents an indoor area occupancy counting system utilizing the ambient structural vibration induced by pedestrian footsteps. Our system achieves 99.55% accuracy in pedestrian footsteps ...detection, 0.2 people mean estimation error in pedestrian traffic estimation, and 0.2 area occupant activity estimation error in real-world uncontrolled experiments.
Many human activities induce excitations on ambient structures with various objects, causing the structures to vibrate. Accurate vibration excitation source detection and characterization enable ...human activity information inference, hence allowing human activity monitoring for various smart building applications. By utilizing structural vibrations, we can achieve sparse and non-intrusive sensing, unlike pressure- and vision-based methods. Many approaches have been presented on vibration-based source characterization, and they often either focus on one excitation type or have limited performance due to the dispersion and attenuation effects of the structures. In this paper, we present our method to characterize two main types of excitations induced by human activities (impulse and slip-pulse) on multiple structures. By understanding the physical properties of waves and their propagation, the system can achieve accurate excitation tracking on different structures without large-scale labeled training data. Specifically, our algorithm takes properties of surface waves generated by impulse and of body waves generated by slip-pulse into account to handle the dispersion and attenuation effects when different types of excitations happen on various structures. We then evaluate the algorithm through multiple scenarios. Our method achieves up to a six times improvement in impulse localization accuracy and a three times improvement in slip-pulse trajectory length estimation compared to existing methods that do not take wave properties into account.
PDF
