We introduce a real-time and calibration-free facial performance capture framework based on a sensor with video and depth input. In this framework, we develop an adaptive PCA model using shape ...correctives that adjust on-the-fly to the actor's expressions through incremental PCA-based learning. Since the fitting of the adaptive model progressively improves during the performance, we do not require an extra capture or training session to build this model. As a result, the system is highly deployable and easy to use: it can faithfully track any individual, starting from just a single face scan of the subject in a neutral pose. Like many real-time methods, we use a linear subspace to cope with incomplete input data and fast motion. To boost the training of our tracking model with reliable samples, we use a well-trained 2D facial feature tracker on the input video and an efficient mesh deformation algorithm to snap the result of the previous step to high frequency details in visible depth map regions. We show that the combination of dense depth maps and texture features around eyes and lips is essential in capturing natural dialogues and nuanced actor-specific emotions. We demonstrate that using an adaptive PCA model not only improves the fitting accuracy for tracking but also increases the expressiveness of the retargeted character.
•Occupancy detection and profiling have been demonstrated by using pure depth data.•Depth data are unsuitable to reveal person's identity, allowing to preserve privacy.•The presented framework scales ...well with the number of people.•Validated with two different depth sensors with initially promising results.•Relevant applications for building energy management have been also demonstrated.
In this paper, a computational framework for occupancy detection and profiling based exclusively on depth data is presented. 3D depth sensors offer many advantages against traditional video cameras. Occupants’ privacy can be assured more effectively because depth information is unsuitable to reveal the person's identity. Notable low-level computer vision tasks can be simplified, thus lightening the computational load. The presented framework is suitable for wall-mounting setups as well as for ceiling-mounting setups, and scales well with the number of people. To take full advantage of depth data and to accommodate specificities of crowded environments, several improvements to the standard computer vision pipeline are suggested. Firstly, the running Gaussian average background model is adapted to work with depth distances in crowded scenes. Secondly, the classical complete linkage agglomerative clustering is boosted by adding edge-based constraints specifically designed for people segmentation in depth data. Thirdly, to reliable discriminate people, specific depth-based features are defined to be used with a Real AdaBoost classifier. The preliminary results achieved by using two different depth sensors and synthetic data are very promising, outperforming existing approaches. Relevant applications for building energy management, such as occupancy profiling and construction of trajectories and density maps, have been also demonstrated.
3D self-portraits Li, Hao; Vouga, Etienne; Gudym, Anton ...
ACM transactions on graphics,
11/2013, Letnik:
32, Številka:
6
Journal Article
Recenzirano
We develop an automatic pipeline that allows ordinary users to capture complete and fully textured 3D models of themselves in minutes, using only a single Kinect sensor, in the uncontrolled lighting ...environment of their own home. Our method requires neither a turntable nor a second operator, and is robust to the small deformations and changes of pose that inevitably arise during scanning. After the users rotate themselves with the same pose for a few scans from different views, our system stitches together the captured scans using multi-view non-rigid registration, and produces watertight final models. To ensure consistent texturing, we recover the underlying albedo from each scanned texture and generate seamless global textures using Poisson blending. Despite the minimal requirements we place on the hardware and users, our method is suitable for full body capture of challenging scenes that cannot be handled well using previous methods, such as those involving loose clothing, complex poses, and props.
•Camera viewing angles could affect depth sensors ability to track gait kinematics.•Kinect v2 and Orbbec Astra track kinematics well but only for a frontal view angle.•Azure Kinect is recommended to ...track gait kinematics at non-frontal viewing angles.
Depth sensors could be a portable, affordable, marker-less alternative to three-dimension motion capture systems for gait analysis, but the effects of camera viewing angles on their joint angle tracking performance have not been fully investigated.
This study evaluated the accuracies of three depth sensors Azure Kinect (AK); Kinect v2 (K2); Orbbec Astra (OA) for tracking kinematic gait patterns during treadmill walking at five camera viewing angles (0°/22.5°/45°/67.5°/90°).
Ten healthy subjects performed fifteen treadmill walking trials (3 speeds × 5 viewing angles) using the three depth sensors to measure joint angles in sagittal hip, frontal hip, sagittal knee, and sagittal ankle. Ten walking steps were recorded and averaged for each walking trial. Range of motion in terms of maximum and minimum joint angles measured by the depth sensors were compared with the Vicon motion capture system as the gold standard. Depth sensors tracking accuracies were compared against the Vicon reference using root-mean-square error (RMSE) on the joint angle time series. Effects of different walking speeds, viewing angles, and depth sensors on the tracking accuracy were observed using three-way repeated-measure analysis of variance (ANOVA).
ANOVA results on RMSE showed significant interaction effects between viewing angles and depth sensors for sagittal hip F(8,72) = 4.404, p = 0.005 and for sagittal knee F(8,72)=13.211, p < 0.001 joint angles. AK had better tracking performance when subjects walked at non-frontal camera viewing angles (22.5°/45°/67.5°/90°); while K2 performed better at frontal viewing angle (0°). The superior tracking performance of AK compared with K2/OA might be attributed to the improved depth sensor resolution and body tracking algorithm.
Researchers should be cautious about camera viewing angle when using depth sensors for kinematic gait measurements. Our results demonstrated Azure Kinect had good tracking performance of sagittal hip and sagittal knee joint angles during treadmill walking tests at non-frontal camera viewing angles.
► Introduction to the recent advances in depth sensing technologies. ► Survey of human motion analysis using depth images, including action recognition. ► Covers significant literature that uses ...Kinect or time-of-flight images. ► Details public software libraries/human action/activity datasets based on Kinect. ► Discusses current research and prospective future research directions.
Analysis of human behaviour through visual information has been a highly active research topic in the computer vision community. This was previously achieved via images from a conventional camera, however recently depth sensors have made a new type of data available. This survey starts by explaining the advantages of depth imagery, then describes the new sensors that are available to obtain it. In particular, the Microsoft Kinect has made high-resolution real-time depth cheaply available. The main published research on the use of depth imagery for analysing human activity is reviewed. Much of the existing work focuses on body part detection and pose estimation. A growing research area addresses the recognition of human actions. The publicly available datasets that include depth imagery are listed, as are the software libraries that can acquire it from a sensor. This survey concludes by summarising the current state of work on this topic, and pointing out promising future research directions. For both researchers and practitioners who are familiar with this topic and those who are new to this field, the review will aid in the selection, and development, of algorithms using depth data.
Jumping spiders (Salticidae) rely on accurate depth perception for predation and navigation. They accomplish depth perception, despite their tiny brains, by using specialized optics. Each principal ...eye includes a multitiered retina that simultaneously receives multiple images with different amounts of defocus, and from these images, distance is decoded with relatively little computation. We introduce a compact depth sensor that is inspired by the jumping spider. It combines metalens optics, which modifies the phase of incident light at a subwavelength scale, with efficient computations to measure depth from image defocus. Instead of using a multitiered retina to transduce multiple simultaneous images, the sensor uses a metalens to split the light that passes through an aperture and concurrently form 2 differently defocused images at distinct regions of a single planar photosensor. We demonstrate a system that deploys a 3-mm-diameter metalens to measure depth over a 10-cm distance range, using fewer than 700 floating point operations per output pixel. Compared with previous passive depth sensors, our metalens depth sensor is compact, single-shot, and requires a small amount of computation. This integration of nanophotonics and efficient computation brings artificial depth sensing closer to being feasible on millimeter-scale, microwatts platforms such as microrobots and microsensor networks.
Seabed residential stations are essential for the future exploration of the deep-sea exploration. Deep-sea carriers accompany transport equipment for underwater resident stations. To accommodate the ...low-energy consumption and low-cost nature of deep-sea carriers, an online vertical velocity estimation method based on depth data is proposed. This method serves as a cost-effective alternative to the expensive acoustic and inertial velocity measurement equipment conventionally used in underwater navigation systems. To meet the demand for low-cost online velocity measurements, a dynamic model was created for buoyancy-driven vehicles such as deep-sea carriers during unpowered ascent. Two distinct methods were used to process depth sensor data: one involves central differencing followed by low-pass filtering, and the other entails the differentiation of the data after fitting it to dynamic equations. This approach yields two different types of estimated velocities. A sliding window was defined and used the variance of the velocity and fitting coefficient within the window as indicators to assess the uncertainty of the velocity data. Based on this evaluation, an adaptive weight allocation rule for the fusion process was formulated. This approach allows us to derive the weighted estimated velocity. Through lake trials, the weighted estimated velocity was validated to closely match the ideal velocity data and showed a substantial reduction in velocity variance compared to the differenced estimated velocity that was not fused. This indicates that the fluctuations in the velocity data were significantly reduced. In addition, the integrated displacement showed an average absolute error of 0.1769 m compared with the measured displacement. This indicates that the weighted estimated velocity effectively reflects the true state of motion of the deep-sea carrier. Furthermore, when unexpected motion occurs in deep-sea carriers, the weighted estimated velocity can prevent falling into the "smoothness trap.” This demonstrated the robustness of the proposed algorithm.
•Online vertical velocity estimation method based on depth data.•Estimated velocity closely approximates the actual velocity of the deep-sea carrier.•Optimal estimated velocity showed good smoothness and robustness.•Adaptive weight allocation based on the uncertainty metric.
Survey on 3D Hand Gesture Recognition Cheng, Hong; Yang, Lu; Liu, Zicheng
IEEE transactions on circuits and systems for video technology,
2016-Sept., 2016-9-00, 20160901, Letnik:
26, Številka:
9
Journal Article
Recenzirano
Three-dimensional hand gesture recognition has attracted increasing research interests in computer vision, pattern recognition, and human-computer interaction. The emerging depth sensors greatly ...inspired various hand gesture recognition approaches and applications, which were severely limited in the 2D domain with conventional cameras. This paper presents a survey of some recent works on hand gesture recognition using 3D depth sensors. We first review the commercial depth sensors and public data sets that are widely used in this field. Then, we review the state-of-the-art research for 3D hand gesture recognition in four aspects: 1) 3D hand modeling; 2) static hand gesture recognition; 3) hand trajectory gesture recognition; and 4) continuous hand gesture recognition. While the emphasis is on 3D hand gesture recognition approaches, the related applications and typical systems are also briefly summarized for practitioners.
Indirect time-of-flight depth sensor can obtain depth from the phase offset between emitted and received modulated infrared (IR) pulsed light. However, these sensors suffer from motion blur artifacts ...when there are moving objects in the scene causing depth measurement distortion. By analyzing the mechanism of motion blur, this paper summarizes motion blur into categories: half-frame motion blur and full-frame motion blur. For both blur categories, this paper respectively proposes the deblurring method based on charge correction with the adjacent frame charge reference and the neighborhood similar charge reference. Based on the range resolution equation, the proposed pixel-based blur detection method can adaptively detect motion blur. For detected blurred pixels, the proposed deblurring method removes blur by recalculating the depth of blurred pixels. Motion blur was suppressed at 30 fps owing to the proposed deblurring method. The depth error is about 1.68% over the range of 1-2m, with a modulation frequency of 40 MHz. Experimental results demonstrate that the proposed deblurring method can effectively eliminate the motion blur of moving objects with minimal computational cost.
PDF
