This article presents a prototype 4-tap indirect time-of-flight (iToF) CMOS imager with high-speed (HS) range imaging capacity. The sensor features an HS charge modulator, in-pixel memory array, and ...sub-frame ToF operation, enabling up to 10 Kfps range imaging with eight recording frames and <inline-formula> <tex-math notation="LaTeX"><</tex-math> </inline-formula>1.82% depth noise for 0.3-1.4 m range under HS mode. This sensor also operates in high-precision (HP) mode, achieving <inline-formula> <tex-math notation="LaTeX"><</tex-math> </inline-formula>1.77% depth noise for the 0.4-5.4 m range at 90 frames/s by averaging the subframe signals. With a pixel size of 22.4<inline-formula> <tex-math notation="LaTeX">^{H}</tex-math> </inline-formula> <inline-formula> <tex-math notation="LaTeX">\times</tex-math> </inline-formula> 16<inline-formula> <tex-math notation="LaTeX">^{V}</tex-math> </inline-formula> <inline-formula> <tex-math notation="LaTeX">\mu</tex-math> </inline-formula>m and 134<inline-formula> <tex-math notation="LaTeX">^{H}</tex-math> </inline-formula> <inline-formula> <tex-math notation="LaTeX">\times</tex-math> </inline-formula> 132<inline-formula> <tex-math notation="LaTeX">^{V}</tex-math> </inline-formula> 4-tap pixel array, the sensor successfully demonstrated precise depth imaging under HP mode and clear 3-D imaging for rapid motion objects under HS mode. The potential application of the sensor and future improvements are also discussed.
Display omitted
•This paper presents a prototypical system based on RGB-D cameras and compares it to a gold standard for gait analysis.•The new system uses 3 interconnected RGB-D cameras and ...BodyTracking software to calculate gait parameters.•Analyzed variability within parameters of healthy gait and gait impaired by a neurological disorder.•Significant gait parameters (step length, cadence and velocity) could be measured with sufficient accuracy.•Important step towards comprehensive self-administered camera-based kinetic analysis.
Gait feature analysis plays an important role in diagnosing and monitoring diseases that compromise motor function. This article presents the results of a study, which was aimed at assessing the accuracy and precision of computer-aided gait feature analysis performed with a system based on Microsoft® Azure™ Kinect™ Cameras (AzureKinect).
Can an AzureKinect-based system measure basic gait parameters with sufficient accuracy for motor status assessments?
The presented AzureKinect-based system was evaluated by measuring the step length (SL), cadence, and velocity, which are important gait features, of both healthy participants and participants with a neurological motor impairment (total number of participants: N = 24). The GAITRite® system, which is an established gold standard for gait analysis, was used as the ground truth.
The results show that the AzureKinect-based system can provide measurements of average SL, cadence, and velocity. A comparison with the ground truth revealed a mean absolute error (MAE) of 1.74 cm in SL, 4.6 cm/s in gait velocity and 6.3 steps/min for cadence. Pearson’s correlation coefficients range from r = 0.8 to r = 0.99, demonstrating a very high correlation between the measurements of the AzureKinect system and the ground truth.
The AzureKinect-based system is able to measure basic gait parameters with sufficient accuracy. This is a first step towards a comprehensive self-measuring marker-less camera-based kinematic analysis that could be performed at home or in general medical practices.
This paper presents the first attempt at fusing data from inertial and vision depth sensors within the framework of a hidden Markov model for the application of hand gesture recognition. The data ...fusion approach introduced in this paper is general purpose in the sense that it can be used for recognition of various body movements. It is shown that the fusion of data from the vision depth and inertial sensors act in a complementary manner leading to a more robust recognition outcome compared with the situations when each sensor is used individually on its own. The obtained recognition rates for the single hand gestures in the Microsoft MSR data set indicate that our fusion approach provides improved recognition in real-time and under realistic conditions.
In this study, we investigated the hydrodynamic characteristics of marine aquaculture net cages used for farming silver salmon (Oncorhynchus kisutch). Two sets of experiments were conducted: ...model-scale testing, which was used to measure the drag force and deformation of a model net cage at different current velocities, and full-scale testing, which involved a sea trial at a silver salmon farm site. In the sea trial, an acoustic wave and current meter was deployed to record the current velocities around the net cage, and the cage deformation was estimated from the several vertical positions of net using 13 depth sensors. The precision of estimating the cage deformation using only the depth information was evaluated by comparing the estimated cage deformation with that determined from the images in model-scale testing. Comparing the model-scale and full-scale testing results showed that only the bottom netting was found to be deformed under lower water currents. The trends observed for the drag force, cage deformation, and cross-section area estimated from the depth data in full-scale testing were generally consistent with those converted from the model-scale testing using Tauti's similarity law. However, the drag force values of a full-scale net cage converted from the model-scale testing were higher than those estimated from the depth data in the full-scale testing. In contrast, the converted cross-sectional areas from model-scale testing were smaller than the estimated values in full-scale testing. In future, the cage deformation should be examined under higher current velocities, and the drag force should be measured in the full-scale testing to validate the results of the model-scale testing and the hydrodynamic model.
•Drag force and cage deformation of an aquaculture net cage were studied by model-scale tests and full-scale sea trails.•Deformations of model cage estimated using only depth information were compared with those results obtained based on the image processing.•Compared the estimated results in full-scale testing with converted results using Tauti's law from model-scale testing.•Converted results from Tauti's model similarity law differed from estimated ones.
An imager for time-resolved optical sensing was fabricated in CMOS technology. The sensor comprises an array of 128times128 single-photon pixels, a bank of 32 time-to-digital-converters, and a 7.68 ...Gbps readout system. Thanks to the outstanding abstract truncated by publisher.
During COVID-19 pandemic, analysis on virus exposure and intervention efficiency in public transports based on real passenger’s close contact behaviors is critical to curb infectious disease ...transmission. A monitoring device was developed to gather a total of 145,821 close contact data in subways based on semi-supervision learning. A virus transmission model considering both short- and long-range inhalation and deposition was established to calculate the virus exposure. During rush-hour, short-range inhalation exposure is 3.2 times higher than deposition exposure and 7.5 times higher than long-range inhalation exposure of all passengers in the subway. The close contact rate was 56.1 % and the average interpersonal distance was 0.8 m. Face-to-back was the main pattern during close contact. Comparing with random distribution, if all passengers stand facing in the same direction, personal virus exposure through inhalation (deposition) can be reduced by 74.1 % (98.5 %). If the talk rate was decreased from 20 % to 5 %, the inhalation (deposition) exposure can be reduced by 69.3 % (73.8 %). In addition, we found that virus exposure could be reduced by 82.0 % if all passengers wear surgical masks. This study provides scientific support for COVID-19 prevention and control in subways based on real human close contact behaviors.
Display omitted
•Average interpersonal distance is 0.8 m during rush hours in subway.•Close contact rate in subway is 56.1 % and face-to-back is the main pattern.•Short-range inhalation exposure is 3.2 times higher than deposition exposure.•Virus exposure could be reduced by 74.1–98.5 % if all passengers stand in order.•Virus exposure could be reduced by 82.0 % if all passengers wear surgical masks.
In this paper, we address the problem of capturing both the shape and the pose of a human character using a single depth sensor. Some previous works proposed to fit a parametric generic human ...template into the depth image, while others developed deep learning (DL) approaches to find the correspondence between depth pixels and vertices of the template. We designed a hybrid approach, combining the advantages of both methods, and conducted extensive experiments on the SURREAL Varol et al. (2017), DFAUST datasets Bogo etal (2017) and a subset of AMASS Mahmood et al. (2019). Results show that this hybrid approach enables us to enhance pose and shape estimation compared to using DL or model fitting separately. We also evaluated the ability of the DL-based dense correspondence method to segment also the background — not only the body parts. We also evaluated 4 different methods to perform the model fitting based on a dense correspondence, where the number of available 3D points differs from the number of corresponding template vertices. These two results enabled us to better understand how to combine DL and model fitting, and the potential limits of this approach to deal with real-depth images. Future works could explore the potential of taking temporal information into account, which has proven to increase the accuracy of pose and shape reconstruction based on a unique depth or RGB image.
Display omitted
•Dense correspondence and model fitting enhance 3D shape and pose reconstruction.•Bodypart and background color embedding enables character segmentation in depth map.•Averaging 3D candidate points before model fitting offers the best reconstruction.
This article proposes an optimization based on guaranteed coverage to address the challenge of continuously optimizing the mobile depth sensor network online due to the insufficient information ...captured in an unknown environment. In this optimization framework, the coverage set representing the coverage area of the depth sensor is introduced. A novel concept of the guaranteed coverage set of depth sensor is proposed, which is a subset of the coverage set. The proposed guaranteed coverage set, plays a crucial role in addressing measurement noises and enabling proper selection of dithers in optimization algorithms to stay within the coverage area. By utilizing the Luus-Jaakola (LJ) algorithm, the optimal pose for each depth sensor in the mobile network is determined. This takes into account the task-relevant cost function at every time instant. This algorithm ensures that the chosen poses effectively optimize the overall performance of the network. A method for the selection of the parameters in the proposed algorithm is also presented. The proposed optimization is then applied to a path planning case with obstacle avoidance, and a simultaneous mapping and coverage case, using stereo cameras as the depth sensors. Simulation and experiment results verify the effectiveness of the optimization for mobile depth sensor network in unknown environment.
Human detection is a popular topic and difficult problem in surveillance. This paper presents a research on human detection in complex indoor space utilizing a depth sensor. In recent years, target ...detection methods based on RGB-D data mainly include background learning, and feature detection operator. The former method depends on the initial background knowledge obtained from the first couple of frames in the video, and the length of frames decides detection quality. The latter method is more time intensive, and insufficient training samples will influence the detection result. Thus, in this paper we analyze the complex scene features thoroughly and integrate the color and depth information, proposing a RGBD+ViBe foreground extraction method. Based on the extraction outcome of the foreground, this research utilizes the 3D Mean-Shift method combined with depth constraints to handle multi-person partial occlusion problems. The experiment results indicate that the proposed RGBD+ViBe method outperforms the methods which only consider color or depth information, as well as the RGBD+MoG method. Furthermore, the proposed 3D Mean-Shift method achieves nearly 90% accuracy in multi-person detection result, and the false rate is merely 5%; while the accuracy of HOG, HOD and Comb-HOD methods are less than 75% and the false rate is around 10%.
•This study collects a depth-sensor-based people detection dataset in indoor space.•This paper proposes a RGBD+ViBe based foreground extraction method.•This paper proposes a 3D Mean-Shift based multi-person detection method.
3D instance segmentation is a fundamental task in sensor data processing for robot manipulation. However, recent works in this field did not pay much attention to 3D instance segmentation under ...cluttered conditions. Specifically, current widely-used 3D instance segmentation datasets mainly focus on indoor scenes, and they often reconstruct the whole 3D scene by multiple depth sensors. Therefore, there are no occluded or cluttered areas in these datasets. Moreover, we explore prevailing 3D instance segmentation models and find that the traditional heuristic algorithms in these models severely harm their performance in cluttered scenes. To tackle these issues, we first build a monocular depth sensor system to collect a new dataset named Cluttered Objects (CO), which is tailored for robotic manipulation under cluttered conditions. Then, we propose Instance-Guided Net (IGN) to segment targets in these complex areas. In our IGN, the point-wise instance information is fully exploited to make all heuristic algorithms task-oriented. The keys to IGN are Dual-Instance-Guided Block (D-IGB), Instance-Guided Upsampler (IGUS), and Instance-Guided Downsampler (IGDS). In D-IGB, we build the instance-aware receptive field and dilated receptive field to gather intra-instance information and long-range context simultaneously. In IGUS, the instance-aware receptive field is applied to realize task-oriented interpolation. Moreover, we propose IGDS to retain more beneficial foreground features by eliminating background points. Extensive experiments on CO and other public datasets show the effectiveness of our IGN.
