The Leap Motion Controller is a new device for hand gesture controlled user interfaces with declared sub-millimeter accuracy. However, up to this point its capabilities in real environments have not ...been analyzed. Therefore, this paper presents a first study of a Leap Motion Controller. The main focus of attention is on the evaluation of the accuracy and repeatability. For an appropriate evaluation, a novel experimental setup was developed making use of an industrial robot with a reference pen allowing a position accuracy of 0.2 mm. Thereby, a deviation between a desired 3D position and the average measured positions below 0.2 mm has been obtained for static setups and of 1.2 mm for dynamic setups. Using the conclusion of this analysis can improve the development of applications for the Leap Motion controller in the field of Human-Computer Interaction.
Full text
Available for:
IZUM, KILJ, NUK, PILJ, PNG, SAZU, UL, UM, UPUK
Hand gesture recognition is still a topic of great interest for the computer vision community. In particular, sign language and semaphoric hand gestures are two foremost areas of interest due to ...their importance in human-human communication and human-computer interaction, respectively. Any hand gesture can be represented by sets of feature vectors that change over time. Recurrent neural networks (RNNs) are suited to analyze this type of set thanks to their ability to model the long-term contextual information of temporal sequences. In this paper, an RNN is trained by using as features the angles formed by the finger bones of the human hands. The selected features, acquired by a leap motion controller sensor, are chosen because the majority of human hand gestures produce joint movements that generate truly characteristic corners. The proposed method, including the effectiveness of the selected angles, was initially tested by creating a very challenging dataset composed by a large number of gestures defined by the American sign language. On the latter, an accuracy of over 96% was achieved. Afterwards, by using the Shape Retrieval Contest (SHREC) dataset, a wide collection of semaphoric hand gestures, the method was also proven to outperform in accuracy competing approaches of the current literature.
•An American Sign Language recognition model was developed using Leap Motion.•LSTM-RNN with kNN method was proposed for recognition 26 alphabets.•3D motion of hand gesture and relevant 30 features ...were extracted.•26 alphabets with recognition rate of 99.44% accuracy was obtained.
Though American sign language (ASL) has gained recognition from the American society, few ASL applications have been developed with educational purposes. Those designed with real-time sign recognition systems are also lacking. Leap motion controller facilitates the real-time and accurate recognition of ASL signs. It allows an opportunity for designing a learning application with a real-time sign recognition system that seeks to improve the effectiveness of ASL learning. The project proposes an ASL learning application prototype. The application would be a whack-a-mole game with a real-time sign recognition system embedded. Since both static and dynamic signs (J, Z) exist in ASL alphabets, Long-Short Term Memory Recurrent Neural Network with k-Nearest-Neighbour method is adopted as the classification method is based on handling of sequences of input. Characteristics such as sphere radius, angles between fingers and distance between finger positions are extracted as input for the classification model. The model is trained with 2600 samples, 100 samples taken for each alphabet. The experimental results revealed that the recognition rate for 26 ASL alphabets yields an average of 99.44% accuracy rate and 91.82% in 5-fold cross-validation with the use of leap motion controller.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UILJ, UL, UM, UPCLJ, UPUK, ZAGLJ, ZRSKP
We present the results of an evaluation of the performance of the Leap Motion Controller with the aid of a professional, high-precision, fast motion tracking system. A set of static and dynamic ...measurements was performed with different numbers of tracking objects and configurations. For the static measurements, a plastic arm model simulating a human arm was used. A set of 37 reference locations was selected to cover the controller's sensory space. For the dynamic measurements, a special V-shaped tool, consisting of two tracking objects maintaining a constant distance between them, was created to simulate two human fingers. In the static scenario, the standard deviation was less than 0.5 mm. The linear correlation revealed a significant increase in the standard deviation when moving away from the controller. The results of the dynamic scenario revealed the inconsistent performance of the controller, with a significant drop in accuracy for samples taken more than 250 mm above the controller's surface. The Leap Motion Controller undoubtedly represents a revolutionary input device for gesture-based human-computer interaction; however, due to its rather limited sensory space and inconsistent sampling frequency, in its current configuration it cannot currently be used as a professional tracking system.
Full text
Available for:
IZUM, KILJ, NUK, PILJ, PNG, SAZU, UL, UM, UPUK
Dynamic hand gesture recognition is a crucial but challenging task in the pattern recognition and computer vision communities. In this paper, we propose a novel feature vector which is suitable for ...representing dynamic hand gestures, and presents a satisfactory solution to recognizing dynamic hand gestures with a Leap Motion controller (LMC) only. These have not been reported in other papers. The feature vector with depth information is computed and fed into the Hidden Conditional Neural Field (HCNF) classifier to recognize dynamic hand gestures. The systematic framework of the proposed method includes two main steps: feature extraction and classification with the HCNF classifier. The proposed method is evaluated on two dynamic hand gesture datasets with frames acquired with a LMC. The recognition accuracy is 89.5% for the LeapMotion-Gesture3D dataset and 95.0% for the Handicraft-Gesture dataset. Experimental results show that the proposed method is suitable for certain dynamic hand gesture recognition tasks.
Modern hardware and software development has led to an evolution of user interfaces from command-line to natural user interfaces for virtual immersive environments. Gestures imitating real-world ...interaction tasks increasingly replace classical two-dimensional interfaces based on Windows/Icons/Menus/Pointers (WIMP) or touch metaphors. Thus, the purpose of this paper is to survey the state-of-the-art Human-Computer Interaction (HCI) techniques with a focus on the special field of three-dimensional interaction. This includes an overview of currently available interaction devices, their applications of usage and underlying methods for gesture design and recognition. Focus is on interfaces based on the Leap Motion Controller (LMC) and corresponding methods of gesture design and recognition. Further, a review of evaluation methods for the proposed natural user interfaces is given.
Full text
Available for:
IZUM, KILJ, NUK, PILJ, PNG, SAZU, UL, UM, UPUK
•The study presents a method for classification of two types of tremor (Essential tremor and Parkinson tremor).•Convolutional Long Short Term Memory (LSTM) is used for differentiation of the ...essential and Parkinson tremor.•Tremor data is gathered from patients with Leap Motion Controller which is a non-wearable 4D camera sensor.•The method is evaluated on 40 patients and the results showed that convolutional LSTM provides successful results for differentiation of tremor.
This study aims to present a novel method for differentiation of Parkinsonian tremor (PT) and essential tremor (ET) using both postural and resting tremor. A convolutional long short-term memory (LSTM) that learns hand tremor recorded at both postural and resting positions is proposed for differentiation of PT and ET. 3D landmark points of hands at resting and postural positions are gathered using a Leap Motion Controller. A convolutional LSTM is trained for differentiation of ET and PT after preprocessing tremor data. The method is evaluated and tested on a dataset containing 40 subjects where 23 of them had PT and 17 of them had ET. The experiments showed that the accuracy of combined positions using both resting and postural tremor was higher than using single kind of positional tremor. The accuracy of tremor classification is 90% for the combined positions.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UILJ, UL, UM, UPCLJ, UPUK, ZAGLJ, ZRSKP
In the recent years, a steady growth of Hand Gesture Recognition (HGR) based applications has been observed. Thus, significant progress has been made in the field of hand detection and tracking. ...However, hand poses remain particularly challenging and can deeply affect the results due to several factors especially congenital hand problems. Besides, HGR applications performance is very dependent on the sensor used for gesture acquisition. Owing to its high performance, the Leap Motion Controller (LMC), has recently become a by default sensor for HGR systems and provide high performance in terms of recognition accuracy. This work proposes a comprehensive literature review of HGR based on different modalities with a focus on the LMC based systems. This survey retrieves the main challenges faced in literature with an investigation of the state of the art solutions to address them. Moreover, we discuss numerous existing methods for static and dynamic HGR from different modalities. We reveal remaining challenges in the literature and highlight promising future research directions.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UILJ, UL, UM, UPCLJ, UPUK, ZAGLJ, ZRSKP
•Guiding surgical robot arm and laparoscopic scissors by Leap Motion Controller (LMC).•Using contactless Leap Motion Controller to detect hand tremors in an active state.•Applying the Fast Fourier ...Transform (FFT) to analyze Fatigue-Induced Tremor (FIT).•Imitating hand motions by the surgical robot and laparoscopic scissors.
UtilizingNatural User Interfaces (NUIs) to access the patient's data in sterile environments is a practical way tomaintain sterility in the operating rooms orto perform remote surgery.Leap Motion Controller (LMC) has recently gained popularity as a NUI with high measurement accuracy, portability, low cost, and excellent data processing capabilities. This article discusses the possibility of using LMC to directly control surgical robot arms and laparoscopic scissors during surgical procedures. Fatigue-Induced Tremor (FIT) is an unavoidable phenomenon that significantly impairs accuracy in long duration laparoscopic surgeries. Determining this tremor when its properties change with increasing muscle fatigue is one of the most effective methods to improve surgical accuracy. Tremors can be divided into resting and active types. This study aims to evaluate the effects of FIT measured by LMC on the amount of error in controlling the surgical robot arm and laparoscopic scissors. Therefore, it is recommended that some specific movement patterns be developed to measure FIT when the surgeon's hand is active. As the LMC's data is in the time domain, it has been converted to a frequency domain using a Fast Fourier Transform (FFT). Intending to evaluate the surgical robot’s performance with and without hand tremors, the hand motion are considered as a manipulator of surgical robot arm, then the tracking errors in both cases are compared. The effect of tremor on tracking the path and the importance of eliminating the error will be discussed. For this purpose, two types of surgical robot arms and laparoscopic scissors have been used. The primary surgical robot have six degrees of freedom (DoF) and is equipped with a mechanism similar to the grasping tool used in laparoscopic surgeries. A second one consists of an arm with three DoF, that can guide and operate a two DoF laparoscopic scissors. The results indicate that FIT in the hand of a healthy surgeon may lead to an increase in the robot’s tracking error of up to 2.5% in long-duration non-invasive surgeries.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UILJ, UL, UM, UPCLJ, UPUK, ZAGLJ, ZRSKP
Sign language is intentionally designed to allow deaf and dumb communities to convey messages and to connect with society. Unfortunately, learning and practicing sign language is not common among ...society; hence, this study developed a sign language recognition prototype using the Leap Motion Controller (LMC). Many existing studies have proposed methods for incomplete sign language recognition, whereas this study aimed for full American Sign Language (ASL) recognition, which consists of 26 letters and 10 digits. Most of the ASL letters are static (no movement), but certain ASL letters are dynamic (they require certain movements). Thus, this study also aimed to extract features from finger and hand motions to differentiate between the static and dynamic gestures. The experimental results revealed that the sign language recognition rates for the 26 letters using a support vector machine (SVM) and a deep neural network (DNN) are 80.30% and 93.81%, respectively. Meanwhile, the recognition rates for a combination of 26 letters and 10 digits are slightly lower, approximately 72.79% for the SVM and 88.79% for the DNN. As a result, the sign language recognition system has great potential for reducing the gap between deaf and dumb communities and others. The proposed prototype could also serve as an interpreter for the deaf and dumb in everyday life in service sectors, such as at the bank or post office.
Full text
Available for:
IZUM, KILJ, NUK, PILJ, PNG, SAZU, UL, UM, UPUK
PDF
