This manuscript reviews historical and recent studies that focus on supplementary sensory feedback for use in upper limb prostheses. It shows that the inability of many studies to speak to the issue ...of meaningful performance improvements in real-life scenarios is caused by the complexity of the interactions of supplementary sensory feedback with other types of feedback along with other portions of the motor control process. To do this, the present manuscript frames the question of supplementary feedback from the perspective of computational motor control, providing a brief review of the main advances in that field over the last 20 years. It then separates the studies on the closed-loop prosthesis control into distinct categories, which are defined by relating the impact of feedback to the relevant components of the motor control framework, and reviews the work that has been done over the last 50+ years in each of those categories. It ends with a discussion of the studies, along with suggestions for experimental construction and connections with other areas of research, such as machine learning.
Bidirectional human-machine interfaces involve commands from the central nervous system to an external device and feedback characterizing device state. Such feedback may be elicited by electrical ...stimulation of somatosensory nerves, where a task-relevant variable is encoded in stimulation amplitude or frequency. Recently, concurrent modulation in amplitude and frequency (multimodal encoding) was proposed. We hypothesized that feedback with multimodal encoding may effectively be processed by the central nervous system as two independent inputs encoded in amplitude and frequency, respectively, thereby increasing state estimate quality in accordance with maximum-likelihood estimation. Using an adaptation paradigm, we tested this hypothesis during a grasp force matching task where subjects received electrotactile feedback encoding instantaneous force in amplitude, frequency, or both, in addition to their natural force feedback. The results showed that adaptations in grasp force with multimodal encoding could be accurately predicted as the integration of three independent inputs according to maximum-likelihood estimation: amplitude modulated electrotactile feedback, frequency modulated electrotactile feedback, and natural force feedback (r
= 0.73). These findings show that multimodal electrotactile feedback carries an intrinsic advantage for state estimation accuracy with respect to single-variable modulation and suggest that this scheme should be the preferred strategy for bidirectional human-machine interfaces with electrotactile feedback.
The stringent timing and reliability requirements in mission-critical applications require a detailed statistical characterization of end-to-end latency. Teleoperation is a representative use case, ...in which a human operator (HO) remotely controls a robot by exchanging command and feedback signals. We present a framework to analyze the latency of a closed-loop teleoperation system consisting of three entities: an HO, a robot located in remote environment, and a Base Station (BS) with Mobile edge Computing (MEC) capabilities. A model of each component is used to analyze the closed-loop latency and optimize the compression strategy. The closed-form expression of the distribution of the closed-loop latency is difficult to estimate, such that suitable upper and lower bounds are obtained. We formulate a non-convex optimization problem to minimize the closed-loop latency. Using the obtained upper and lower bound on the closed-loop latency, a computationally efficient procedure to optimize the closed-loop latency is presented. The simulation results reveal that compression of sensing data is not always beneficial, while system design based on average performance leads to under-provisioning and may cause performance degradation. The applicability of the proposed analysis is much wider than teleoperation, including a large class of systems whose latency budget consists of many components.
Modern myoelectric prostheses can perform multiple functions (e.g., several grasp types and wrist rotation) but their intuitive control by the user is still an open challenge. It has been recently ...demonstrated that semi-autonomous control can allow the subjects to operate complex prostheses effectively; however, this approach often requires placing sensors on the user. The present study proposes a system for semi-autonomous control of a myoelectric prosthesis that requires a single depth sensor placed on the dorsal side of the hand. The system automatically pre-shapes the hand (grasp type, size, and wrist rotation) and allows the user to grasp objects of different shapes, sizes and orientations, placed individually or within cluttered scenes. The system "reacts" to the side from which the object is approached, and enables the user to target not only the whole object but also an object part. Another unique aspect of the system is that it relies on online interaction between the user and the prosthesis; the system reacts continuously on the targets that are in its focus, while the user interprets the movement of the prosthesis to adjust aiming. Experimental assessment was conducted in ten able-bodied participants to evaluate the feasibility and the impact of training on prosthesis-user interaction. The subjects used the system to grasp a set of objects individually (Phase I) and in cluttered scenarios (Phase II), while the time to accomplish the task (TAT) was used as the performance metric. In both phases, the TAT improved significantly across blocks. Some targets (objects and/or their parts) were more challenging, requiring thus significantly more time to handle, but all objects and scenes were successfully accomplished by all subjects. The assessment therefore demonstrated that the system is indeed robust and effective, and that the subjects could successfully learn how to aim with the system after a brief training. This is an important step toward the development of a self-contained semi-autonomous system convenient for clinical applications.
Electrocutaneous stimulation is a promising approach to provide sensory feedback to amputees, and thus close the loop in upper limb prosthetic systems. However, the stimulation introduces artifacts ...in the recorded electromyographic (EMG) signals, which may be detrimental for the control of myoelectric prostheses. In this study, artifact blanking with three data segmentation approaches was investigated as a simple method to restore the performance of pattern recognition in prosthesis control (eight motions) when EMG signals are corrupted by stimulation artifacts. The methods were tested over a range of stimulation conditions and using four feature sets, comprising both time and frequency domain features. The results demonstrated that when stimulation artifacts were present, the classification performance improved with blanking in all tested conditions. In some cases, the classification performance with blanking was at the level of the benchmark (artifact-free data). The greatest pulse duration and frequency that allowed a full performance recovery were 400 μs and 150 Hz, respectively. These results show that artifact blanking can be used as a practical solution to eliminate the negative influence of the stimulation artifact on EMG pattern classification in a broad range of conditions, thus allowing to close the loop in myoelectric prostheses using electrotactile feedback.
Abstract
Background
Closing the control loop in myoelectric prostheses by providing artificial somatosensory feedback is recognized as an important goal. However, designing a feedback interface that ...is effective in realistic conditions is still a challenge. Namely, in some situations, feedback can be redundant, as the information it provides can be readily obtained through hearing or vision (e.g., grasping force estimated from the deformation of a compliant object). EMG feedback is a non-invasive method wherein the tactile stimulation conveys to the user the level of their own myoelectric signal, hence a measurement intrinsic to the interface, which cannot be accessed incidentally.
Methods
The present study investigated the efficacy of EMG feedback in prosthesis force control when 10 able-bodied participants and a person with transradial amputation used a myoelectric prosthesis to grasp compliant objects of different stiffness values. The performance with feedback was compared to that achieved when the participants relied solely on incidental cues.
Results
The main outcome measures were the task success rate and completion time. EMG feedback resulted in significantly higher success rates regardless of pin stiffness, indicating that the feedback enhanced the accuracy of force application despite the abundance of incidental cues. Contrary to expectations, there was no difference in the completion time between the two feedback conditions. Additionally, the data revealed that the participants could produce smoother control signals when they received EMG feedback as well as more consistent commands across trials, signifying better control of the system by the participants.
Conclusions
The results presented in this study further support the efficacy of EMG feedback when closing the prosthesis control loop by demonstrating its benefits in particularly challenging conditions which maximized the utility of intrinsic feedback sources.
Myoelectric activity volitionally generated by the user is often used for controlling hand prostheses in order to replicate the synergistic actions of muscles in healthy humans during grasping. ...Muscle synergies in healthy humans are based on the integration of visual perception, heuristics and proprioception. Here, we demonstrate how sensor fusion that combines artificial vision and proprioceptive information with the high-level processing characteristics of biological systems can be effectively used in transradial prosthesis control.
We developed a novel context- and user-aware prosthesis (CASP) controller integrating computer vision and inertial sensing with myoelectric activity in order to achieve semi-autonomous and reactive control of a prosthetic hand. The presented method semi-automatically provides simultaneous and proportional control of multiple degrees-of-freedom (DOFs), thus decreasing overall physical effort while retaining full user control. The system was compared against the major commercial state-of-the art myoelectric control system in ten able-bodied and one amputee subject. All subjects used transradial prosthesis with an active wrist to grasp objects typically associated with activities of daily living.
The CASP significantly outperformed the myoelectric interface when controlling all of the prosthesis DOF. However, when tested with less complex prosthetic system (smaller number of DOF), the CASP was slower but resulted with reaching motions that contained less compensatory movements. Another important finding is that the CASP system required minimal user adaptation and training.
The CASP constitutes a substantial improvement for the control of multi-DOF prostheses. The application of the CASP will have a significant impact when translated to real-life scenarious, particularly with respect to improving the usability and acceptance of highly complex systems (e.g., full prosthetic arms) by amputees.
Abstract
Background
One of the drawbacks of lower-limb prostheses is that they do not provide explicit somatosensory feedback to their users. Electrotactile stimulation is an attractive technology to ...restore such feedback because it enables compact solutions with multiple stimulation points. This allows stimulating a larger skin area to provide more information concurrently and modulate parameters spatially as well as in amplitude. However, for effective use, electrotactile stimulation needs to be calibrated and it would be convenient to perform this procedure while the subject is seated. However, amplitude and spatial perception can be affected by motion and/or physical coupling between the residual limb and the socket. In the present study, we therefore evaluated and compared the psychometric properties of multichannel electrotactile stimulation applied to the thigh/residual limb during sitting versus walking.
Methods
The comprehensive assessment included the measurement of the sensation and discomfort thresholds (ST & DT), just noticeable difference (JND), number of distinct intervals (NDI), two-point discrimination threshold (2PD), and spatial discrimination performance (SD). The experiment involved 11 able-bodied participants (4 females and 7 males; 29.2 ± 3.8 years), 3 participants with transtibial amputation, and 3 participants with transfemoral amputation.
Results
In able-bodied participants, the results were consistent for all the measured parameters, and they indicated that both amplitude and spatial perception became worse during walking. More specifically, ST and DT increased significantly during walking vs. sitting (2.90 ± 0.82 mA vs. 2.00 ± 0.52 mA; p < 0.001 for ST and 7.74 ± 0.84 mA vs. 7.21 ± 1.30 mA; p < 0.05 for DT) and likewise for the JND (22.47 ± 12.21% vs. 11.82 ± 5.07%; p < 0.01), while the NDI became lower (6.46 ± 3.47 vs. 11.27 ± 5.18 intervals; p < 0.01). Regarding spatial perception, 2PD was higher during walking (69.78 ± 17.66 mm vs. 57.85 ± 14.87 mm; p < 0.001), while the performance of SD was significantly lower (56.70 ± 10.02% vs. 64.55 ± 9.44%; p < 0.01). For participants with lower-limb amputation, the ST, DT, and performance in the SD assessment followed the trends observed in the able-bodied population. The results for 2PD and JND were however different and subject-specific.
Conclusion
The conducted evaluation demonstrates that electrotactile feedback should be calibrated in the conditions in which it will be used (e.g., during walking). The calibration during sitting, while more convenient, might lead to an overly optimistic (or in some cases pessimistic) estimate of sensitivity. In addition, the results underline that calibration is particularly important in people affected by lower-limb loss to capture the substantial variability in the conditions of the residual limb and prosthesis setup. These insights are important for the implementation of artificial sensory feedback in lower-limb prosthetics applications.
Closing the prosthesis control loop by providing artificial somatosensory feedback can improve utility and user experience. Additionally, closed-loop control should be more robust with respect to ...disturbance, but this might depend on the type of feedback provided. Thus, the present study investigates and compares the performance of EMG and force feedback in the presence of control disturbances. Twenty able-bodied subjects and one transradial amputee performed delicate and power grasps with a prosthesis in a functional task, while the control signal gain was temporarily increased (high-gain disturbance) or decreased (low-gain disturbance) without their knowledge. Three outcome measures were considered: the percentage of trials successful in the first attempt (reaction to disturbance), the average number of attempts in trials where the wrong force was initially applied (adaptation to disturbance), and the average completion time of the last attempt in every trial. EMG feedback was shown to offer significantly better performance compared to force feedback during power grasping in terms of reaction to disturbance and completion time. During power grasping with high-gain disturbance, the median first-attempt success rate was significantly higher with EMG feedback (73.3%) compared to that achieved with force feedback (60%). Moreover, the median completion time for power grasps with low-gain disturbance was significantly longer with force feedback than with EMG feedback (3.64 against 2.48 s, an increase of 32%). Contrary to our expectations, there was no significant difference between feedback types with regards to adaptation to disturbances and the two feedback types performed similarly in delicate grasps. The results indicated that EMG feedback displayed better performance than force feedback in the presence of control disturbances, further demonstrating the potential of this approach to provide a reliable prosthesis-user interaction.
To effectively replace the human hand, a prosthesis should seamlessly respond to user intentions but also convey sensory information back to the user. Restoration of sensory feedback is rated highly ...by the prosthesis users, and feedback is critical for grasping in able-bodied subjects. Nonetheless, the benefits of feedback in prosthetics are still debated. The lack of consensus is likely due to the complex nature of sensory feedback during prosthesis control, so that its effectiveness depends on multiple factors (e.g., task complexity, user learning).
We evaluated the impact of these factors with a longitudinal assessment in six amputee subjects, using a clinical setup (socket, embedded control) and a range of tasks (box and blocks, block turn, clothespin and cups relocation). To provide feedback, we have proposed a novel vibrotactile stimulation scheme capable of transmitting multiple variables from a multifunction prosthesis. The subjects wore a bracelet with four by two uniformly placed vibro-tactors providing information on contact, prosthesis state (active function), and grasping force. The subjects also completed a questionnaire for the subjective evaluation of the feedback.
The tests demonstrated that feedback was beneficial only in the complex tasks (block turn, clothespin and cups relocation), and that the training had an important, task-dependent impact. In the clothespin relocation and block turn tasks, training allowed the subjects to establish successful feedforward control, and therefore, the feedback became redundant. In the cups relocation task, however, the subjects needed some training to learn how to properly exploit the feedback. The subjective evaluation of the feedback was consistently positive, regardless of the objective benefits. These results underline the multifaceted nature of closed-loop prosthesis control as, depending on the context, the same feedback interface can have different impact on performance. Finally, even if the closed-loop control does not improve the performance, it could be beneficial as it seems to improve the subjective experience.
Therefore, in this study we demonstrate, for the first time, the relevance of an advanced, multi-variable feedback interface for dexterous, multi-functional prosthesis control in a clinically relevant setting.
PDF
