Neural scene representation and rendering Eslami, S M Ali; Jimenez Rezende, Danilo; Besse, Frederic ...
Science (American Association for the Advancement of Science),
06/2018, Letnik:
360, Številka:
6394
Journal Article
Recenzirano
Odprti dostop
Scene representation-the process of converting visual sensory data into concise descriptions-is a requirement for intelligent behavior. Recent work has shown that neural networks excel at this task ...when provided with large, labeled datasets. However, removing the reliance on human labeling remains an important open problem. To this end, we introduce the Generative Query Network (GQN), a framework within which machines learn to represent scenes using only their own sensors. The GQN takes as input images of a scene taken from different viewpoints, constructs an internal representation, and uses this representation to predict the appearance of that scene from previously unobserved viewpoints. The GQN demonstrates representation learning without human labels or domain knowledge, paving the way toward machines that autonomously learn to understand the world around them.
Machine vision systems that capture images for visual inspection and recognition tasks must be able to perceive, memorize, and compute any color scene. To achieve this, most of the current visual ...systems use circuits and algorithms which may reduce efficiency and increase complexity. Herein, a 2D semiconductor tungsten diselenide (WSe2)‐based phototransistor that successfully demonstrates an artificial vision system integrating the processing capability of visual information sensing memory, is reported. Furthermore, based on a 6 × 6 fabricated retinal perception array, artificial visual information sensing memory and processing system are proposed to perform image recognition tasks, which can avoid the time delay and energy consumption caused by data conversion and movement. On the other hand, highly linear symmetric synaptic plasticity can be achieved based on the modulation of carrier types in WSe2 transistors with different thicknesses, facilitating the high level of training and inference accuracy for artificial neural networks. Last, through training and inference simulations, the feasibility of the hybrid synapses for optical neural networks (ONN) is demonstrated.
Abstract
In the quest for emerging in-sensor computing, materials that respond to optical stimuli in conjunction with non-volatile phase transition are highly desired for realizing bioinspired ...neuromorphic vision components. Here, we report a non-volatile multi-level control of VO
2
films by oxygen stoichiometry engineering under ultraviolet irradiation. Based on the reversible regulation of VO
2
films using ultraviolet irradiation and electrolyte gating, we demonstrate a proof-of-principle neuromorphic ultraviolet sensor with integrated sensing, memory, and processing functions at room temperature, and also prove its silicon compatible potential through the wafer-scale integration of a neuromorphic sensor array. The device displays linear weight update with optical writing because its metallic phase proportion increases almost linearly with the light dosage. Moreover, the artificial neural network consisting of this neuromorphic sensor can extract ultraviolet information from the surrounding environment, and significantly improve the recognition accuracy from 24% to 93%. This work provides a path to design neuromorphic sensors and will facilitate the potential applications in artificial vision systems.
Adaptation is essential for the life activities of organisms. Simulating biological visual adaptation behavior is still a serious challenge in the development of artificial visual perception systems. ...Here, an organic artificial optoelectronic neuromorphic device is presented, which has a memory window greater than 20V, multi-level storage characteristics and reliable retention characteristics. Furthermore, the simulation of human brain synaptic behaviors, such as paired-pulse facilitation (PPF), transition from short-term memory (STM) to long-term memory (LTM), and learning behaviors, are achieved through optical modulation. Importantly, by coupling optical excitation and electrical modulation, the device successfully mimics the biological visual adaptation function. The proposed device provides a new avenue for the creation of artificial visual perception systems with important implications for the development of neuromorphic electronics.
Optoelectronic synaptic transistors with hybrid heterostructure channels have been extensively developed to construct artificial visual systems, inspired by the human visual system. However, ...optoelectronic transistors taking full advantages of superior optoelectronic synaptic behaviors, low-cost processes, low-power consumption, and environmental benignity remained a challenge. Herein, we report a fully printed, high-performance optoelectronic synaptic transistor based on hybrid heterostructures of heavy-metal-free InP/ZnSe core/shell quantum dots (QDs) and n-type SnO2 amorphous oxide semiconductors (AOSs). The elaborately designed heterojunction improves the separation efficiency of photoexcited charges, leading to high photoresponsivity and tunable synaptic weight changes. Under the coordinated modulation of electrical and optical modes, important biological synaptic behaviors, including excitatory postsynaptic current, short/long-term plasticity, and paired-pulse facilitation, were demonstrated with a low power consumption (∼5.6 pJ per event). The InP/ZnSe QD/SnO2 based artificial vision system illustrated a significantly improved accuracy of 91% in image recognition, compared to that of bare SnO2 based counterparts (58%). Combining the outstanding synaptic characteristics of both AOS materials and heterojunction structures, this work provides a printable, low-cost, and high-efficiency strategy to achieve advanced optoelectronic synapses for neuromorphic electronics and artificial intelligence.
The challenges of developing neuromorphic vision systems inspired by the human eye come not only from how to recreate the flexibility, sophistication, and adaptability of animal systems, but also how ...to do so with computational efficiency and elegance. Similar to biological systems, these neuromorphic circuits integrate functions of image sensing, memory and processing into the device, and process continuous analog brightness signal in real-time. High-integration, flexibility and ultra-sensitivity are essential for practical artificial vision systems that attempt to emulate biological processing. Here, we present a flexible optoelectronic sensor array of 1024 pixels using a combination of carbon nanotubes and perovskite quantum dots as active materials for an efficient neuromorphic vision system. The device has an extraordinary sensitivity to light with a responsivity of 5.1 × 10
A/W and a specific detectivity of 2 × 10
Jones, and demonstrates neuromorphic reinforcement learning by training the sensor array with a weak light pulse of 1 μW/cm
.
Human eyes possess exceptional image-sensing characteristics such as an extremely wide field of view, high resolution and sensitivity with low aberration
. Biomimetic eyes with such characteristics ...are highly desirable, especially in robotics and visual prostheses. However, the spherical shape and the retina of the biological eye pose an enormous fabrication challenge for biomimetic devices
. Here we present an electrochemical eye with a hemispherical retina made of a high-density array of nanowires mimicking the photoreceptors on a human retina. The device design has a high degree of structural similarity to a human eye with the potential to achieve high imaging resolution when individual nanowires are electrically addressed. Additionally, we demonstrate the image-sensing function of our biomimetic device by reconstructing the optical patterns projected onto the device. This work may lead to biomimetic photosensing devices that could find use in a wide spectrum of technological applications.
Falls are the leading cause of injury and death in elderly individuals. Unfortunately, fall detectors are typically based on wearable devices, and the elderly often forget to wear them. In addition, ...fall detectors based on artificial vision are not yet available on the market. In this paper, we present a new low-cost fall detector for smart homes based on artificial vision algorithms. Our detector combines several algorithms (background subtraction, Kalman filtering and optical flow) as input to a machine learning algorithm with high detection accuracy. Tests conducted on over 50 different fall videos have shown a detection ratio of greater than 96%.
Is it possible to explore an association between individual sperm kinematics evaluated in real time and spermatozoa selected by an embryologist for intracytoplasmic sperm injection (ICSI), with ...subsequent normal fertilization and blastocyst formation using a novel artificial vision-based software (SiD V1.0; IVF 2.0, UK)?
ICSI procedures were randomly video recorded and subjected to analysis using SiD V1.0, proprietary software developed by our group. In total, 383 individual spermatozoa were retrospectively analysed from a dataset of 78 ICSI-assisted reproductive technology cycles. SiD software computes the progressive motility parameters, straight-line velocity (VSL) and linearity of the curvilinear path (LIN), of each sperm trajectory, along with a quantitative value, head movement pattern (HMP), which is an indicator of the characteristics of the sperm head movement patterns. The mean VSL, LIN and HMP measurements for each set of spermatozoa were compared based on different outcome measures.
Statistically significant differences were found in VSL, LIN and HMP among those spermatozoa selected for injection (P < 0.001). Additionally, LIN and HMP were found to be significantly different between successful and unsuccessful fertilization (P = 0.038 and P = 0.029, respectively). Additionally, significantly higher SiD scores were found for those spermatozoa that achieved both successful fertilization (P = 0.004) and blastocyst formation (P = 0.013).
The possibility of carrying out real-time analyses of individual spermatozoa using an automatic tool such as SiD creates the opportunity to assist the embryologist in selecting the better spermatozoon for injection in an ICSI procedure.
PDF
