In this work, we propose ReStoCNet, a residual stochastic multilayer convolutional Spiking Neural Network (SNN) composed of binary kernels, to reduce the synaptic memory footprint and enhance the ...computational efficiency of SNNs for complex pattern recognition tasks. ReStoCNet consists of an input layer followed by stacked convolutional layers for hierarchical input feature extraction, pooling layers for dimensionality reduction, and fully-connected layer for inference. In addition, we introduce residual connections between the stacked convolutional layers to improve the hierarchical feature learning capability of deep SNNs. We propose Spike Timing Dependent Plasticity (STDP) based probabilistic learning algorithm, referred to as Hybrid-STDP (HB-STDP), incorporating Hebbian and anti-Hebbian learning mechanisms, to train the binary kernels forming ReStoCNet in a layer-wise unsupervised manner. We demonstrate the efficacy of ReStoCNet and the presented HB-STDP based unsupervised training methodology on the MNIST and CIFAR-10 datasets. We show that residual connections enable the deeper convolutional layers to self-learn useful high-level input features and mitigate the accuracy loss observed in deep SNNs devoid of residual connections. The proposed ReStoCNet offers >20 × kernel memory compression compared to full-precision (32-bit) SNN while yielding high enough classification accuracy on the chosen pattern recognition tasks.
Recent advancements in the development of memristive devices has opened new opportunities for hardware implementation of new computing models. Researchers have shown the suitability of memristive ...devices for swarm intelligence algorithms to solve a maze in hardware. In this paper, we utilize swarm intelligence of memristive networks to perform image edge detection. First, we propose a hardware-friendly algorithm for image edge detection based on ant colony optimization. Second, we implement the image edge detection algorithm using memristive networks. Furthermore, we explain the impact of various parameters of the memristors on the efficacy of the implementation. Our results show 28% improvement in the energy compared to a low power CMOS hardware implementation based on stochastic circuits. Furthermore, our design occupies up to <inline-formula> <tex-math notation="LaTeX">5\boldsymbol \times </tex-math></inline-formula> less area.
Deep spiking neural networks (SNNs) have emerged as one of the popular architectures in complex pattern recognition and classification tasks that can be enabled by low-power neuromorphic hardware. ...However, hardware implementation of such algorithms using conventional CMOS devices is area expensive and energy inefficient. This is owing to the fundamental mismatch between the underlying neuromophic computations and the CMOS transistors along with energy consumption involved in synaptic memory-access operations. Hence, there is a need for novel "neuro-mimetic" devices offering a direct mapping to synaptic and neuronal functionalities together with the possibility of providing in situ synaptic storage. Magnetic skyrmions have recently been proposed as a promising alternative for next-generation information carrier due to remarkably high stability, ultra-low depinning current density, and extremely compact size. In this paper, the design of skyrmion-based devices to emulate biological synapses and neurons is explored, and skyrmionic synapse-based crossbar architectures driving skyrmionic neurons are proposed. We perform a systematic device-circuit-architecture co-design for digit recognition with the MNIST handwritten digits dataset to evaluate the feasibility of our proposal. The device-to-system simulations indicate that the proposed skyrmion-based devices in deep SNNs can potentially achieve two orders of magnitude improvement in energy consumption over an optimized CMOS implementation at a 45 nm technology node.
Adversarial examples are perturbed inputs that are designed (from a deep learning network's (DLN) parameter gradients) to mislead the DLN during test time. Intuitively, constraining the ...dimensionality of inputs or parameters of a network reduces the "space" in which adversarial examples exist. Guided by this intuition, we demonstrate that discretization greatly improves the robustness of the DLNs against adversarial attacks. Specifically, discretizing the input space (or allowed pixel levels from 256 values or 8<inline-formula> <tex-math notation="LaTeX">bit </tex-math></inline-formula> to 4 values or 2<inline-formula> <tex-math notation="LaTeX">bit </tex-math></inline-formula>) extensively improves the adversarial robustness of the DLNs for a substantial range of perturbations for minimal loss in test accuracy. Furthermore, we find that binary neural networks (BNNs) and related variants are intrinsically more robust than their full precision counterparts in adversarial scenarios. Combining input discretization with the BNNs furthers the robustness, even waiving the need for adversarial training for the certain magnitude of perturbation values. We evaluate the effect of discretization on MNIST, CIFAR10, CIFAR100, and ImageNet datasets. Across all datasets, we observe maximal adversarial resistance with 2<inline-formula> <tex-math notation="LaTeX">bit </tex-math></inline-formula> input discretization that incurs an adversarial accuracy loss of just ~1% - 2% as compared to clean test accuracy against single-step attacks. We also show standalone discretization remains vulnerable to stronger multi-step attack scenarios necessitating the use of adversarial training with discretization as an improved defense strategy.
Brain-inspired computing architectures attempt to mimic the computations performed in the neurons and the synapses in the human brain in order to achieve its efficiency in learning and cognitive ...tasks. In this work, we demonstrate the mapping of the probabilistic spiking nature of pyramidal neurons in the cortex to the stochastic switching behavior of a Magnetic Tunnel Junction in presence of thermal noise. We present results to illustrate the efficiency of neuromorphic systems based on such probabilistic neurons for pattern recognition tasks in presence of lateral inhibition and homeostasis. Such stochastic MTJ neurons can also potentially provide a direct mapping to the probabilistic computing elements in Belief Networks for performing regenerative tasks.
"Spintronics refers to the understanding of the physics of electron spin-related phenomena. While most of the significant advancements in this field has been driven primarily by memory, recent ...research has demonstrated that various facets of the underlying physics of spin transport and manipulation can directly mimic the functionalities of the computational primitives in neuromorphic computation, i.e., the neurons and synapses. Given the potential of these spintronic devices to implement bio-mimetic computations at very low terminal voltages, several spin-device structures have been proposed as the core building blocks of neuromorphic circuits and systems to implement brain-inspired computing. Such an approach is expected to play a key role in circumventing the problems of ever-increasing power dissipation and hardware requirements for implementing neuro-inspired algorithms in conventional digital CMOS technology. Perspectives on spin-enabled neuromorphic computing, its status, and challenges and future prospects are outlined in this review article.
Deep learning models hold state of the art performance in many fields, yet their design is still based on heuristics or grid search methods that often result in overparametrized networks. This work ...proposes a method to analyze a trained network and deduce an optimized, compressed architecture that preserves accuracy while keeping computational costs tractable. Model compression is an active field of research that targets the problem of realizing deep learning models in hardware. However, most pruning methodologies tend to be experimental, requiring large compute and time intensive iterations of retraining the entire network. We introduce structure into model design by proposing a single shot analysis of a trained network that serves as a first order, low effort approach to dimensionality reduction, by using PCA (Principal Component Analysis). The proposed method simultaneously analyzes the activations of each layer and considers the dimensionality of the space described by the filters generating these activations. It optimizes the architecture in terms of number of layers, and number of filters per layer without any iterative retraining procedures, making it a viable, low effort technique to design efficient networks. We demonstrate the proposed methodology on AlexNet and VGG style networks on the CIFAR-10, CIFAR-100 and ImageNet datasets, and successfully achieve an optimized architecture with a reduction of up to 3.8X and 9X in the number of operations and parameters respectively, while trading off less than 1% accuracy. We also apply the method to MobileNet, and achieve 1.7X and 3.9X reduction in the number of operations and parameters respectively, while improving accuracy by almost one percentage point.
Probabilistic inference from real-time input data is becoming increasingly popular and may be one of the potential pathways at enabling cognitive intelligence. As a matter of fact, preliminary ...research has revealed that stochastic functionalities also underlie the spiking behavior of neurons in cortical microcircuits of the human brain. In tune with such observations, neuromorphic and other unconventional computing platforms have recently started adopting the usage of computational units that generate outputs probabilistically, depending on the magnitude of the input stimulus. In this work, we experimentally demonstrate a spintronic device that offers a direct mapping to the functionality of such a controllable stochastic switching element. We show that the probabilistic switching of Ta/CoFeB/MgO heterostructures in presence of spin-orbit torque and thermal noise can be harnessed to enable probabilistic inference in a plethora of unconventional computing scenarios. This work can potentially pave the way for hardware that directly mimics the computational units of Bayesian inference.
Spiking neural networks (SNNs) have emerged as a promising brain inspired neuromorphic-computing paradigm for cognitive system design due to their inherent event-driven processing capability. The ...fully connected (FC) shallow SNNs typically used for pattern recognition require large number of trainable parameters to achieve competitive classification accuracy. In this paper, we propose a deep spiking convolutional neural network (SpiCNN) composed of a hierarchy of stacked convolutional layers followed by a spatial-pooling layer and a final FC layer. The network is populated with biologically plausible leaky-integrate-and-fire (LIF) neurons interconnected by shared synaptic weight kernels. We train convolutional kernels layer-by-layer in an unsupervised manner using spike-timing-dependent plasticity (STDP) that enables them to self-learn characteristic features making up the input patterns. In order to further improve the feature learning efficiency, we propose using smaller <inline-formula> <tex-math notation="LaTeX">3\boldsymbol \times 3 </tex-math></inline-formula> kernels trained using STDP-based synaptic weight updates performed over a mini-batch of input patterns. Our deep SpiCNN, consisting of two convolutional layers trained using the unsupervised convolutional STDP learning methodology, achieved classification accuracies of 91.1% and 97.6%, respectively, for inferring handwritten digits from the MNIST data set and a subset of natural images from the Caltech data set.
PDF
