Spatially varying exposure (SVE) is a promising choice for high-dynamic-range (HDR) imaging (HDRI). The SVE-based HDRI, which is called single-shot HDRI, is an efficient solution to avoid ghosting ...artifacts. However, it is very challenging to restore a full-resolution HDR image from a real-world image with SVE because: a) only one-third of pixels with varying exposures are captured by camera in a Bayer pattern, b) some of the captured pixels are over- and under-exposed. For the former challenge, a spatially varying convolution (SVC) is designed to process the Bayer images carried with varying exposures. For the latter one, an exposure-guidance method is proposed against the interference from over- and under-exposed pixels. Finally, a joint demosaicing and HDRI deep learning framework is formalized to include the two novel components and to realize an end-to-end single-shot HDRI. Experiments indicate that the proposed end-to-end framework avoids the problem of cumulative errors and surpasses the related state-of-the-art methods. Related codes and datasets will be provided at https://github.com/yilun-xu/SVEHDRI/ .
Electroencephalography (EEG) signals have been proven to be one of the most predictive and reliable indicators for estimating driving fatigue state. However, how to make full use of EEG data for ...driving fatigue detection remains a challenge. Many existing methods include a time-consuming manual process or tedious parameter tunings for feature extraction, which is inconvenient to train and implement. On the other hand, most models ignore or manually determine EEG connectivity features between different channels, thus failing to thoroughly exploit the intrinsic interchannel relations for classification. In this article, we introduce a new attention-based multiscale convolutional neural network-dynamical graph convolutional network (AMCNN-DGCN) model, aiming to conquer these two issues in a unified end-to-end model. AMCNN-DGCN starts with attention-based multiscale temporal convolutions to automatically learn frequency filters to extract the salient pattern from raw EEG data. Subsequently, AMCNN-DGCN uses dynamical graph convolutional networks (DGCNs) to learn spatial filters, in which the adjacency matrix is adaptively determined in a data-driven way to exploit the intrinsic relationship between channels effectively. With the temporal-spatial structure, AMCNN-DGCN can capture highly discriminative features. To verify the effectiveness of AMCNN-DGCN, we conduct a simulated fatigue driving environment to collect EEG signals from 29 healthy subjects (male/female = 17/12 and age = 23.28±2.70 years) through a remote wireless cap with 24 channels. The results demonstrate that our proposed model outperforms six widely used competitive EEG models with high accuracy of 95.65%. Finally, the critical brain regions and connections for driving fatigue detection were investigated through the dynamically learned adjacency matrix.
Deep neural network (DNN) processing units, or DPUs, are one of the most energy-efficient platforms for DNN applications. However, designing new DPUs for every DNN model is very costly and time ...consuming. In this article, we propose an alternative approach: to specialize coarse-grained reconfigurable architectures (CGRAs), which are already quite capable of delivering high performance and high energy efficiency for compute-intensive kernels. We identify a small set of architectural features on a baseline CGRA to enable high-performance mapping of depthwise convolution (DWC) and pointwise convolution (PWC) kernels, which are the most important building block in recent light-weight DNN models. Our experimental results using MobileNets demonstrate that our proposed CGRA enhancement can deliver <inline-formula> <tex-math notation="LaTeX">8\sim 18\times </tex-math></inline-formula> improvement in area-delay product (ADP) depending on layer type, over a baseline CGRA with a state-of-the-art CGRA compiler. Moreover, our proposed CGRA architecture can also speed up 3-D convolution with similar efficiency as previous work, demonstrating the effectiveness of our architectural features beyond depthwise separable convolution (DSC) layers.
•A convolutional network module with irregular convolutions is provided for image denoising.•The module combines deformable convolution and side window filter to fit the geometric structure of image ...and restore image edges.•The module obtains more appropriate receptive fields to make the image edge sharper.•The module obtains 0.39 dB, 0.47 dB improvments over FFDNet on Kodak24 with noise level σ=25 and σ=50.
A noval neural networks with irregular convolution block is proposed for image denoising. In the field of image processing, convolutional neural networks have shown great advantages compared with traditional approaches, however, it is found that standard convolution does not work well on image edge, and it has some drawbacks when dealing with variable noised images. In this paper, we numerically illustrate that the irregular convolution, including deformable convolutional kernel and side window filtering technique, is beneficial for finding effective receptive field and improving image edge. Quantitative and qualitative experimental results are demonstrated, which outperforms classical convolution neural networks in denoising tasks.
Sparse approximate solutions to linear equations are classically obtained via L1 norm regularized least squares, but this method often underestimates the true solution. As an alternative to the L1 ...norm, this paper proposes a class of nonconvex penalty functions that maintain the convexity of the least squares cost function to be minimized, and avoids the systematic underestimation characteristic of L1 norm regularization. The proposed penalty function is a multivariate generalization of the minimax-concave penalty. It is defined in terms of a new multivariate generalization of the Huber function, which in turn is defined via infimal convolution. The proposed sparse-regularized least squares cost function can be minimized by proximal algorithms comprising simple computations.
The approaches for analyzing the polarimetric scattering matrix of polarimetric synthetic aperture radar (PolSAR) data have always been the focus of PolSAR image classification. Generally, the ...polarization coherent matrix and the covariance matrix obtained by the polarimetric scattering matrix are used as the main research object to extract features. In this paper, we focus on the original polarimetric scattering matrix and propose a polarimetric scattering coding way to deal with polarimetric scattering matrix and obtain a close complete feature. This encoding mode can also maintain polarimetric information of scattering matrix completely. At the same time, in view of this encoding way, we design a corresponding classification algorithm based on the convolution network to combine this feature. Based on the polarimetric scattering coding and convolution neural network, the polarimetric convolutional network is proposed to classify PolSAR images by making full use of polarimetric information. We perform the experiments on the PolSAR images acquired by AIRSAR and RADARSAT-2 to verify the proposed method. The experimental results demonstrate that the proposed method get better results and has huge potential for PolSAR data classification. Source code for polarimetric scattering coding is available at https://github.com/liuxuvip/Polarimetric-Scattering-Coding .
The extent of the area covered by polar sea ice is an important indicator of global climate change. Continuous monitoring of Arctic sea ice concentration (SIC) primarily relies on passive microwave ...images. However, passive microwave images have coarse spatial resolution, resulting in SIC production with significant blurring at the ice-water divides. In this article, a novel multi-image super-resolution (MISR) network called progressive multiscale deformable residual network (PMDRnet) is proposed to improve the spatial resolution of sea ice passive microwave images according to the characteristics of both passive microwave images and sea ice motions. To achieve image alignment with complex and large Arctic sea ice motions, we design a novel alignment module that includes a progressive alignment strategy and a multiscale deformable convolution alignment unit. In addition, the temporal attention mechanism is used to adaptively fuse the effective spatiotemporal information across image sequence. The sea ice-related loss function is designed to provide more detailed sea ice information of the network to improve super-resolution performance and further benefit finer Arctic SIC results. Experimental results demonstrate that PMDRnet significantly outperforms the current state-of-the-art MISR methods and can generate super-resolved SIC products with finer texture features and much sharper sea ice edges. The code and datasets of PMDRnet are available at https://doi.org/10.5061/dryad.k3j9kd590 .
Forecasting traffic flow and speed in the urban is important for many applications, ranging from the intelligent navigation of map applications to congestion relief of city management systems. ...Therefore, mining the complex spatio-temporal correlations in the traffic data to accurately predict traffic is essential for the community. However, previous studies that combined the graph convolution network or self-attention mechanism with deep time series models (e.g., the recurrent neural network) can only capture spatial dependencies in each time slot and temporal dependencies in each sensor, ignoring the spatial and temporal correlations across different time slots and sensors. Besides, the state-of-the-art Transformer architecture used in previous methods is insensitive to local spatio-temporal contexts, which is hard to suit with traffic forecasting. To solve the above two issues, we propose a novel deep learning model for traffic forecasting, named Locality-aware spatio-temporal joint Transformer (Lastjormer), which elaborately designs a spatio-temporal joint attention in the Transformer architecture to capture all dynamic dependencies in the traffic data. Specifically, our model utilizes the dot-product self-attention on sensors across many time slots to extract correlations among them and introduces the linear and convolution self-attention mechanism to reduce the computation needs and incorporate local spatio-temporal information. Experiments on three real-world traffic datasets, England, METR-LA, and PEMS-BAY, demonstrate that our Lastjormer achieves state-of-the-art performances on a variety of challenging traffic forecasting benchmarks.
Object detection is one of the essential tasks in computer vision, with most detection methods relying on a limited number of sizes for anchor boxes. However, the boundaries of particular composite ...objects, such as ports, highways, and golf courses, are ambiguous in remote sensing images, and therefore, it is challenging for the anchor-based method to accommodate the substantial size variation of the objects. In addition, the dense placement of anchor boxes imbalances the positive and negative samples, which affects the end-to-end architecture of deep learning methods. Hence, this paper proposes a single-stage object detection model named Xnet to address this issue. The proposed method designs a deformable convolution backbone network used in the feature extraction stage. Compared to the standard convolution, it adds learnable parameters for dynamically analyzing the boundary and offset of the receptive field, rendering the model more adaptable to size variations within the same class. Moreover, this paper presents a novel anchor-free detector that classifies objects in feature images point-by-point, without relying on anchor boxes. Several experiments on the large remote sensing dataset DIOR challenging Xnet against other popular methods demonstrate that our method attains the best performance, surpassing by 4.7% on the mAP (mean average precision) metric.
The prediction of the remaining useful life (RUL) of wind turbine gearbox bearings is critical to avoid catastrophic accidents and minimize downtime. Temporal convolutional network (TCN), as a ...potential method of RUL prediction, utilizes dilated causal convolution to extract historic information in the time series, by which it can avoid the disadvantage of long-term dependence faced by classical recurrent neural networks (RNNs). However, a large amount of local information is lost after dilated causal convolution, restricting further improvement of accuracy in RUL prediction or even making TCN invalid when the time series data are not sufficient. To address this issue, an improved TCN denoted as self-calibration temporal convolutional network (SCTCN) is proposed for RUL prediction of wind turbine gearbox bearings, in which the dilated causal convolution of TCN is inherited to extract the long-term historic information, and the self-calibration module is used to focus on the local information in the time series. As a result, SCTCN can learn more complete historic information to improve the accuracy of RUL prediction. Bearing RUL prediction experiments on both test bench and wind turbine gearbox are performed to verify the effectiveness of the proposed method, and the experimental results show that SCTCN has higher prediction accuracy compared with other state-of-the-art methods.
PDF
