Driver attention estimation is one of the key technologies for intelligent vehicles. The existing related methods only focus on the scene image or the driver's gaze or head pose. The purpose of this ...article is to propose a more reasonable and feasible method based on a dual-view scene with calibration-free gaze direction. According to human visual mechanisms, the low-level features, static visual saliency map, and dynamic optical flow information are extracted as input feature maps, which combine the high-level semantic descriptions and a gaze probability map transformed from the gaze direction. A multiresolution neural network is proposed to handle the calibration-free features. The proposed method is verified on a virtual reality experimental platform that collected more than 550 000 samples and obtained a more accurate ground truth. The experiments show that the proposed method is feasible and better than the state-of-the-art methods based on multiple widely used metrics. This study also provides a discussion of the effects of different landscapes, times, and weather conditions on the performance.
Reinforcementlearning holds the promise of allowing autonomous vehicles to learn complex decision making behaviors through interacting with other traffic participants. However, many real-world ...driving tasks involve unpredictable perception errors or measurement noises which may mislead an autonomous vehicle into making unsafe decisions, even cause catastrophic failures. In light of these risks, to ensure safety under perception uncertainty, autonomous vehicles are required to be able to cope with the worst case observation perturbations. Therefore, this paper proposes a novel observation adversarial reinforcement learning approach for robust lane change decision making of autonomous vehicles. A constrained observation-robust Markov decision process is presented to model lane change decision making behaviors of autonomous vehicles under policy constraints and observation uncertainties. Meanwhile, a black-box attack technique based on Bayesian optimization is implemented to approximate the optimal adversarial observation perturbations efficiently. Furthermore, a constrained observation-robust actor-critic algorithm is advanced to optimize autonomous driving lane change policies while keeping the variations of the policies attacked by the optimal adversarial observation perturbations within bounds. Finally, the robust lane change decision making approach is evaluated in three stochastic mixed traffic flows based on different densities. The results demonstrate that the proposed method can not only enhance the performance of an autonomous vehicle but also improve the robustness of lane change policies against adversarial observation perturbations.
The energy economy of fuel cell electric vehicles (FCEVs) plays a crucial role in determining their practicality, making the optimization of energy management strategies (EMS) essential. Predictive ...EMS (PEMS) based on future vehicle speed prediction offers great potential for enhancing EMS performance. However, current PEMS prediction models rely on historical speed data or static traffic information, overlooking the impact of real-time traffic conditions. In this article, we introduce a Transformer-based PEMS (TPEMS) that incorporates real-time predicted surrounding traffic information to improve FCEV operational economy. To better predict vehicle speed by accounting for the complex interactions between the controlled vehicle and surrounding vehicles, we developed a Transformer network-based predictor, which considers the speed and relative distance of six vehicles surrounding the controlled vehicle, generating speed predictions for the next 10 s. We then employ the deep reinforcement learning (DRL) method as a downstream optimizer, creating a fully data-driven PEMS. For training the TPEMS, we developed a dataset derived from the NGSIM dataset, consisting of numerous driving profile segments that include temporal-sequential characteristics of the controlled vehicle and surrounding traffic. Furthermore, we utilize the SUMO simulator to generate a traffic information-enabled driving profile for performance evaluation. Experimental results reveal our Transformer-based predictor outperforms existing predictors, i.e., recurrent neural networks (RNN), in processing traffic information and achieving improved predictions. The TPEMS enhances the economic efficiency of FCEVs by 4.6% relative to the current state-of-the-art long short-term memory (LSTM)-based PEMS.
Making safe and human-like decisions is an essential capability of autonomous driving systems, and learning-based behavior planning presents a promising pathway toward achieving this objective. ...Distinguished from existing learning-based methods that directly output decisions, this work introduces a predictive behavior planning framework that learns to predict and evaluate from human driving data. This framework consists of three components: a behavior generation module that produces a diverse set of candidate behaviors in the form of trajectory proposals, a conditional motion prediction network that predicts future trajectories of other agents based on each proposal, and a scoring module that evaluates the candidate plans using maximum entropy inverse reinforcement learning (IRL). We validate the proposed framework on a large-scale real-world urban driving dataset through comprehensive experiments. The results show that the conditional prediction model can predict distinct and reasonable future trajectories given different trajectory proposals and the IRL-based scoring module can select plans that are close to human driving. The proposed framework outperforms other baseline methods in terms of similarity to human driving trajectories. Additionally, we find that the conditional prediction model improves both prediction and planning performance compared to the non-conditional model. Lastly, we note that the learning of the scoring module is crucial for aligning the evaluations with human drivers.
This study aims to improve the performance and generalization capability of end-to-end autonomous driving with scene understanding leveraging deep learning and multimodal sensor fusion techniques. ...The designed end-to-end deep neural network takes as input the visual image and associated depth information in an early fusion level and outputs the pixel-wise semantic segmentation as scene understanding and vehicle control commands concurrently. The end-to-end deep learning-based autonomous driving model is tested in high-fidelity simulated urban driving conditions and compared with the benchmark of CoRL2017 and NoCrash. The testing results show that the proposed approach is of better performance and generalization ability, achieving a 100% success rate in static navigation tasks in both training and unobserved situations, as well as better success rates in other tasks than the prior models. A further ablation study shows that the model with the removal of multimodal sensor fusion or scene understanding pales in the new environment because of the false perception. The results verify that the performance of our model is improved by the synergy of multimodal sensor fusion with scene understanding subtask, demonstrating the feasibility and effectiveness of the developed deep neural network with multimodal sensor fusion.
A light field camera can capture light information from various directions within a scene, allowing for the reconstruction of the scene. The light field image inherently contains the depth ...information of the scene, and depth estimations of light field images have become a popular research topic. This paper proposes a depth estimation network of light field images with occlusion awareness. Since light field images contain many views from different viewpoints, identifying the combinations that contribute the most to the depth estimation of the center view is critical to improving the depth estimation accuracy. Current methods typically rely on a fixed set of views, such as vertical, horizontal, and diagonal, which may not be optimal for all scenes. To address this limitation, we propose a novel approach that considers all available views during depth estimation while leveraging an attention mechanism to assign weights to each view dynamically. By inputting all views into the network and employing the attention mechanism, we enable the model to adaptively determine the most informative views for each scene, thus achieving more accurate depth estimation. Furthermore, we introduce a multi-scale feature fusion strategy that amalgamates contextual information and expands the receptive field to enhance the network’s performance in handling challenging scenarios, such as textureless and occluded regions.
The fluctuation and intermission of large‐scale wind power integration is a serious threat to the stability and security of the power system. Accurate prediction of wind power is of great ...significance to the safety of wind power grid connection. This study proposes a novel spatio‐temporal correlation model (STCM) for ultra‐short‐term wind power prediction based on convolutional neural networks‐long short‐term memory (CNN‐LSTM). The original meteorological factors at multi‐historical time points of different sites throughout the target wind farm can be reconstructed into the input window of the model, and thus a new data reconstruction method is represented. CNN is used to extract the spatial correlation feature vectors of meteorological factors of different sites and the temporal correlation vectors of the meteorological features in ultra‐short term, which are reconstructed in time series and used as the input data of LSTM. Then, LSTM extracts the temporal feature relationship between the historical time points for multi‐step wind power forecasting. The STCM based on CNN‐LSTM proposed in this study is suitable for wind farms that can collect meteorological factors at different locations. Taking the measured meteorological factors and wind power dataset of a wind farm in China as an example, four evaluation metrics of the CNN‐LSTM model, CNN and LSTM individually used for multi‐step wind power prediction, are obtained. The results show that the proposed STCM based on CNN‐LSTM has better spatial and temporal characteristics extraction ability than the traditional structure model and can forecast the power of wind farm more accurately.
To further advance the performance and safety of autonomous mobile robots (AMRs), an integrated chassis control framework is proposed. In the longitudinal motion control module, a velocity-tracking ...controller was designed with the integrated feedforward and feedback control algorithm. Besides, the nonlinear model predictive control (NMPC) method was applied to the four-wheel steering (4WS) path-tracking controller design. To deal with the failure of key actuators, an active fault-tolerant control (AFTC) algorithm was designed by reallocating the driving or braking torques of the remaining normal actuators, and the weighted least squares (WLS) method was used for torque reallocation. The simulation results show that AMRs can advance driving stability and braking safety in the braking failure condition with the utilization of AFTC and recapture the braking energy during decelerations.
To improve the safety and efficiency of the intelligent transportation system, particularly in complex urban scenarios, in this paper a game theoretic decision-making framework is designed for ...connected automated vehicles (CAVs) at unsignalized roundabouts considering their personalized driving behaviours. Within the decision-making framework, a motion prediction module is designed and optimized using model predictive control (MPC) to enhance the effectiveness and accuracy of the decision-making algorithm. Besides, the payoff function of decision making is defined with the consideration of vehicle safety, ride comfort and travel efficiency. Additionally, the constraints of the decision-making problem are constructed. Based on the established decision-making model, Stackelberg game and grand coalition game approaches are adopted to address the decision making of CAVs at an unsignalized roundabout. Three testing cases considering personalized driving behaviours are carried out to verify the performance of the developed decision-making algorithms. The testing results show that the proposed game theoretic decision-making framework is able to make safe and reasonable decisions for CAVs in the complex urban scenarios, validating its feasibility and effectiveness. Stackelberg game approach shows its advantage in guaranteeing personalized driving objectives of individuals, while the grand coalition game approach is advantageous regarding the efficiency improvement of the transportation system.
PDF
