This paper provides a survey of the latest developments in visual signal coding and processing with generative models. Specifically, our focus is on presenting the advancement of generative models ...and their influence on research in the domain of visual signal coding and processing. This survey study begins with a brief introduction of well-established generative models, including the Variational Autoencoder (VAE) models, Generative Adversarial Network (GAN) models, Autoregressive (AR) models, Normalizing Flows and Diffusion models. The subsequent section of the paper explores the advancements in visual signal coding based on generative models, as well as the ongoing international standardization activities. In the realm of visual signal processing, our focus lies on the application and development of various generative models in the research of visual signal restoration. We also present the latest developments in generative visual signal synthesis and editing, along with visual signal quality assessment using generative models and quality assessment for generative models. The practical implementation of these studies is closely linked to the investigation of fast optimization. This paper additionally presents the latest advancements in fast optimization on visual signal coding and processing with generative models. We hope to advance this field by providing researchers and practitioners a comprehensive literature review on the topic of visual signal coding and processing with generative models.
•A novel eutecto-/hydro-gel (EHG) is proposed for flexible wearable strain sensing.•Dynamic balance between hydration and dehydration endows EHG with super stability.•EHG shows excellent ionic ...conductivity, mechanical strength and transparency.•EHG containing photonic crystals (EHG-PC) enables visual and electronic sensing.•Dual-mode visual and electronic sensing work independently and complementarily.
In recent years, hydrogels (HGs) have been extensively utilized as potential substrates for flexible wearable strain sensors, due to their excellent biocompatibility, stretchability and ease to incorporate various ionic /electronic conductive materials. However, water evaporation (i.e., dehydration) and low temperature intolerance severely restrain their sensing performance and lifetime. Eutectogels (EGs) as new emerging substrates, exhibit impressive low temperature tolerance and sound ionic conductivity, but they are prone to absorb water (i.e., hydration) in air and thus instable in morphology and ionic conductivity. Herein, we propose a novel concept of eutecto-/hydro-gel (EHG) with super stability, high ionic conductivity and transparency. As a proof-of-concept, we demonstrate the fabrication and strain sensing performance of a new EHG containing N-acryloyl glycinamide (NAGA), choline chloride (ChCl), glycerol (Gly) and water. EHG can maintain the dynamic balance between hydration and dehydration, and thus show much better long-term sensing performance. More importantly, we further show that flexible wearable visual and electronic strain sensor can be constructed by integrating photonic crystals into EHG. Due to the remarkable difference in sensing principles, the two sensing modes in this strain sensor can not only work properly without mutual interference, but also supplement each other to improve the sensing accuracy. The EHG material system and the fabrication strategy for dual-mode flexible wearable strain sensor provide a new solution for the development of high-performance flexible wearable devices.
The evolution of sexual communication is critically important in the diversity of arthropods, which are declining at a fast pace worldwide. Their environments are rapidly changing, with increasing ...chemical, acoustic, and light pollution. To predict how arthropod species will respond to changing climates, habitats, and communities, we need to understand how sexual communication systems can evolve. In the past decades, intraspecific variation in sexual signals and responses across different modalities has been identified, but never in a comparative way. In this review, we identify and compare the level and extent of intraspecific variation in sexual signals and responses across three different modalities, chemical, acoustic, and visual, focusing mostly on insects. By comparing causes and possible consequences of intraspecific variation in sexual communication among these modalities, we identify shared and unique patterns, as well as knowledge needed to predict the evolution of sexual communication systems in arthropods in a changing world.
Online penetration monitoring for complex butt welding is challenging due to steel plate's groove instability and welding heat deformation. In this paper, automatic cold metal transfer (CMT) welding ...is used to join two complex bevelled austenitic stainless steel with SS304 as the base metal. This work reports a hybrid approach combining deep learning, computer vision, and sound signal processing to monitor groove welding penetration under full penetration in real time. Sequence signals such as video and sound can complimentarily characterize the melt pool state. In this paper, the proposed Multimodal continuous signals Characteristic Reinforcement Network (MCRNet) utilizes 3D convolution and multiscale convolution with channel attention to considerably improve the performance of lightweight networks. At the same time, a new fusion method with similarity loss is proposed to cope with the input of visual and acoustic signals. That improves the effect by at least 18 % compared with the single-modal signal input. The experimental results show that the Mean Square Error (MSE) of MCRNet improved the performance by 44 % compared with the mainstream deep learning framework. Meanwhile, the inference speed under multimodal input reaches 57 frames per second (FPS). MCRNet finally achieves online penetration accurate prediction of the melt pool.
The worldwide flourishing of the Internet of Things (IoT) in the past decade has enabled numerous new applications through the internetworking of a wide variety of devices and sensors. More recently, ...visual sensors have seen their considerable booming in IoT systems because they are capable of providing richer and more versatile information. Internetworking of large-scale visual sensors has been named Internet of Video Things (IoVT). IoVT has its own unique characteristics in terms of sensing, transmission, storage, and analysis, which are fundamentally different from the conventional IoT. These new characteristics of IoVT are expected to impose significant challenges to existing technical infrastructures. In this article, an overview of recent advances in various fronts of IoVT will be introduced and a broad range of technological and system challenges will be addressed. Several emerging IoVT applications will be discussed briefly to illustrate the potentials of IoVT in a broad range of practical scenarios.
Multifarious sources of selection shape visual signals and can produce phenotypic divergence. Theory predicts that variance in warning signals should be minimal due to purifying selection, yet ...polymorphism is abundant. While in some instances divergent signals can evolve into discrete morphs, continuously variable phenotypes are also encountered in natural populations. Notwithstanding, we currently have an incomplete understanding of how combinations of selection shape fitness landscapes, particularly those which produce polymorphism. We modelled how combinations of natural and sexual selection act on aposematic traits within a single population to gain insights into what combinations of selection favours the evolution and maintenance of phenotypic variation. With a rich foundation of studies on selection and phenotypic divergence, we reference the poison frog genus
to model signal evolution. Multifarious selection on aposematic traits created the topology of our model's fitness landscape by approximating different scenarios found in natural populations. Combined, the model produced all types of phenotypic variation found in frog populations, namely monomorphism, continuous variation and discrete polymorphism. Our results afford advances into how multifarious selection shapes phenotypic divergence, which, along with additional modelling enhancements, will allow us to further our understanding of visual signal evolution.
The photothermal biosensing principle is of increasing interest for point-of-care detection, but has rarely been applied in portable analytical devices in a lab-on-a-chip format. Herein, a ...photothermally responsive poly (methyl methacrylate) (PMMA)/paper hybrid disk (PT-Disk) was developed as a novel photothermal immunoassay device with the integration of a clip-magazine-assembled photothermal biosensing strategy. The PT-Disk consisted of a dissociative thermoresponsive hydrogel-loaded clip unit where the sandwich-type immunoreaction with an iron oxide-to-Prussian blue nanoparticle (PB NP) conversion took place and a magazine bearer for the rotational clip assembly and visual signal outputs. Upon laser irradiation of the clip-magazine-assembled PT-Disk, on-chip photothermal effect of PB NPs triggered both dose-dependent temperature elevation and the subsequent release of dye solutions from the central clip unit to surrounding magazine-bearing paper channels as the result of phase transition of the hydrogels, realizing multiplexed thermal image- and distance-based visual quantitative signal outputs in combination with the preliminary colorimetric readout on the PT-Disk. Using the multiplexed tri-mode signal outputs, the PT-Disk can quantify prostate specific antigen with limits of detection of 1.4–2.8 ng mL−1. This is the first attempt to apply the photothermal biosensing principle in portable PMMA/paper-based analytical devices, which offers not only versatile on-chip visual quantitative signal outputs, but also the implementation of the photothermal biosensing principle in a lab-on-a-chip format.
•Application of photothermal biosensing principle in PMMA/paper-based devices.•Development of a novel photothermal immunoassay disk.•Use of a clip-magazine-assembled photothermal biosensing strategy.•Realization of on-chip multiplexed tri-mode visual quantitative signal outputs.•Tri-mode visual quantification of prostate specific antigen (PSA).
Emotion recognition will be challenged for individuals when visual signals are degraded in real-life scenarios. Recently, researchers have conducted many studies on the distinct neural activity ...between clear and degraded audiovisual stimuli. These findings addressed the “how” question, but the precise stage of the distinct activity that occurred remains unknown. Therefore, it is crucial to use event-related potential (ERP) to explore the “when” question, just the time course of the neural activity of degraded audiovisual stimuli. In the present research, we established two conditions: clear auditory + degraded visual (AcVd) and clear auditory + clear visual (AcVc) multisensory conditions. We enlisted 31 participants to evaluate the emotional valence of audiovisual stimuli. The resulting data were analyzed using ERP in time domains and Microstate analysis. Current results suggest that degraded vision impairs the early-stage processing of audiovisual stimuli, with the superior parietal lobule (SPL) regulating audiovisual processing in a top-down fashion. Additionally, our findings indicate that negative and positive stimuli elicit greater EPN compared to neutral stimuli, pointing towards a subjective motivation-related attentional regulation. To sum up, in the early stage of emotional audiovisual processing, the degraded visual signal affected the perception of the physical attributes of audiovisual stimuli and had a further influence on emotion extraction processing, leading to the different regulation of top-down attention resources in the later stage.
•The temporal processing of audiovisual emotional perception with degraded vision.•Degraded vision impairs the early-stage processing of audiovisual stimuli in N1 and P1.•Clear audiovisual stimuli were regulated by superior parietal lobule in 286–386ms.•Activation of audiovisual stimuli was related to visual clarity and emotional valence.
