This essay is part of a research project that seeks to analyze, based on Intermediality, the boundaries between sound media products such as audiobooks and podcasts. One of the conclusions of this ...larger project is in the way fictional sound media use cinema features to create what we can understand as immersion in the narrative diegesis. In this essay, we show how cinema, as a type of multimodal media, becomes fundamental to the construction of features for fiction audiobooks, such as tracks and soundeffects, as well as vocal performance, a feature that, along with the rhythm, is the oldest in literary history. Using Lars Elleström’s model in The modalities of media II (2020) and analyzing different audiobooks, we establish some categories. For soundtracks: iconic, based on the similarity with non-musical elements (Mickey Mousing); symbolic, the conventions in rhythms and other musical elements; and indexical, which refers to the manifestation of time, geographical location, or specific social group. Sound effects are distinguished into depictional sounds (to illustrate actions), deictional sounds (to create ambiance), and descriptional sounds (by convention). The methodology is comparative, seeking similarities and differences between these sound media products. Our intention is to expand Elleström’s model so that it can support the understanding of these new—and not-so-new—qualified media types. In addition to Elleström’s theories, we also bring those of Agnes Petho, Jorgen Bruhn and André Bazin.
Emulating the human ability to solve the cocktail party problem, i.e., focus on a source of interest in a complex acoustic scene, is a long standing goal of audio source separation research. Much of ...this research investigates separating speech from noise, speech from speech, musical instruments from each other, or sound events from each other. In this paper, we focus on the cocktail fork problem, which takes a three-pronged approach to source separation by separating an audio mixture such as a movie soundtrack or podcast into the three broad categories of speech, music, and sound effects (SFX - understood to include ambient noise and natural sound events). We benchmark the performance of several deep learning-based source separation models on this task and evaluate them with respect to simple objective measures such as signal-to-distortion ratio (SDR) as well as objective metrics that better correlate with human perception. Furthermore, we thoroughly evaluate how source separation can influence downstream transcription tasks. First, we investigate the task of activity detection on the three sources as a way to both further improve source separation and perform transcription. We formulate the transcription tasks as speech recognition for speech and audio tagging for music and SFX. We observe that, while the use of source separation estimates improves transcription performance in comparison to the original soundtrack, performance is still sub-optimal due to artifacts introduced by the separation process. Therefore, we thoroughly investigate how remixing of the three separated source stems at various relative levels can reduce artifacts and consequently improve the transcription performance. We find that remixing music and SFX interferences at a target SNR of 17.5 dB reduces speech recognition word error rate, and similar impact from remixing is observed for tagging music and SFX content.
The paper presents the most currently used driving simulators. Based on the analysis of the operating conditions of the rail vehicles the criteria to be met by this type of vehicle simulator have ...been characterized. In addition, the paper defines the animation requirements. Reference was made to both graphics and video presentation techniques. The paper also discusses the design requirements for the installation of a cab simulator and extras such as the sound effects. The most important requirements have been defined for programming of the simulated routes, weather conditions, collisions and other parameters that can significantly improve the quality of a training simulator. The paper discusses the function of control of the course of the exercise trainer and the requirements of his job. Finally the paper discusses the issues of the negative impact of driving simulators on the human body and methods of prevention thereof.
Sound design involves creatively selecting, recording, and editing sound effects for various media like cinema, video games, and virtual/augmented reality. One of the most time-consuming steps when ...designing sound is synchronizing audio with video. In some cases, environmental recordings from video shoots are available, which can aid in the process. However, in video games and animations, no reference audio exists, requiring manual annotation of event timings from the video. We propose a system to extract repetitive actions onsets from a video, which are then used - in conjunction with audio or textual embeddings - to condition a diffusion model trained to generate a new synchronized sound effects audio track. In this way, we leave complete creative control to the sound designer while removing the burden of synchronization with video. Furthermore, editing the onset track or changing the conditioning embedding requires much less effort than editing the audio track itself, simplifying the sonification process. We provide sound examples, source code, and pretrained models to faciliate reproducibility
Despite the significant positive characteristics game-based learning offers to pupil learning and assessment, preserving pupils' interest and keeping them engaged in an educational game is still a ...challenge. To this end, the study and implementation of motivation mechanisms in educational games are considered crucial. Typical examples of motivators in electronic games include points (coins), avatar icons, visualization of achievement levels, NPCs (non-player characters) giving helpful information to users, children-friendly graphics and sound effects, comparison with classmates, and leaderboards. In this paper, we conduct a preliminary study of the effectiveness of these GBL motivators in MG, an educational game for practicing and assessing multiplication skills. The study combined eye-tracking with a short, semi-structured interview session with the four elementary school students that took part in the experiment. Eye-tracking provides detailed monitoring and visualization of gaze behavior in the form of fixation (point and duration of visual focus) and saccade sequences. Given that the way users allocate their visual focus is spontaneous, the data collected and analyzed by eye-tracking are unbiased and give a new spectrum of insight into how users perceive a visual stimulus. In this study, we investigate how users visually respond to the implemented motivators and their visual behavior when deciding between two or more available answers and when given feedback after a wrong answer. The paper discusses useful eye-tracking metrics, provides adequate visualizations of the main findings, and concludes with the ways eye-tracking can help education scientists and practitioners gain a better understanding of the behavior of users of GBL applications and the motivation mechanisms they support.
Footsteps are among the most ubiquitous sound effects in multimedia applications. There is substantial research into understanding the acoustic features and developing synthesis models for footstep ...sound effects. In this paper, we present a first attempt at adopting neural synthesis for this task. We implemented two GAN-based architectures and compared the results with real recordings as well as six traditional sound synthesis methods. Our architectures reached realism scores as high as recorded samples, showing encouraging results for the task at hand.
One-to-Many Conversion for Percussive Samples Fagerstrom, Jon; Schlecht, Sebastian J.; Valimaki, Vesa
2021 24th International Conference on Digital Audio Effects (DAFx),
2021-Sept.-8
Conference Proceeding
Odprti dostop
A filtering algorithm for generating subtle random variations in sampled sounds is proposed. Using only one recording for impact sound effects or drum machine sounds results in unrealistic ...repetitiveness during consecutive playback. This paper studies spectral variations in repeated knocking sounds and in three drum sounds: a hihat, a snare, and a tomtom. The proposed method uses a short pseudo-random velvet-noise filter and a low-shelf filter to produce timbral variations targeted at appropriate spectral regions, yielding potentially an endless number of new realistic versions of a single percussive sampled sound. The realism of the resulting processed sounds is studied in a listening test. The results show that the sound quality obtained with the proposed algorithm is at least as good as that of a previous method while using 77% fewer computational operations. The algorithm is widely applicable to computer-generated music and game audio.
This paper compares the sound design in the 2004 Chinese version of Letter from an Unknown Woman to the 1948 Hollywood version of the same film. Analysis of the issues of female discourse, female ...silence, and theme music in the two films,shows how historical and cultural differences determine different sound discourses for women. Furthermore, sound design influences the ways we interpret and understand the "unknown woman" in the Chinese version of Letter from an Unknown Woman.
Finding the right sound effects (SFX) to match moments in a video is a difficult and time-consuming task, and relies heavily on the quality and completeness of text metadata. Retrieving high-quality ...(HQ) SFX using a video frame directly as the query is an attractive alternative, removing the reliance on text metadata and providing a low barrier to entry for non-experts. Due to the lack of HQ audio-visual training data, previous work on audio-visual retrieval relies on YouTube (in-the-wild) videos of varied quality for training, where the audio is often noisy and the video of amateur quality. As such it is unclear whether these systems would generalize to the task of matching HQ audio to production-quality video. To address this, we propose a multimodal framework for recommending HQ SFX given a video frame by (1) leveraging large language models and foundational vision-language models to bridge HQ audio and video to create audio-visual pairs, resulting in a highly scalable automatic audio-visual data curation pipeline; and (2) using pre-trained audio and visual encoders to train a contrastive learning-based retrieval system. We show that our system, trained using our automatic data curation pipeline, significantly outperforms baselines trained on in-the-wild data on the task of HQ SFX retrieval for video. Furthermore, while the baselines fail to generalize to this task, our system generalizes well from clean to in-the-wild data, outperforming the baselines on a dataset of YouTube videos despite only being trained on the HQ audio-visual pairs. A user study confirms that people prefer SFX retrieved by our system over the baseline 67% of the time both for HQ and in-the-wild data. Finally, we present ablations to determine the impact of model and data pipeline design choices on downstream retrieval performance. Please visit our project website to listen to and view our SFX retrieval results.
In recent years, researchers have discovered a new type of signaling mechanism based on cyclic di-GMP (c-di-GMP). This newly recognized second messenger is strongly associated with the transition ...between the motile and sessile states of many bacteria as well as the formation and dispersion of biofilms in many bacteria.
PDF
