•Quantifiable movement signals can be obtained from regular video using the methods described here.•Speech-accompanying gesture can be detected semi-automatically from movement signals.•Communicative ...tasks can be distinguished by measuring quantitative properties of movement signals in a given task.•Techniques described here can improve existing observational gesture annotation strategies in multimodal speech research.
This study presents techniques for quantitatively analyzing coordination and kinematics in multimodal speech using video, audio and electromagnetic articulography (EMA) data. Multimodal speech research has flourished due to recent improvements in technology, yet gesture detection/annotation strategies vary widely, leading to difficulty in generalizing across studies and in advancing this field of research. We describe how FlowAnalyzer software can be used to extract kinematic signals from basic video recordings; and we apply a technique, derived from speech kinematic research, to detect bodily gestures in these kinematic signals. We investigate whether kinematic characteristics of multimodal speech differ dependent on communicative context, and we find that these contexts can be distinguished quantitatively, suggesting a way to improve and standardize existing gesture identification/annotation strategy. We also discuss a method, Correlation Map Analysis (CMA), for quantifying the relationship between speech and bodily gesture kinematics over time. We describe potential applications of CMA to multimodal speech research, such as describing characteristics of speech-gesture coordination in different communicative contexts. The use of the techniques presented here can improve and advance multimodal speech and gesture research by applying quantitative methods in the detection and description of multimodal speech.
•Visual information for lexical tone is more resilient in running speech than auditory information for tone.•Locus of visual information for lexical tone identified; face and especially head ...motion.•Indication that up-down rotation of the head differentiates production of different tones.•Face and head motion in a speaker's articulations are used by perceivers to discriminate tones.•Information for visual speech perception differs for tone and phones.•For tone discrimination, face and head motion are both necessary; for phone (consonants, vowels) discrimination, face motion sufficient, head motion irrelevant.
Previous studies show that lexical tones can be discriminated visually, but the locus of this information is unknown. Here we investigate the role of visual face and head information in the production and perception of the six Cantonese lexical tones. Experiment 1 (speech production) showed that tone category membership is predicted independently by each of three visual components, Head-only, Face-only, and combined Face&Head motion, that Face&Head combined is a better predictor than the sum of Head-only and Face-only, that Head-only was a better predictor than Face-only, and that up-down head motion may be involved. In Experiment 2 (speech perception) discrimination of Cantonese tones and phones (consonants and vowels) was investigated in Auditory-Only (AO), Visual-Only (VO), and Auditory-Visual (AV) presentations of Face-only, Head-only or combined Face&Head information. There was successful visual discrimination of both phones and tones, but in different manners. For VO, face movement (Face-only, Face&Head) was sufficient for Phone discrimination, whereas both head and face (Face&Head) were necessary for Tone discrimination; and Phone discrimination was better for Face-only than Head-only, whereas Tone discrimination was the better for Head-only than Face-only. For visual augmentation of auditory information, (AV-AO)/AO), for Phones there was significant facilitation in Head-only and Face-only, but not Face&Head, whereas for Tones there was significant facilitation only in the Face&Head condition. Overall, head motion is integrally involved in the differential production of Cantonese tones and, as for consonants and vowels, there is visual perception of lexical tone, albeit based on entirely different cues: face motion is sufficient for phone perception (head motion is not necessary), whereas head motion is necessary but not sufficient for tone perception (Face&Head required). Head motion is integral both to the differential production of Cantonese tones and to the visual discrimination of those tones.
Articulatory coordination of two vocal tracts Vatikiotis-Bateson, Eric; Barbosa, Adriano Vilela; Best, Catherine T.
Journal of phonetics,
05/2014, Letnik:
44, Številka:
May
Journal Article
Recenzirano
Odprti dostop
In this paper, we quantify the time-varying coordination of articulator motion for two speakers engaged in two face-to-face speaking tasks: repetition of simultaneously produced words (and ...word-pairs) and conversation. Correlation map analysis is used to compute the time-varying, correlation between pairs of signals for all possible pairings of signal values. The resulting two-dimensional map of correlation values captures the fluctuations that occur naturally in the coordination within and between speakers, regardless of whether the behaviors are intentionally synchronized or merely entrained (as in conversation). This not only provides a more accurate picture of the correlation between signals, but also underlines the importance of treating fluctuations as potentially important components of biological behavior, rather than discounting them simply as noise.
•Overview of methodology for assessing coordination computationally.•Preliminary method introduced for tracking temporal fluctuations in correlation.•Correlation analysis of articulatory coordination between two speakers.•Correlation analysis of interarticulator coordination within each speaker.
This paper demonstrates an algorithm for computing the instantaneous correlation coefficient between two signals. The algorithm is the computational engine for analyzing the time-varying coordination ...between signals, which is called correlation map analysis (CMA). Correlation is computed around any pair of points in the two input signals. Thus, coordination can be assessed across a continuous range of temporal offsets and be detected even when changing over time due to temporal fluctuations. The correlation algorithm has two major features: (i) it is structurally similar to a tunable filter, requiring only one parameter to set its cutoff frequency (and sensitivity), (ii) it can be applied either uni-directionally (computing correlation based only on previous samples) or bi-directionally (computing correlation based on both previous and future samples). Computing instantaneous correlation for a range of time offsets between two signals produces a 2D correlation map, in which correlation is characterized as a function of time and temporal offset. Graphic visualization of the correlation map provides rapid assessment of how correspondence patterns progress through time. The utility of the algorithm and of CMA are exemplified using the spatial and temporal coordination of various audible and visible components associated with linguistic performance.
A comparação forense de locutor (CFL) consiste no confronto entre características de dois áudios, com o objetivo de associar as falas do áudio questionado a um indivíduo conhecido. Esse áudio, na ...maioria dos casos, é oriundo de interceptações telefônicas e possui codificação GSM (Global System Mobile), banda estreita e ruído de canal. Levantamentos do cenário mundial em CFL, realizados em 2011 e 2016, respectivamente pela Universidade de York e INTERPOL, indicaram que muitos peritos forenses baseavam-se em análises perceptuais e acústicas. Em contrapartida, a utilização de metodologias automáticas e assistidas são menos utilizadas. Nesse nicho, o presente trabalho busca explorar o potencial de características/descritores acústicos, como Componentes Mel Cepstrais e analisar o poder discriminante destas características acústicas extraídas de corpus. Os experimentos utilizaram cinco tipos de ruído em seis níveis de relação sinal ruído. Os cenários das comparações visam aproximar as condições forenses considerando a codificação GSM, a banda do sinal e o ruído de canal.
This work presents a method for lexical tone classification in audio-visual speech. The method is applied to a speech data set consisting of syllables and words produced by a female native speaker of ...Cantonese. The data were recorded in an audio-visual speech production experiment. The visual component of speech was measured by tracking the positions of active markers placed on the speaker's face, whereas the acoustic component was measured with an ordinary microphone. A pitch tracking algorithm is used to estimate F0 from the acoustic signal. A procedure for head motion compensation is applied to the tracked marker positions in order to separate the head and face motion components. The data are then organized into four signal groups: F0, Face, Head, Face+Head. The signals in each of these groups are parameterized by means of a polynomial approximation and then used to train an LDA (Linear Discriminant Analysis) classifier that maps the input signals into one of the output classes (the lexical tones of the language). One classifier is trained for each signal group. The ability of each signal group to predict the correct lexical tones was assessed by the accuracy of the corresponding LDA classifier. The accuracy of the classifiers was obtained by means of a k-fold cross validation method. The classifiers for all signal groups performed above chance, with F0 achieving the highest accuracy, followed by Face+Head, Face, and Head, respectively. The differences in performance between all signal groups were statistically significant.
Video Tracking Of 2D Face Motion During Speech Barbosa, A.V.; Vatikiotis-Bateson, E.
2006 IEEE International Symposium on Signal Processing and Information Technology,
2006-Aug.
Conference Proceeding
We present a video-based system for tracking 2D face motion during speech. The system tracks dot markers on the speaker's face from video sequences. A digital video camera is used to film the speaker ...during speech production experiments. The acquired digital video is converted into an image sequence, which is then provided as input to the tracking algorithm. The algorithm locates all dots in every frame of the sequence, and outputs the x and y coordinates of the dots in the images over time. Continuity constraints on the face motion are imposed when searching for the dots. The algorithm was tested on video sequences for both spontaneous and non-spontaneous (i.e., reading) speech conditions, and no mistracking occurred for the sequences used
PDF
