Using Novel Dynamic Modeling Technique to Explore Synchrony of Facial Expressions and Speech in Dyadic Conversations 276 views

Communication and conversation are important human behaviors and have been modeled before. However, very little research has encompassed the full dynamics of and the relationship between facial expressions and speech between people measured intensively across time, including the processes of turn-taking, delays between participants, and synchrony. Furthermore, these dynamics may be best represented in a multidimensional framework, which is not typically used in this research. In this work, I introduce the novel method of windowed canonical correlation analyses in order to be able to analyze the relationships between two sets of intense longitudinal multidimensional data, such as data from two individuals in a conversation. I then apply this method and windowed cross-correlations to point coordinates from motion tracked faces and amplitudes across frequencies from transformed speech in order to explore whether synchrony and turn-taking can be found between facial expressions and speech in unscripted dyadic conversations. After performing these analyses, the speech component of conversations was found to drive the correlations between speech and faces and play a large part in the dynamics of the conversations. Limited evidence of synchrony and some evidence towards turn-taking were also found between individuals.

PHD (Doctor of Philosophy)

facial expressions; speech; synchrony; turn-taking; active appearance models; short-time Fourier transform; windowed cross-correlations; windowed canonical correlation analysis

English

2018-05-02