Music-driven dance synthesis by multimodal dance performance analysis

Abstract

Bu tezde coklu modelli dans performans analizi ile muzikle surulen dans sentezininyapılabilmesi amaclanmıstır. Dans ? gurleri, muzik harmonisi ile uyumlu, muzik ritmiyleeszamanlidir ve isitsel oznitelikler kullanılarak analiz edilebilir. Dans ? gurleriyle ilintiliisitsel oruntuleri belirlemek amacıyla, isitsel veri spektrumuna ait oznitelikler, SMM yapılarıkullanılarak modellenmistir. Elde edilen isitsel oruntuler ve dans ? gurleri arasındaki ilinti,eszamanlı gerceklesme performansları hesaplanarak de gerlendirilmistir. Isitsel veri olarak mel frekansı sepstral katsayıları ve renksel parlaklıgı temsil eden kroma oznitelikleri elealınmıstır. Bu tezde sunulan sistem kullanılarak, isitsel veriyle surulen vucut animasyonusentezi calısmalarının gelistirilmesi amaclanmaktadır.

We present a framework for audio-visual analysis of dance performances towards the goalof music-driven dance synthesis. Dance ? gures, which are performed synchronously with themusical rhythm, can be analyzed through the audio spectra using spectral and chromaticmusical features. In the proposed multimodal dance performance analysis system, dance? gures are manually labeled over the video stream and modeled by employing HMMs.The music segments, which correspond to beat and meter boundaries, are used to trainhidden Markov model (HMM) structures to learn meter related temporal audio patternswhich are correlated with the dance ? gures. Bi-gram based co-occurences of temporal audiopatterns and dance ? gures are calculated. and bi-gram based co-occurrence performancesfor two different audio feature streams are evaluated. In our evaluations, mel-scale cepstralcoeffcients (MFCC) with their ? rst and second derivatives and chroma features are usedas our candidate audio feature set. The proposed framework in this thesis, can be usedtowards analysis and synthesis of audio-driven human body animation.