How can we take advantage from the information coming from the analysis side in order to synthesize a suitable expressive content?
A main research issue in the MEGA project is the study of the mapping strategies
of the recognized expressive content onto multimodal outputs, i.e. the exploration
about how the recognized expressiveness can influence the generation of multimodal
(sound, music, lights, movements, visual media) outputs.
The following figures show two examples employed in the MEGA
performances in years 2 and 3.
In the EyesWeb patch displayed in the figure here below the word “Gesture” is processed in real-time and moved on the screen depending on the motion cues extracted during a dance performance (the processed silhouette of the dancer is also shown in the output image). Several motion cues are used: for example, overall motion direction is used to control the motion of the word on the screen, Quantity of Motion is mapped on the color of the word, movement is segmented in pause and motion phases and the actual phase is used to switch among two different words: thus, the word “Expressive” is displayed during pauses, while the word “Gesture” is displayed when the dancer is moving. The amount of shadow and its color, both of the word and of the dancer’s silhouette can also change in real-time depending on motion and audio cues.