Full metadata record
DC FieldValueLanguage
dc.contributorDepartment of Computingen_US
dc.contributor.advisorChan, Stephen (COMP)-
dc.contributor.advisorNgai, Grace (COMP)-
dc.creatorTang, Wai Wa-
dc.publisherHong Kong Polytechnic University-
dc.rightsAll rights reserveden_US
dc.titleAn investigation into crossmodal and multimodal human computer interaction : handwriting, music generation and body gestureen_US
dcterms.abstractThis thesis presents an investigation into cross-modal and multimodal human computer interaction. Human interaction is by nature multimodal and crossmodal. Human beings communicate through multiple channels of speech, gestures, facial expressions, body language, etc. In addition, cross-modal transformations, where ideas and feelings are expressed in multiple modalities, are fairly common in art. However, these modes of expression and communication have so far been mostly ignored in human computer interaction. We focus on the transformation of input modalities to alternate output modalities. The primary focus is on the transformation from writing to music, specifically Chinese calligraphy to Chinese music. We investigate both real-time as well as offline modes of transformation, and experiment with different levels of human control versus statistically driven modeling. We present evaluations that show that the music that is generated is both pleasing to the ear as well as correlates with the Chinese style of music. As an additional investigation, we also consider two other cross-modal transformations: the mapping of body motions and gestures to music manipulation operations, and the mapping of handwriting to graphics. Both transformations are motivated and demonstrated with real-life applications; preliminary evaluations show that they can encourage interaction, stimulate creativity as well as serve as a motivation to practice the input modality.en_US
dcterms.extentx, 62 pages : color illustrationsen_US
dcterms.isPartOfPolyU Electronic Thesesen_US
dcterms.educationalLevelAll Masteren_US
dcterms.LCSHHong Kong Polytechnic University -- Dissertationsen_US
dcterms.LCSHHuman-computer interactionen_US
dcterms.accessRightsopen accessen_US

Files in This Item:
File Description SizeFormat 
991022249145703411.pdfFor All Users7.81 MBAdobe PDFView/Open

Copyright Undertaking

As a bona fide Library user, I declare that:

  1. I will abide by the rules and legal ordinances governing copyright regarding the use of the Database.
  2. I will use the Database for the purpose of my research or private study only and not for circulation or further reproduction or any other purpose.
  3. I agree to indemnify and hold the University harmless from and against any loss, damage, cost, liability or expenses arising from copyright infringement or unauthorized usage.

By downloading any item(s) listed above, you acknowledge that you have read and understood the copyright undertaking as stated above, and agree to be bound by all of its terms.

Show simple item record

Please use this identifier to cite or link to this item: https://theses.lib.polyu.edu.hk/handle/200/10087