| Author: | Chen, Zhuojia |
| Title: | Subtitling as multimodal representation : a corpus-based experimental approach to text-image relations |
| Advisors: | Wu, Zhiwei (CBS) Li, Dechao (CBS) |
| Degree: | Ph.D. |
| Year: | 2024 |
| Subject: | Subtitles (Motion pictures, television, etc.) Eye tracking Translating and interpreting Hong Kong Polytechnic University -- Dissertations |
| Department: | Department of Chinese and Bilingual Studies |
| Pages: | xix, 165 pages : color illustrations |
| Language: | English |
| Abstract: | Communication is intrinsically a multimodal act. In an audiovisual product, communication seldom involves mere language but incorporates a strong relation between language and image. One of the challenges in audiovisual translation (AVT) is the thin line between assisting and distracting the audience. While translators need to maximize viewers’ comprehension of the linguistic content, they also need to minimize viewers’ efforts in enjoying the audiovisual product as a multimodal ensemble. This project is composed of a corpus-based study and an eye-tracking experiment, attempting to explore: (a) how interlingual subtitles interact with the image in films, and (b) how the target subtitle’s interactions with the image may affect viewer reception. The corpus-based study and the experiment are underlined by two frameworks proposed by the author to theorize text-image relations. The first framework is put forward to analyze text-image relations in subtitled films. It is adapted from previous frameworks by Martinec and Salway (2005), Unsworth (2006, 2007), and Pastra (2008), with four major categories of text-image relations (4 Cs): Concurrence (text = image), Complementarity (text > image), Condensation (text < image), and Contradiction (text ≠ image). The other framework focuses on translation shifts in text-image relations specifically within the context of interlingual subtitling. This framework is formulated through a bottom-up approach, drawing on real-life subtitling cases. It comprises five major categories: non-shifts, obligatory shifts, preferential shifts, strengthening shifts (4 Es: expansion, explicitation, enhancement, and elaboration), and weakening shifts (4 Ds: detachment, diminution, dilution, and decrement). The first three types of shifts are believed to be language-induced, driven by the subtitler’s linguistic consideration between the target and source languages. The latter two types are considered as image-induced shifts, arising from the subtitler’s conscious or unconscious attention to the pictorial elements during the translation process. For a more systematic analysis, the two frameworks draw on transitivity systems from Systemic Functional Grammar (Halliday & Matthiessen, 2014) and Visual Grammar (Kress & Van Leeuwen, 2021) to delineate the basic comparative units for analyzing text-image relations, namely, the verbal and visual participants, processes, and circumstances. To explore how the text interacts with the image in audiovisual products, the first main study employs a corpus-based approach. A multimodal corpus was compiled, consisting of 30 scenes sampled from 10 English films, each containing source English subtitles and target Chinese subtitles. The findings revealed that both the source and target texts were semantically more specific than the image, as the frequency of complementarity relations was over seven times that of condensation. Moreover, the meanings of the text and image tended to be closely associated, as evidenced by the substantial number of concurrence relations. In terms of translation shifts in text-image relations through subtitling, three patterns were observed. The target subtitles tended to (a) avoid mentioning the visual participants (e.g., people and things) and leave them implicit; (b) verbalize and explicate the co-occurring visual processes (e.g., actions and gestures); and (c) include additional linguistic circumstances (e.g., the manner of an action) to describe or modify the visual processes. The corpus findings shed light on the role of interlingual subtitling as multimodal representation, which may remove, reiterate, or reinforce the visual information in the multimodal film narrative. To investigate the potential impact of the translation shifts observed in the corpus, the other main study of the project adopts an experimental approach. Drawing on eye-tracking data, comprehension tests, perception questionnaires, and semi-structured interviews, the experiment assigned 82 participants to either a Control Group or an Experimental Group and investigated the impact of translation shifts in text-image relations on the participants’ visual attention, comprehension, perception of subtitle quality, and preferences for subtitling methods. The results showed that explicating the visual actions in the target subtitles induced significantly better comprehension and shorter gaze time to the subtitle-related visual information. Moreover, the viewers appeared to perceive lower subtitle quality when target subtitles contained less referential information related to the visual entities. However, this did not necessarily hinder the viewers’ comprehension or affect their distribution of visual attention. In terms of the viewers’ preferences for subtitling methods, both groups strongly favored the method of providing more descriptive information to modify the visual kinesics, which helped reinforce the traits or emotions of the portrayed character. The findings from the experiment highlight the potential influence of subtitlers in guiding, or even manipulating, the cognitive attention of target viewers towards the audiovisual content. The research contributions of this thesis are threefold. Theoretically, two frameworks have been proposed to systematically (re)conceptualize subtitled products as multimodal ensembles by highlighting the text-image interplay, thus further moving AVT research in the direction of multimodality. Methodologically, this thesis represents one of the few attempts to combine the corpus-based approach and the experimental approach in a single project, which has demonstrated the feasibility of using multiple methods in empirical AVT studies. For translation praxis, the findings from the thesis can support audiovisual translators or trainees in making an informed decision when tackling non-verbal meaning-making elements in audiovisual products, so as to achieve a more comprehensible and multimodally immersive translation. |
| Rights: | All rights reserved |
| Access: | open access |
Copyright Undertaking
As a bona fide Library user, I declare that:
- I will abide by the rules and legal ordinances governing copyright regarding the use of the Database.
- I will use the Database for the purpose of my research or private study only and not for circulation or further reproduction or any other purpose.
- I agree to indemnify and hold the University harmless from and against any loss, damage, cost, liability or expenses arising from copyright infringement or unauthorized usage.
By downloading any item(s) listed above, you acknowledge that you have read and understood the copyright undertaking as stated above, and agree to be bound by all of its terms.
Please use this identifier to cite or link to this item:
https://theses.lib.polyu.edu.hk/handle/200/12821