Keywords
Systemic functional approach to multimodal discourse analysis
Multimodality
Pedagogy
Students from Hong Kong, Macao, and Taiwan regions
Submitted : 2025-01-19
Accepted : 2025-02-03
Published : 2025-02-18
Abstract
This study investigates the digital multimodal composing competence of fifty freshmen from Hong Kong, Macao, and Taiwan regions in a Chinese University based on a systemic functional approach to multimodal discourse analysis (SF-MDA) and sociosemiotic ethnography. These students are divided into six groups and are required to make presentations on topics like sports, fashion, festivals, environment, and cities. The collected data include students’ digital multimodal texts and interview responses. Findings show that a) students have the awareness of salience, but often misrepresent it; b) students are likely to confuse the relationship and function between the text and image as they often use the image with the least or redundant information to complement the text; c) students are likely to incorporate something irrelevant to the topic; d) students misuse the immersive image as a background due to lack of the concept of context; e) they usually provide excessive language information. This study also puts forward suggestions on the pedagogy of improving these students’ multimodal literacies through optimizing the existing teaching method and setting up an instantaneous recall feedback form.
References
New London Group, 1996, A Pedagogy of Multiliteracies: Designing Social Features. Harvard Educational Review, 66(1): 60–92.
Street B, 1984, Literacy in Theory and Practice. Cambridge University Press, Cambridge.
Dalton BK, Robinson J, Lavvorn BE, et al., 2015, Fifth-grade Students’ Multimodal Compositions: Modal Use and Design Intentionality. Elementary School Journal, 115(4): 548–569.
Takayoshi P, Seife CL, 2007, Thinking about Multimodality, in Multimodality Composition: Resources for Teachers. Hampton, New Jersey, 1–12.
Vasudevan L, Schultz K, Bateman J, 2010, Rethinking Composing in a Digital Age: Authoring Literate Identities through Multimodal Storytelling. Written Communication, 27(4): 442–468.
Beach R, O’Brien D, 2015, Fostering Students’ Science Inquiry through App Affordances of Multimodality, Collaboration, Interactivity, and Connectivity. Reading & Writing Quarterly, 31(2): 119–134.
Burnett C, 2016, Being Together in Classrooms at the Interface of the Physical and Virtual: Implications for Collaboration in On/off-screen Sites. Learning, Media and Technology, 41(4): 566–589.
Smith BE, 2019, Collaborative Multimodal Composing: Tracing the Unique Partnerships of Three Pairs of Adolescents Composing across Three Digital Projects. Literacy, 53(1): 14–21.
Ito M, Horst H, Bittanti M, et al., 2010, Hanging Out, Messing around and Geeking out: Kids Learning and Living in New Media. MIT Press, Cambridge.
Jocius RK, 2018, Becoming Entangled: An Analysis of 5th Grade Students’ Collaborative Multimodal Composing Practices. Computers and Composition, 2018(47): 14–30.
Jiang LJ, Hafner C, 2024, Digital Multimodal Composing in L2 Classrooms: A Research Agenda. Language Teaching, 1–19. https://doi.org/10.1017/S0261444824000107
Kim Y, Kang S, Nam Y, et al., 2022, Peer Interaction, Writing Proficiency, and the Quality of Collaborative Digital Multimodal Composing Task: Comparing Guided and Unguided Planning. System, 106(4): 102722. https://doi.org/10.1016/j.system.2022.102722
Chandler PD, 2017, To What Extent Are Teachers Well Prepared to Teach Multimodal Authoring? Cogent Education, 4(1): 1–19.
Liang WJ, Lim FV, 2020, A Pedagogical Framework for Digital Multimodal Composing in the English Language Classroom. Innovation in Language Learning and Teaching, 15(4): 306–320. https://doi.org/10.1080/17501229.2020.1800709
Ajayi L, 2009, English as a Second Language Learners’ Exploration of Multimodal Texts in a Junior High School. Journal of Adolescent & Adult Literacy, 2009(52): 585–595.
Toohey K, Dagenais D, Fodor A, et al., 2015, That Sounds so Cooool: Entanglements of Children, Digital Tools, and Literacy Practices. TESOL Quarterly, 49(3): 461–485.
Shin D, 2018, Multimodal Mediation and Argumentative Writing: A Case Study of a Multilingual Writer’s Metalanguage Awareness Development, in Bilingual Learners and Social Equity: Critical Approaches to Systemic Functional Linguistics. Springer, New York, 225–242.
Smith B, Pacheco M, de Almeida CR, 2017, Multimodal Codemeshing: Bilingual Adolescents’ Processes Composing across Modes and Languages. Journal of Second Language Writing, 2017(36): 6–22.
Shin D, Cimasko T, Yi Y, 2020, Development of Metalanguage for Multimodal Composing: A Case Study of an L2 Writer’s Design of Multimedia Texts. Journal of Second Language Writing, 2020(47): 100714. https://doi.org/10.1016/j.jslw.2020.100714
Kress G, van Leeuwen T, 2001, Multimodal Discourse: The Modes and Media of Contemporary Communication. Oxford University Press, New York.
Prior P, 2005, Moving Multimodality beyond the Binaries: A Response to Gunther Kress’ “Gains and Losses”. Computers and Composition, 2005(22): 23–30.
Halliday MAK, 1985, Spoken and Written Language. Deakin University Press, Geelong.
Halliday MAK, Matthiessen CMIM, 2014, An Introduction to Functional Grammar. Hodder Arnold, London.
Eggins S, 2004, An Introduction to Systemic Functional Linguistics. Bloomsbury, London.
Unsworth L, 2006, E-literature for Children: Enhancing Digital Literacy Learning. Routledge, England.
Painter C, Martin JR, Unsworth L, 2013, Reading Visual Narratives: Image Analysis of Children’s Picture Books. Equinox, London.
Kress G, van Leeuwen T, 2006, Reading Images: The Grammar of Visual Design. Routledge, London.
Unsworth L, 2006, Towards a Metalanguage for Multiliteracies Education: Describing the Meaning-making Resources of Language-image Interactions. English Teaching: Practice & Critique, 5(1): 55–76.
Jewitt C, Bezemer J, O’Halloran K, 2016, Introducing Multimodality. Routledge, London.
Unsworth L, Mills KA, 2021, The Affordances of Multimodal Literacy Practices in Digital and Non-digital Learning Contexts for Language and Literacy Learners. Literacy, 55(2): 75–85. https://doi.org/10.1111/lit.12212
Shanks N, Bakay L, 2022, Examining the Impact of Multimodal Composing on English Learners’ Academic Literacy Skills. Journal of Adolescent & Adult Literacy, 65(6): 563–573. https://doi.org/10.1002/jaal.1167
Kress G, Van Leeuwen T, 2020, Reading Images: The Grammar of Visual Design. Routledge, London.
Tversky B, Morrison JB, Betrancourt M, 2002, Animation: Can it Facilitate? International Journal of Human-Computer Studies, 57(4): 243–253.
Mayer RE, 2001, Multimedia Learning. Cambridge University Press. Cambridge.
Pirini J, 2017, Approaches to Multimodal Discourse Analysis. Oxford Bibliographies in Communication. https://doi.org/10.1093/obo/9780199756841-0183
Wilmes F, Siry C, 2020, Multimodal Interaction Analysis: a Powerful Tool for Examining Plurilingual Students’ Engagement in Science Practices. Journal of Science Education and Technology, 29(6): 1–15. https://doi.org/10.1007/s11165-020-09977-z
Hansford D, Adlington R, 2009, Digital Spaces and Young People’s Online Authoring: Challenges for Teachers. Australian Journal of Language and Literacy, 32 (1–2): 55–68.
Rose D, Martin J, 2012, Learning to Write, Reading to Learn: Genre, Knowledge and Pedagogy of the Sydney School. Equinox, Oakville.
Lim FV, Unsworth L, 2023, Multimodal Composing in the English Classroom: Recontextualising the Curriculum to Learning. English in Education, 57(2): 102–119. https://doi.org/10.1080/04250494.2023.2187696
Lim FV, Tan-Chia L, 2018, Designing Learning for Multimodal Literacy: Teaching Viewing and Representing. Routledge, London.
Lim FV, Cope B, Kalantzis M, 2022, A Pedagogic Metalanguage for Learning: Rebalancing the Cognitive with the Socio-Material. Frontiers in Communication, 2022(7): 830613. https://doi.org/10.3389/fcomm.2022.830613
Hafner CA, Ho WYJ, 2020, Assessing Digital Multimodal Composing in Second Language Writing: Towards a Process-based Model. Journal of Second Language Writing, 2020(47): 100710.
Hung HT, Chiu YC J, Yeh HC, 2013, Multimodal Assessment of and for Learning: A Theory-driven Design Rubric. British Journal of Educational Technology, 44(3): 400–409. https://doi.org/10.1111/j.1467-8535.2012.01337.x
Alt D, Raichel N, 2022, Problem-Based Learning, Self- and Peer Assessment in Higher Education: Towards Advancing Lifelong Learning Skills. Research Papers in Education, 37(3): 370–394. https://doi.org/10.1080/02671522.2020.1849371
Dzekoe R, 2017, Computer-based Multimodal Composing Activities, Self-revision, and L2 Acquisition through Writing. Language Learning & Technology, 21(2): 73–95. https://doi.org/10125/44612
Lee SY, Lo YH, Chin TC, 2019, Practicing Multiliteracies to Enhance EFL Learners’ Meaning-making Process and Language Development: A Multimodal Problem-based Approach. Computer Assisted Language Learning, 34(1–2): 66–91. https://doi.org/10.1080/09588221.2019.1614959
Li M, 2020, Multimodal Pedagogy in TESOL Teacher Education: Students’ Perspectives. System, 2020(94): 102337. https://doi.org/10.1016/j.system.2020.102337
Zhang ED, Yu S, 2023, Implementing Digital Multimodal Composing in L2 Writing Instruction: A Focus on Developing L2 Student Writers. Innovation in Language Learning and Teaching, 17(4): 769–777.
Zheng YL, 2019, Meaning Construction of Multimodal Discourse in High School English PPT Courseware—A Case Study of the 10th National Demonstration Lectures for High School English Teachers. Theory and Practice in Language Studies, 9(9): 1134–1139. http://dx.doi.org/10.17507/tpls.0909.09
Ledin P, Machin D, 2019, Doing Critical Discourse Studies with Multimodality: From Metafunctions to Materiality. Critical Discourse Studies, 16(5): 497–513. https://doi.org/10.1080/17405904.2018.1468789
Feng D, 2021, Genre, Pedagogy, and PowerPoint Design: A Multimodal Move Analysis of Linguistics Lecture Slideshows, in Approaches to Specialized Genres: In Memory of Stephen Evans. Routledge, London.