This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Dowell, J. Articles by Shmueli, Y. Search for Related Content PubMed Articles by Dowell, J. Articles by Shmueli, Y. Social Bookmarking                         What's this?

Blending Speech Output and Visual Text in the Multimodal Interface

University College London, London, United Kingdom, j.dowell{at}cs.ucl.ac.uk

Yael Shmueli

University College London, London, United Kingdom

Objective: Simultaneous reading and listening with a redundant display of visual text with speech output was investigated to determine how variations in verbal working memory capacity and content complexity affected comprehension. Background: Previous work has found some evidence of a benefit for displays that blend speech and visual text; content complexity and verbal working memory capacity are likely to significantly determine this benefit. Method: In the experiment reported here, a multimodal display of e-mail messages was compared with speech output alone and with a purely visual display. Comprehension of the messages was examined in relation to verbal working memory capacity and the complexity of the messages. Thirty-two users participated in the study, which used a repeated measures design. Results: The data show that the multimodal interface did not affect comprehension relative to a purely visual interface, even when the content was more complex, although it did improve the comprehension of complex information relative to a purely auditory interface. Lower-capacity participants were neither especially advantaged nor disadvantaged by the multimodal interface. Participants expressed a marked preference for the multimodal display of the more complex sentences. Conclusion: The experiment suggests that a redundant multimodal display will neither assist nor disrupt understanding when compared with a purely visual display, but it will assist understanding of complex content when compared with speech output alone. Application: Redundant displays of visual text and speech have potential application in multitask situations, in multimedia presentations, and for devices with small screens.

Human Factors: The Journal of the Human Factors and Ergonomics Society, Vol. 50, No. 5, 782-788 (2008)
DOI: 10.1518/001872008X354165


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?