Speech and Audio in a Multimodal Collaborative Environment

Samuel L. Bayer, Erica Bernstein, David Duff, Lynette Hirschman Rod Holland, Robyn Kozierok, Jeffrey Kurtz, Susann LuperFoy, Margot Peet

The MITRE Corporation
sam@linus.mitre.org

CHI '97 Workshop: Speech User Interface Design Challenges

Short Biography

The HCI group at the MITRE Corporation has been involved in research in intelligent multimodal interaction for more than ten years, with specific strengths in speech, discourse, multimodal presentation and software engineering issues relating to HCI. Recently, the HCI group has begun to address interaction issues involving collaborative technologies, in part due to MITRE's investment in collaborative computing.

Jeff Kurtz: My areas of interest are interface usability, CSCW, and software architecture. Recent projects include:

I have a bachelor's degree in math/computer science from Carnegie Mellon.

Position Paper

In this problem statement, we will address the role of speech resources in a collaborative environment inhabited by both human and computational (software) participants. As professional, academic and social interactions and collaborations become more global, we see more and more the need for software environments which minimize the tensions imposed by differences in space and time. One such environment, known as a MUD (multi-user dungeon), allows "players" to interact with the MUD and each other in such a way that computational participants can enter the environment at relatively low cost. Since all interactions with the MUD can be achieved via text, the potential for employing speech recognition (that is, speech to text) to enhance MUD clients is considerable. However, because the environment might host a mix of computational and human participants, and because there may be many such computational participants, the problem of directing speech input to the appropriate participant in these environments becomes critical. In this problem statement, we will show how this problem manifests itself in the MITRE Collaborative Virtual Workspace (CVW), a multimodal MUD client.

Background: CVW

MITRE's CVW is a multimodal client for the Xerox PARC LambdaMOO server; MOO stands for "MUD, Object Oriented". Each player in the MOO has a location; these locations are implemented in the CVW as rooms on floors of a building, and the player may move from room to room. As part of the CVW multimodal display, each player is presented with a mouseable room map when she enters the CVW, as well as an iconic inventory of the other players in the current room.

The room metaphor serves to establish a context for communication in the MOO. For instance, while a player may page another player in another room, a player may speak (communicate publicly) or whisper (communicate privately) only with other players in the same room. The protocol by which the player uses MOO commands to interact with other participants is textual, and relies on commands defined in the server; so one way for a player to address another player named Sam is to type "sayto Sam 'Hello there'".

MITRE has customized and extended the basic MOO client design to support a wider range of interaction types, including a document server supported by MIME types, a shared whiteboard, and two loosely integrated, externally developed conferencing tools: VAT, an audioconferencing tool, and NV, a videoconferencing tool. These applications both work by IP multicast, and they are integrated into the CVW to the extent that each room corresponds to a different IP multicast address. When a player using VAT or NV moves from one room to another, the current application is shut down and restarted on the new multicast address. Although the CVW client is aware that the player has spawned VAT or NV, neither the content of the audio or video communications nor the fact that the communications are made are noted by the MOO server.

The added complexity of computational participants

At this point, consider how one might introduce computational participants into such an environment. We call such participants INTERACTIVE MULTIMODAL PARTICIPANTS (IMPs), in order to avoid the rather loaded term "agent" (For our notion of "multimodal", see immediately below). On some level, each IMP in the MOO must realize when it is being addressed, digest the contents of what is addressed to it, and respond "appropriately". (This in itself is not unusual; one of the more interesting examples of this type of interaction is Julia, a so-called "chatterbot" which knows a great deal about hockey which logs onto various MUDs around the Internet. There's at least one documented report of someone trying to pick her up for two weeks without ever realizing she wasn't human. See http://foner.www.media.mit.edu/people/foner/Julia/Julia.html for the details.) On the text level, this might simply reduce to the well-examined issue of creating a computational object which understands a limited grammar and responds helpfully to utterances outside the grammar; but the problem is actually more complex than this.

The complexity arises from the multimodal aspect of IMPs. In addition to interacting with humans via text, computational objects can present graphical interfaces which provide a much wider range of interactions to the user (the CVW enhancement of the "basic" MOO client is a perfect example). In other words, the interface that the IMP presents might be different than the "interface" another human presents. We are currently working to realize this possibility within CVW. We envision a set of IMP-specific multimodal "helper applications" which players may summon to communicate specifically with a given IMP, using the CVW client as a "host" for the IMP-specific application. For example, if the IMP's area of expertise is travel arrangements, the "helper application" might feature a map which the player might be able to query, via mouse, for travel alternatives, weather conditions, etc.

At this point, speech finally enters the picture. We assume a speech recognition service which has at its disposal the recognition grammars for all "active" speech-enabled applications at any time. A human player in the environment we've just described can address, via speech, three different classes of addressees:

So let's assume that the current room is occupied by Sam, Robyn, and two IMPs: the travel IMP described above, and a second IMP whose area of expertise is automobile repair. Sam says, "How do I rent a car in Dallas?"

Three issues arise immediately, which radically extend the normal discussions about speech focus. All these problems assume that the best way to limit speech recognition errors is to limit the exposure of the speech recognizer to out-of-grammar input. All these problems involve addressing, and might profitably be conceived of in terms of determining the appropriate paradigm for reconstructing, in a collaborative computational environment, human strategies for determinating the addressee(s) in multiparty interactions.

First, how does the speech recognizer know to listen at all? In an environment where the only speech input is to a suite of speech-enabled applications, there is no question about whether the speech recognizer is processing speech; the only consideration is which application the input is intended for. In this environment, on the other hand, the input might not be intended for any application at all, but rather for another human (or group of humans).

Second, if the speech recognizer knows to listen, how does the speech recognizer determine which application speech is directed to in the multimodal context we've just described? If the addressee application is determined by mouse or keyboard focus, how does the recognizer differentiate between a focus on CVW and a focus on the IMP, since these foci might overlap in the icon inventory? What if multiple IMPs in the same environment might be qualified to respond to an utterance, such as "Does anybody know what happened in Malaysia yesterday?" If more than one IMP is identified as a possible addressee, do all of them receive a speech transcription? What if the transcriptions differ from IMP to IMP?

Third, where should speech recognition occur? The description of the problem so far assumes that speech recognition (and the choice of VAT vs. speech recognition) occurs in the context of the speaker's CVW client. One could imagine a different architecture, where speech recognition occurs on the IMP's "side" of the interaction, with the IMP listening to speech "on the wire". This architecture is more problematic because the "wire" might be carrying multiple speakers simultaneously, and the IMP might be denied access to valuable mouse and keyboard focus information which might aid it in deciding whether to "listen". Are there any advantages to this architecture?

Conclusion

We believe that as collaborative technologies become more and more available, and as speech recognition and language understanding improve, it will be inevitable that speech-enabled applications will inhabit an environment where they will not only "share" speech resources with other speech-enabled applications, but with humans as well. In this problem statement, we've described a wide range of problems. Some, such as addressing multiple IMPs, look fairly far into the future. However, all the technologies required to manifest these problems are already in place.

CHI '97 Workshop: Speech User Interface Design Challenges