Using an Audio Companion Device
Noi Sukaviriya
Mobile Solutions/Personal Systems
IBM Thomas J Watson Research Center
noi@watson.ibm.com
CHI '97 Workshop: Speech User Interface
Design Challenges
Short Biography
I have some experience building systems which coordinated (automatically
generated) spoken text and animation in context-sensitive animated help. I
played with a small command-based spoken interface, mostly to replace
existing single commands using IN3. So far my experience has been mostly
with visual interfaces. I have joined IBM and the mobile solution group
for a year and a half. This group mainly looks at small devices hence the
use of speech as an alternative interaction technique is strong for lack of
screen space. I just started looking into speech interface design issues.
Design Context
The problem we are currently looking into is how an audio companion device
could be used in our daily lives. This device is a small handheld device
which users can carry with them all the time. The device has audio input
and output capability, some limited amount of memory, and perhaps with or
without a limited amount of a display surface. The goal is to give users
connectivity to their back office systems or their home systems. The
device could be connected wirelessly, but should always be operational in a
disconnected mode.
The speech group at IBM has been working on dictation systems and speech
input for command and control for quite some time now. The quality of the
recognition has been acceptable and a commercial product has been released
(VoiceType). Our group is looking to exploit the IBM speech recognition
system for mobile use. One can think of many useful functions to be
designed and incorporated into this device. Some of these functions will
lead to using speech as a means to enter long input entries. For this
workshop, I'll focus on the problem related to navigating and correcting
long content entries using speech on this device.
Design Problem
Consider telling the device to send e-mail to Judy Smith and the content is
"Judy, could you please pick up my children for me after work? I need to
take my mom to the hospital this afternoon. I don't know how long it will
take. Will call if the plan changes. Thank you." Imagine a user wanting
to correct audio reports which he collected while sampling dishes at an
International Food Contest. The report will be used later for his
newspaper article about the contest. Consider a case where a user asks the
system to read out a report halfly written while she was in the office and
now she is on a long train ride to another city. She does not want to
carry a heavy notebook system and wishes to utilize her time finishing up
this report while sitting on the train.
Using dictation has the challenge on small devices since the dictation
engine is currently too large to run locally. The device has to send the
audio signal to a recognition server and receive the recognized text back
for further access. The recognition may or may not be done while the user
is speaking into the device depending on the status of the connection. If
the user asks the system to play back what it has, first of all the
feedback must be clear whether the input has been processed or not. When
the device plays back the recorded message, there may be errors which the
user wants to correct. Even if the recognition system is perfect, there
will always be times when the user may want to correct input because wrong
words were said or the user may wish to add or subtract to the text. The
correction should be allowed whether the audio input has been recognized or
not.
Regardless of the reasons for correction, the design problem has to do with
positioning into the content of a single long input (an e-mail message, the
content of a report, annotating an audio textbook, preparing a lecture,
etc.), selecting the range of the input, calling out the action to be done
to the content, and entering new input for substitution or addition. Let's
assume that re-recording the whole input over again is too tedious and not
practical for some of these applications. The design problem we are
looking at entails the following issues:
- Is the metaphor of positioning and replacing audio input appropriate for
speech interface? Is it an inappropriate metaphor carried over from the
cut-and-paste paradigm in a visual interface?
- If it is not the right metaphor, what is a better metaphor for
manipulating audio content?
- If it is indeed an acceptable metaphor, what is an effective technique
to position and select a range of audio input (or output if we count when
it's played back)? What are design alternatives?
- Is the technique different if a small display is used with the
manipulation task? (This only works if the audio input has been recognized
and transcribed into text.)
Please note that the use of a dictation engine is somewhat an orthogonal
issue to the design problem here. I mentioned dictation to state the
design space -- that is dictation enables long text entries through speech.
CHI '97 Workshop: Speech User Interface
Design Challenges