Direct Selection of Text in a GUI using Speech Recognition

Jennifer Lai

IBM Thomas J. Watson Research Center
Jennifer_Lai@vnet.ibm.com

CHI '97 Workshop: Speech User Interface Design Challenges

Short Biography

I have been working with the Speech Recognition effort at IBM Research for 8 years. For the past two years, I was responsible for the design of the User Interface for MedSpeak/Radiology; a continuous speech dictation product. As part of that responsibility, I was the primary contact with the radiologists at our client sites. I developed an understanding of the workflow and used that knowledge to define usage scenarios for the development team as well as the screens and dialogs for the application. During the four trial periods when iterations of the software were installed out at the sites, I collected feedback from the radiologists and modified the UI to reflect their input.

Position Paper

Using Speech Recognition to dictate text gives one the ability to interact with the computer in a more "natural" way, especially if one is using continuous recognition, which eliminates the need to insert distinct pauses between each word. In developing the MedSpeak solution, we created a product for radiologists that allowed them to dictate their findings when interpreting x-rays and other types of patient images [1]. Observations of users in hospitals showed that the radiologists that are able to dictate fluently, almost as if they have the report pre-written in their heads before starting to speak, are the ones that adapt most readily to using our system, and that have the lowest error rates. Radiology may lend itself somewhat towards this style of writing since many of the interpretations for normal x-rays are short and repetitive.

When looking at a broader spectrum of users with a need to create documents, for example lawyers, journalists and scientists, it is unusual to find users that can dictate in such a manner. Many of us, go through a process that has been referred to as "tinkering" [2], or what I think of as the crafting of a document. The process of crafting with speech-only input severely taxes the naturalness of the interface since many speech systems require the user to move between command mode and dictation mode in order to make changes to the dictated text. This switching of modes can be cumbersome and error-prone. Additionally, a cognitive burden is placed on the user to remember what state he is in, and what the correct sequence of actions is to perform what would be considered a simple operation if working with paper and pencil. In this position paper I will propose one such simple operation, a cut-and-paste, as an interesting challenge when designing a GUI that uses speech as its primary mode of input. I will discuss direct cursor placement with speech as the primary obstacle to implementing an intuitive and effective method for highlighting and moving text.

The Problem

Imagine sitting down to create a document, a CHI Workshop position paper for example. You've managed to string the perfect sequence of words together to express an idea but you change your mind about where you want the idea placed in the document. Such rearrangement of text is a common occurrence when crafting a document. We do this because the order in which we present the facts is often as important as the facts themselves, and certainly part of the effectiveness of language in writing, is in the presentation. If working with a GUI, the standard process for moving text is to highlight the words to be included, cut the highlighted section and then place the words in the new location. These three steps: highlight, cut and paste translate to a series of commands that involve direct selection of text, and placement of that text by means that are random in nature rather than sequential. For this reason, cursor placement with a mouse has always been the most effective means of carrying out the task. Clicking with a mouse is our way of pointing, of saying "there" and this is hard to reproduce with speech input.

Examples

Most speech recognition commands for cursor movement are relative to the current placement of the cursor and allow for sequential access of the text rather than random. Because we can't "pick up the cursor" and move it as we do with a mouse, we must advance towards the desired location with a series of relative commands such as "up one", or "over two". These two commands have a built-in understanding that we are talking about lines when we say up, and words when we say over. There is also an implied left to right movement with the command "over". Usually more explicit commands such as "right two" or "left two" leave less room for error. They require the user to count the number of lines or words he wants to move, but are less tedious than commands that move the cursor one unit at a time, such as "up", "up", "up".

In human to human interaction, we tend to speak in terms of paragraphs and sentences. For example we might say "it's in the second sentence of the first paragraph". This can be done programmatically, allowing the user to issue a command such as "move back 3 sentences", if the application is savvy enough to understand about sentences and paragraphs. This method for interaction is still far from ideal since it does not allow for direct random access, and is prone to error. Errors can result either from incorrectly decoded speech, or from a mistake in the logic for identifying a sentence. Take the following sentence for example:

I told Mr. Wright that Dr. R. Gregory and I have every intention of moving the T.L.C company to L.A. to produce 3.5 inch drives for 100 MHz. machines, and all he said was "ahhh ...".

There is ample opportunity for the application to select the wrong period as the sentence boundary. As a matter of fact, the Lotus Word Pro grammar check identified what it thought was four sentences in the single example sentence. This method for moving the cursor also places a burden on the user to count how many sentences backward or forward he wants to move. The possibility for compounded errors and thus high frustration on the part of the user, abound. Once the cursor has been placed at the starting point for the marked text the end point still has to be indicated. Having highlighted the text , the user must go through the same process again to indicate where the text must be moved to.

StoryWriter introduced both the Point and Speak and Pointer Touch interaction methods for cursor movement [2], allowing users who suffer from RSI to position the cursor by moving the pointer and not clicking on the mouse. While these methods support the needs of the targeted user population, they still require the mouse for positioning.

MedSpeak attempts to support a hands-free interface with the commands "rewind" and "forward". These commands move the cursor continuously from its current position until the user says "stop". The cursor may be moved a line at a time or a word at a time; the default setting is to move by word. Additionally, the user may chose to cumulatively highlight text as the cursor moves, or merely to move the cursor without highlighting. The default setting is to not highlight. We invested some time fine-tuning the mechanism for stopping because initially the delay in decoding the stop command was sufficient to cause the cursor to have moved past the desired location. We fixed this by both tweaking the speed at which the cursor moves and by halting the movement as soon as the microphone picks up a change in input volume indicating that the user is about to speak. Once halted, we wait to see if the user has issued the stop command before continuing. This actually happens fast enough that the pause is not disconcerting to the user if it turns out that he has merely coughed or said something other than stop.

This type of movement remains sequential, and our users found it cumbersome. During the trial periods, not a single user opted for voice navigation over direct cursor placement with mouse. This finding may be skewed by the fact that less time was spent during the training period on this type of navigation than on other aspects of the interface.

In order to truly support the crafting of a document using speech as the primary mode of input, a more intuitive and satisfying means of direct cursor placement has to be found.

References

1. Lai, J. and Vergo, J. MedSpeak: Report Creation with Continuous Speech Recognition. Paper to be presented at ACM SIGCHI '97.

2. Danis, C. et. al. Story Writer: A Speech Oriented Editor. ACM SIGCHI '94 Conference Companion, pp. 277-278.

CHI '97 Workshop: Speech User Interface Design Challenges