CHI 97 Electronic Publications: Late-Breaking/Short Talks
World Wide Web as Usability Tester, Collector, Recruiter
Christopher (Blade) Kotelly
Wildfire Communications Inc.
20 Maguire Rd
Lexington, MA 02173
+1 617 674 1587
chris@wildfire.com
ABSTRACT
The usability team at Wildfire Communications Inc.
conducted a usability test using the World Wide Web (WWW) as a
method to advertise the test, recruit participants and gather
data - all automatically.
The test was conducted over the course of only 2
days during which we collected useful information from 96 people.
The usability test was for a speech system using
participants recruited by Internet Newsgroups, e-mail lists and
the WWW. Using these resources helped us to get a large population
to test the system in a short period of time.
Keywords
Usability, World Wide Web (WWW), Testing, Speech,
VUI
© 1997 Copyright on this material is held by the authors.
BACKGROUND - What is "Wildfire?"
Wildfire Communications Inc. makes an electronic
assistant called "Wildfire." The main features of the
product enable a user to place calls and receive messages with
the added benefit of navigating the system entirely by voice commands.
The Wildfire system works over regular phone lines, from any phone
in the world, and interacts with the user verbally using common
office parlance.
WHAT WE WERE TESTING
The system has a component which somewhat mimics
the functionality of an answering machine when a person calls
a Wildfire user. This "incaller" component is the component
we were concerned with testing. There are two issues which contribute
to the importance of usability testing the incaller component:
- Callers do not perceive that they get any benefit
from leaving a message with the Wildfire system. Leaving a message
with Wildfire requires that the caller interact with a system
which is slightly more complex then an answering machine (where
the person can simply wait for a familiar <beep>). We want
to determine if this version of the incaller interface reduces
the lack-of-perceived-benefit problem.
- Also, because people who call into a Wildfire account
(to leave a message) do not have an opportunity to learn about
the system or get trained on it (like a Wildfire account holder
would), they are less forgiving about the interface. We want to
ensure that these callers will be able to successfully navigate
the system with little difficulty.
The incaller portion of the product affects the largest
segment of the population who encounter Wildfire. This populations
is made up of people with the following characteristics:
- People of a wide age range
- People who have never encountered a computer that
asks them to speak a response
-
People who don't understand, at first or subsequently,
that they are talking to a computer
- People who don't speak English as their primary language
- People with varied educational levels
- People who have varied levels of comfort/anxiety
talking to a computer
How the Incaller portion of the system works
A person calls someone's Wildfire account and the
system asks for the incaller's name, phone number and then the
message - after which the incaller can re-record their message,
listen to it, etc. Here's an example of how the incaller aspect
of the system works. A person calls someone's Wildfire account
and hears (the italicized words are spoken by the incaller and
the unformatted words are spoken by Wildfire):
I'm sorry, "Chris Kotelly" is not available.
Please say your full name.
Mark Borenstine
Please enter your phone number followed by the pound
key so I can have him get back to you.
617-555-1212, #
Now leave your message, then hang up or press pound.
"Hey Chris, Give me a call..."
Please say SendIt or WhatAreMyOptions.
What are my options?
Say SendIt, RerecordIt, ThrowItAway, MarkItUrgent
or ReadItBackToMe
Send it
I'll see that gets to him. Goodbye.
Why we used the Web/Net to accumulate test subjects
We determined that we could not get access to enough
people by going "door-to-door" for this test - as the
deadline for testing was near and the time to administer the test
could be up to 25 or 30 minutes per session.
WE USED THE WEB/NET FOR:
- Getting participants for the test
Using links from other Web pages, and the net to
distribute messages to e-mail lists and newsgroups eliminated
the time needed to canvas people individually or collectively
in person.
- Distributing the information about the test
Information about the test was in the form of an
electronic message that told the potential user how they could
go about trying the test. This message eliminated the time for
the usability tester to explain the test to each person, each
time.
- Conducting the test
People could fill out pre-test and post-test questionnaires
on-line, either on an e-mail form or on our Usability Web site.
The participants did, however, need to use a phone to call the
800 number we set up for them to listen and interact with the
incaller interface. These questionnaires eliminated the time needed
to score the participants.
A Known Hazard
We knew that we would not get a good cross section
of the general population because the people we were canvassing
are all techno-literate. Also, since we targeted groups that are
usability aware we would not get "the-man-on-the-street"
perspective.
THE TEST ITSELF
Would-be participants found out about the test by
one of the said sources, and had the option of taking the test
by using the Web page or by filling out the questionnaire using
an electronic form that appeared on the e-mail/other electronic
messages.
Most participants (88%) went to the Web site, where
a greeting page welcomed them to the opportunity to test the new
interface or go to the main Wildfire page, lest they got to this
link accidentally.
The next Web page described what we were testing,
why we were testing it, and how we were testing it. Also on that
same page were the pre-test questionnaire instructions for calling
the 800 number (to try out the interface) and the post-test
questionnaire.
What we found out
One problem with the test was that the e-mail sent
out had the dates and time of the test on it, which caused a problem
when the testing time was delayed. Although this error was corrected
on the Web page, the e-mail had already reached some people and
could not be "traced" and corrected. Another problem
with email messages is that they get forwarded endlessly: as late
as 3 months after the end of the test, we still received some
responses.
Another unforeseen problem was created by the format
of the Web page. A person who was at home and only had one telephone
line, accessed the Web page, and began filling out the pre-test
questionnaire. At the end of the questionnaire, the person was
asked to call an 800 number to try out the new interface, which
required a one-phone-line user to hang up the Web connection (to
free up the phone line), dial up the 800 number to work with the
interface, then reconnect with the Web page, re-enter all the
data in the pre-test questionnaire again and remember their thoughts
from a few minutes ago when they were taking the test to fill-out
the post test questionnaire.
IDEAS/REDUCTION
To get people: The newsgroups we canvassed did not
attract many people but the 'listserv' groups were gold mines.
The Web page: We determined that the use of only
one scrollable web page was easy for people to work through and
did not require the user to progress through several pages which
might convey the feeling that the test is interminable. We wanted
to make sure that the participants understood the scope of the
test.
We did not think that many people might use one telephone
line for both voice and for Internet activities and did not design
the pages to accommodate this population.
But what if you want to test a GUI, not a VUI (Vocal User Interface)?
The Wildfire application lent itself well to remote
testing, but what if you want to test a GUI application which
people can't try out by simply using their telephone or other
common appliance? We looked at the issue of testing applications
using a GUI and have considered several routes which would enable
one to use the Web for a remote usability test of a GUI. The most
successful solution appears to be to use the Java language for
developing Java applets. A strong advantage of the Java environment
is that a C++ developer can easily transfer their knowledge to
get up to speed in a Java environment. What Java does, is to allow
a Web page designer/programmer to build a small, functioning program
directly into their Web pages. A great advantage of an applet
is it's ability to not only perform the function of the application
being tested, but also it's ability to gather performance metrics
such as task completion times, various choices made by the user
in specific orders, etc. If one were to incorporate a CUSEEME
sync up - one could also gather facial expressions (and construct
an electronic highlight tape) from people taking a usability test
anywhere and everywhere on the planet.
CHI 97 Electronic Publications: Late-Breaking/Short Talks
