CHI 97 Electronic Publications: Demonstrations
Using Music as a Communication Medium
James L. Alty
Department of Computer Science Loughborough
University
Loughborough
Leicestershire, UK, LE11 3TU
+1509 222
648
j.l.alty@lboro.ac.uk
Dimitrios Rigas
Department of Computer Science Loughborough
University
Loughborough
Leicestershire, UK, LE11 3TU
+1509 222
648
d.rigas@lboro.ac.uk
Paul Vickers
School of Computing and Mathematical
Sciences
Liverpool John Moores University
Liverpool, L3 3AF, UK
+151
231 2104
p.vickers@livjm.ac uk
ABSTRACT
Music is a rich communication medium, and there are some
similarities between the job of a music composer and that of an HCI designer
(although their objectives may be different). Whilst sound has been used in
interfaces, its use has mainly been at a primitive level, often involving
real-world sound. Since music offers a highly structured set of mechanisms for
communicating, it is surprising that there have been so few attempts at
exploring its possibilities. Our current activity involves investigations into
the use of music in algorithmic audiolisation and program debugging.
Keywords
Music, Interface Design, Debugging, Multi-media, Audiolisation.
© 1997 Copyright on this material is held by the authors.
INTRODUCTION
Most output activity in HCI research has concentrated on the visual
medium, and many investigations involving a large number of output media have
been presented. The auditory medium, on the other hand, has been exploited
only to a limited extent. Since Bly's original thesis [1] on the use of sound
in interfaces, attempts have been made to design and build the aural equivalent
of visual icons, or "earcons" [2][3], voice output and artificial or natural
sounds are now used fairly routinely, and systems such as the SonicFinder [4],
which uses natural sound to indicate the state of the natural environment, have
been described The auditory channel has also been used to augment the visual
channel in algorithm visualisation [5] and to assist in physical process
audiolisation [6]. However, the potential for using music as an output medium
(the most sophisticated of the auditory media) has hardly been examined at all
apart from some use of music in data analysis [7]. For a full discussion of
the possibilities offered by music in HCI see [8].
THE IMPORTANCE OF AUDIO IN INTERFACE DESIGN
The Audio channel is an important information channel, and has a long
established tradition of conveying rich meanings. However, there is another
important reason for encouraging more research in the area. The explosive
growth in visual interfaces (such as GUIs) has put visually- challenged people
at a considerable disadvantage. Most interfaces now carry an implicit
assumption that the recipient has excellent visual capabilities. Such a
situation is particularly upsetting because blind and partially-sighted users
were, until recently, making great strides forward in using computers
successfully, (for example, using screen readers). Unfortunately, screen
readers cannot successfully describe GUIs. In an ideal world one would
envisage an interface design approach which could be accessed by audio alone,
by viewing alone, or through a combination of the two (and all under user
control).
Another related point is the current overcrowding of visual interfaces. Audio
could have a real use here in supplementing the visual interface in a
complementary manner.
MAPPING BETWEEN DOMAIN AND MUSIC
The key issue in using music in interface design is how to map between
domain entities and procedures, and equivalent musical structures. To
investigate this we have carried out experiments on using pitch, timbre, rhythm
and melodies. Users competence at recognizing pitch intervals, and different
timbres has been determined. The results suggest that, provided precise
numerical relationships are not being communicated, music can transfer
information successfully. We have been particularly concerned with the mapping
for subjects who have had no special musical training.
USING MUSIC FOR ALGORITHM AURALISATION
We have implemented a number of sorting algorithms and path following
algorithms using music alone (this distinguishes the work from that of Brown
and Herchberger [5] who supplemented the visual output with sound). We have
also measured how successfully the algorithm activity is communicated via the
music
USING MUSIC IN PROGRAM OR ALGORITHM DEBUGGING
The use of music in algorithm auralisation also suggests another
possible use - that of program debugging. In a sense the application is the
same The algorithm being auralised is simply faulty.
The possibility of using musical output for debugging has already been
suggested by Francioni [9] for debugging a parallel processor message system.
They point out that the musical representation can highlight situations which
could easily be missed in the visual representation (no doubt there are also
cases in the reverse direction). For example, a move of one semitone of a note
in a musical chord can change the whole sense of that chord and produce an
immediate and compelling effect. A similar movement in the value of one data
variable in a graph might not be noticed. Bock [11] has developed an Auditory
Domain Specification language which has been used in debugging.
In the programming language debugging situation, the richness provided by a
musical representation can offer fairly precise bug location possibilities
(whether used in isolation or in conjunction with the visual media). One
obvious possible mapping is to map the tracing of the execution path through
different modules to different instruments. This is an area which is not
handled well by visual media, causing frequent screen shifts. Using timbre, the
switches between modules are capable of being followed easily, allowing the
visual sense to concentrate upon program detail.
We believe that music can also be used for language debugging. We have
constructed a PASCAL preprocessor (CAITLIN - Computer Audio Interface to Locate
Incorrect Nonesense [10] ) which maps PASCAL constructs to particular musical
structures. A FOR-NEXT loop for example is mapped to a rising pitch based upon
the loop variable. REPEAT-UNTIL and REPEAT-WHILE loops are distinguished by a
simple comparison at either the start or the end of the loop. An IF THEN
clause is mapped to distinctive timbre and pitch so that the subsequent path
can be clearly identified.
Early experimentation with the system has shown that users can distinguish the
different constructs.
COMMUNICATING GRAPHICAL INFORMATION TO BLIND USERS
The third area in which we have investigated possible uses of music is
in communicating graphical information to blind users. The graphical area is
defined in terms of coordinated which are communicated via sequences of
ascending pitch. All graphcal objects are communicated via musical sequences
which reflect their shape, and all the graphical controls are musical in
nature.
The system, called the AUDIOGRAPH, has been developed from feedback from a
number of blind users and the initial results are encouraging.
REFERENCES
[1] Bly,S., (1982), Presenting Information in sound. Proc. CHI'82 on
Human Factors in Computer Systems, pp 371 - 375
[2] Gaver, W., (1986), Auditory Icons: Using Sound in Computer Interfaces,
Human Computer Interaction, Vol. 2(1)., pp. 167 - 177.
[3] Blattner M., Greenberg, R.M., and Kamegai, M., (1992), Listening to
Turbulence: an Example of Scientific Audiolisation, in Multimedia Interface
Design, (Blattner, M., and Dannenberg, R.M., eds.) , Chapter 6, pp. 87 -
102, Wokingham: ACM Press.
[4] Gaver, W., (1989), The Sonic finder: an interface that uses Auditory
Icons, Human Computer Interaction, Vol. 4(1), No. 1, pp. 67 - 94.
[5] Brown, M.H., and Herchberger, J., (1992), Colour and Sound in Algorithm
Animation, IEEE Computer, December, 1992, pp. 52 - 63.
[6] Mansur, D.L., Blattner, MM., and Joy, K.I., (1985), Soundgraphs: A
Numerical Data Analysis method for the Blind, Proc. 18th Hawaii Conf. on
System Sciences, Vol. 18, pp 163 - 174.
[7] Mezrich J.J., Frysinger, S., and Slivjanovski, R., (1984), Dynamic
Representation of Multivariate Time Series Data, J. Am. Statistical
Soc., Vol. 79, No. 385, pp. 5.22 - 5.28.
[8] Alty, J.L., (1995), Can we use Music in Human computer interaction ? in
People and Computers X, Diaper, D., and Winder, R., (eds.), Cambridge
University Press. pp 409 - 423.
[9] Francioni, J.M., Albright, L., and Jackson, J.A., (1991), Debugging
Parallel Programs Using Sound, SIGPLAN Notices, Vol. 26 (12), pp. 68 -
75.
[10] Alty, J.L.and Vickers, P., (1996), CAITLIN: A Musical Program
Auralisation Tool to assist Novice Programmers with Debugging, to appear in
Proceedings.ICAD'96.
[11] Bock, D.S., (1994), An Auditory Domain Specification Language for Program
Auralisation", 2nd Int. Conf. on Auditory Display, ICAD'94, Santa Fe, New
Mexico, USA.
CHI 97 Electronic Publications: Demonstrations
