|
|
Experience has shown that it is a challenge to apply these techniques in practice, causing us to ask what are the special problems associated with involving users in design? Many practical considerations can hinder or help the participation of users. Designers are faced with balancing conflicting demands, and with making harsh decisions necessitated by circumstances. Surveys of design practice such as [2] and [14] have offered some insight into the problems. These have ranged from the need to convince stakeholders of the importance of involving users, to obtaining the right kind of input from users once they are involved. Contractual and confidentiality concerns may prevent end-user involvement in the first place, while other problems, such as limited budgets, are encountered in situations where consent for user involvement has been granted in principle. Other researchers have described their experiences in specific design projects (e.g. [1]). Noyes et al [12] describe the involvement of users in the development of an aircraft warning system, and Poltrock and Grudin [13] describes obstacles to effective interface design in two participant observer studies that they conducted. For example, one obstacle they encountered was that responsibility for user interface design was distributed across different parts of the organisation.
There remains a need for further study of typical design situations from which we may learn how to tackle the problems of user involvement and the consequent poor usability of systems. In this paper, we focus on one aspect of user involvement in design and draw out a number of lessons for interactive system designers. We present a study of the factors that facilitated and hindered user participation in one design project, the so-called 'obstacles' and 'facilitators'. An obstacle is defined as a factor that prevents the user from making an effective contribution to design, and a facilitator is defined as a factor that enables the user to make an effective contribution to design. The intention of the study was not to assess the strengths and weaknesses of the design approach adopted on the project but, by focusing on obstacles and facilitators, to gain insights into how to enhance and support effective user involvement. The issue is not solely one of improving existing design techniques, nor of proposing new approaches, but is also one of the effective application of existing techniques in their organisational and social context.
The approach we adopted in the study is novel in several respects. We report and contrast the views of the various stakeholders in the design process, as well as offering our observations as non-participant observers. Furthermore, we analyse not just the obstacles to user involvement, as others have tended to do (e.g. [5]), but also the factors that helped to facilitate it. The analysis was based on a wide variety of data sources and covered various aspects of the development process, including organisational and social issues.
The design project was characterised by a high level of user involvement, with users actively involved in both analysis and design activities. While this might not be unusual within the CHI community, it contrasts with current design practice. Surveys such as that reported in [16] suggest that user involvement is largely confined to analysis and evaluation activities at present, with users tending to be passive rather than active participants. The results of our study offer interesting insights into the particular challenges of adopting a more collaborative approach to the design of interactive systems, and ways in which these may be addressed. In subsequent sections of this paper, we describe the set-up of the study, the data analysis and its results and draw out a number of lessons for design practitioners and researchers.
After an earlier, unsuccessful, in-house attempt to design this system, management made the decision to contract an external designer (the "application designer") to carry out the design and implementation. It was subsequently determined that HCI expertise would be valuable and a second designer (the "user interface designer") was also brought into the project. One member of staff (the "manager-user") was assigned responsibility for managing the project.
Our study commenced at the time the two designers started work on the project. At this stage, a few high-level decisions had been made concerning the anticipated scope of the system and the implementation environment. The original assumption had been that the designers would proceed by replicating existing paper forms in software form. However, under the direction of the user interface designer, the focus shifted to place greater emphasis on the design of the system and the development process was adjusted to incorporate more user involvement.
The project had an initial eight week period of intensive analysis and design activity (phase one), followed by a further 6 months of lower level implementation and evaluation activity (phase two). A first cut at the design was available at the end of phase one. Figure 1 gives an overview of the development activities and the people who were involved in them. Due to limited resources, the two designers were only employed during phase one of the project and led the design activities during this period. After this time, the manager-user assumed responsibility for completing the implementation (he had technical expertise). The project lost some momentum at this time, as the manager-user had to implement the system alongside his other responsibilities.
As mentioned above, the user interface designer was responsible for the design approach. After some initial scoping of the project with management, the designers carried out preliminary interviews with fourteen users. They then performed a detailed analysis of the users' existing work (the existing task model) with three users, followed by an analysis involving six users of how the work might be changed with the introduction of the new system (the envisioned task model). Both task modelling activities involved users and designers co-operatively constructing models of the tasks on a whiteboard. Design of the new system to support the envisioned tasks commenced with two users involved in paper prototyping sessions [9]. Phase one ended with the application designer producing some preliminary screen designs from the paper prototypes. In phase two, the manager-user completed the implementation and obtained some informal feedback on the system, and a usability evaluation was conducted.
A detailed analysis of the interviews with designers and users has been conducted in order to identify the obstacles and facilitators they perceived to user involvement. The results and conclusions presented in this paper are largely based on this analysis, with information from the video tapes, design artefacts and usability study used as supplementary evidence and background information to support our observations as non-participant observers.
In analysing the interviews, an obstacle was coded when the interviewees mentioned a factor which they perceived prevented the users from making a contribution to a design activity, or where they felt there would have been better or more user input if this factor did not exist. A facilitator was coded when the interviewees mentioned some factor which they perceived facilitated the users in making a contribution to a design activity, or where they felt there would have been worse or less user input if this factor did not exist.
Two interviews (one with a designer and one with a user) were selected for use in an initial analysis. Two researchers independently analysed and coded these interview transcripts for obstacles and facilitators in accordance with the definitions given above. The codings were compared and discussed to ensure consistency. The two researchers then independently coded the remaining user and designer interviews in the agreed manner. Following this, the researchers compared and discussed the coding of all the interview transcripts to ensure agreement on the obstacles and facilitators identified by the users and designers in the interviews. Detailed coding tables were produced. The obstacles and facilitators in the tables were grouped to remove duplicate entries. The data was later supplemented by additional obstacles identified by the researchers themselves.
The importance of this individual in facilitating user involvement was highlighted during phase two of the project, the implementation phase. His absence meant that there was no longer someone to facilitate user involvement. There was little user involvement during this period beyond some informal comments made by a couple of the users to the implementor. The structure within which users could offer their input became unclear. They felt their views were no longer taken into account, and one user commented that although he would like to see certain modifications to the system, he had no idea who to talk to about it.
While the designers promoted user involvement in the project as a whole, they made the decision not to involve users in the design of one sub-system. They argued that the task supported by this subsystem was simple and therefore user involvement was unnecessary. Interestingly, this resulted in this part of the system being marginalised during the design and implementation activities, and it has not yet been brought into use.
Selecting users
One crucial factor in this project, as in many others, was the time required to involve
users. The limited time for the first phase of the design meant that various decisions
were made which proved obstacles to the involvement of users. The designers
determined at the outset that they would only be able to involve a limited number of
the users, and therefore they made a selection under the guidance of the manager-
user.
They initially interviewed fourteen users selected from each of the major work areas in the department. Many of these users were relatively senior in the department and were chosen because of their perceived knowledge and experience. Three of these initial interviewees were then invited to contribute to the subsequent modelling and design activities. These three users represented three of the main work areas in the department and, in addition to their perceived knowledge, seniority and experience, were motivated and keen to be involved in the project. (The importance of user motivation is elaborated further in the following theme).
The designers felt that they had made a good selection in these three users, and they saw it as being in their own best interests to select those who knew most. However, it emerged from the user interviews and usability evaluation that the needs of other users had been neglected. Most strikingly, the needs of junior, part-time or temporary staff had been disregarded, but so had those of the more senior managers. The user interviews revealed that some of these part-time staff can now see no good reason to use the resulting system Ñ they cited reasons such as the system offers them no help in doing their work and adds to their workload. The designers made a practical decision, motivated by circumstances, to focus on the high-level knowledge of some users who knew about several jobs, thus preventing, or at best not facilitating, the involvement of users who did the jobs on a day-to-day basis.
| O/ F | Summary of comment(s) | View | |
|---|---|---|---|
| F | UI designer championed cause for user involvement | R | |
| F | UI designer was highly motivated to involve users | D | |
| F | UI designer convinced management of the value of user involvement in design | D | |
| F | Management receptive to UI designer�s ideas | D | |
| O | Absence of the UI designer in phase two resulted in little user involvement | R | |
| O | Structure within which users could offer comments became unclear in second phase | U | |
| O | Users felt their views were not taken into account in second phase of project | U | |
| O | One user did not know to whom he should give his comments on the system | U | |
| O | The designers decided not to involve users in the design of one subsystem | D | |
| O | There was limited time for the project, and thus also to involve the users | D | |
| O | The designers decided that they would only involve a limited set of users | D | |
| F | Designers made the selection of users with the help of the manager-user | D | |
| F | Users were chosen because of perceived knowledge and experience | D | |
| F | Three users were selected to participate in subsequent modelling and design activities | D | |
| F | Users were keen to be involved in design | U | |
| O | The needs of other users were disregarded | R |
The manager-user explained the structure of the department to the designers and outlined the jobs and tasks of individual users. The designers found this useful and it provided an initial starting point for involving certain of the users. Beyond this, it was the designers' responsibility to contact the users and arrange meetings. None of the users were explicitly told that it was okay to take time away from their normal work to attend design meetings or to read design documents. Neither management nor the designers appeared to appreciate that they might need to do more than simply agree that the users could be involved. This problem was compounded by the fact that there was poor dissemination of information about the design activity in the department. Senior staff were aware of the design project, but junior staff had little idea of what was happening and no idea that it might be to their advantage to contribute. This added to the problems of selecting users outlined above.
The designers saw the fact that the users were very busy as an obstacle to their involvement. It was difficult to make appointments. Users would fail to show up, would cancel appointments at the last minute or would get bleeped in the middle of a meeting. One user commented that he was surprised at the difficulties the designers encountered in persuading the users to attend meetings and talk to them. He thought one problem was that the designers expected the users to come to them, rather than the designers going to the users. However, the designers said they found it difficult to talk to the users in their own working environment Ñ there were constant interruptions from telephone calls, customers or colleagues.
The designers tried to organise some of their access to users via email. This posed various problems. In particular, general messages to all the staff requesting input elicited virtually no response which, in turn, irritated the designers. Various meetings were badly organised, with the designers failing to organise or specify a room for the meeting, and the users unsure where they should be. This resulted in some meetings starting off on the wrong foot.
Another issue which arose was the proximity of the designers to the users (see also [7] for a discussion on how distance adversely affects frequency and quality of communication). As the designers were based in the users' working environment, it was easy for users to approach them on an informal basis thus facilitating additional user input. Several of the users who were involved did this on a regular basis. However, there was also a negative aspect to the proximity. The designers sometimes felt that the users were interfering or checking on their progress and, as a consequence, tried to avoid the users on occasion.
Users' attitude and motivation
It is clear from the analysis that the attitude and motivation of the users were crucial
factors influencing their involvement in the design activities. In a sense, those who
were highly motivated helped to facilitate their own involvement Ñ the designers
were eager to involve those who displayed interest. Therefore, the reasons
underlying the users' motivation, or lack thereof, are of particular interest, causing us
to ask whether designers could do more to draw in the less motivated or more
reticent users.
The users who were involved cited several reasons underlying their eagerness to participate. One user was motivated by prior experience of situations where he had not been asked for his views, and did not appreciate the consequences. Another motivation was more political: there was a history of conflict between different sections and users wanted to defend their own position. There were also a number of individual motives: one user wanted to ensure that the system incorporated functionality which would help in his job, while the manager-user was motivated because he had overall responsibility for the project.
Of the users who had less involvement in the project, one said that he had not seen it as being relevant to his work. Others claimed to be unaware that there might even have been an opportunity to contribute, or felt that they had nothing to contribute. Some users appeared to lack confidence saying that, as they came from a non- technical background, they would have nothing useful to say and were reluctant to talk to the designers. The users who put more effort into the project said they felt that they got more out of it in return. They were interested and keen to be involved further. They volunteered additional information and helped to facilitate the involvement of others. One user, in particular, distributed information to others in his section and encouraged them to attend meetings and put forward their views about the proposed system.
| O/ F | Summary of comment(s) | View | |
|---|---|---|---|
| F | Management gave consent to involve users | D | |
| F | Users were willing to talk to designers | D | |
| F | Manager-user explained structure of dept | D | |
| O | Users not told it was okay to take time to be involved in design | R | |
| O | Management and designers put no extra effort into convincing users to be involved | R | |
| O | There was poor dissemination of information about design project within department | U | |
| O | Junior staff were unaware of the project | U | |
| O | Users were very busy | D | |
| O | It was difficult to make appointments with users | D | |
| O | Designers expected users to come to them | U | |
| O | Designers found it difficult to talk to users in their working environment | D | |
| O | Users did not respond to email messages | D | |
| O | Meetings were badly organised | D/U | |
| O | Meetings started off on the wrong foot | D | |
| F | Designers were located close to the users making contact easy | R | |
| O | Designers felt the users were checking on their progress | D | |
| F | Designers eager to involve motivated users | D | |
| O | Designers reluctant to involve less motivated users | R | |
| F | Users were motivated because of previous experiences and politics | D | |
| O | User felt system was irrelevant to his work | U | |
| O | Users were unaware of opportunity to be involved | U | |
| O | Users lacked confidence and were reluctant to talk to the designers | U | |
| F | Users that were involved became more motivated and volunteered extra input | D |
Meeting facilitation
Effective facilitation of design meetings is another critical factor in supporting user
involvement in design. The designers in this study employed various tactics to try to
make the users feel at ease in design meetings and to show that their opinions were
valued. Their initial discussions with the manager-user ensured that they came well
prepared to the meetings, with information about the users' names, roles and tasks.
One user mentioned it was useful that the designers were not judgmental, and that
the meetings were not intimidating or imposing. Furthermore, it was clear to him
that his input was treated as confidential, a fact which was especially important to
him because of departmental politics.
One perceived shortcoming of the meetings was that users tended to agree with the designers too quickly, simply because they were the designers. There was a suggestion that the user interface designer might have been leading the users. Another user said he was hindered by the fact that he had no idea of the design constraints of the implementation environment, and suggested that it would have been useful to have the implementation system at hand to understand the design possibilities. The attitude of the mediator, usually the user interface designer, in meetings was important. In one meeting the user interface designer did not mediate effectively because he was under pressure from concerns external to the project. The consequence was that the meeting failed to accomplish what it had set out to achieve and all participants expressed irritation at the way things had gone.
When initially exploring how users' work might be changed with the new system, designers first had meetings with each user individually, before bringing them together to combine the different views into the one envisioned task model. In this way they tried to ensure that all users could express their views. One user mentioned this worked well, because apart from being able to give his opinion, it allowed him to clarify his thoughts about what he wanted before he went into the group meeting. This user also said it was clear that some users, who had not been involved before the group meeting, had more trouble voicing their ideas. However, the group meeting did pose certain problems. One user said he felt uncomfortable giving his opinion because of the reaction of other users in the meeting. Furthermore, one user thought it was difficult for his section to contribute their views, because users from another section had attended the meeting in greater numbers. The designers said that conflicts between the user groups were brought out into the open, which at first hindered user input into the design process, but some of the conflicts were eventually negotiated and resolved, leading to useful design ideas.
| O/ F | Summary of comment(s) | View | F | Easier for users to contribute if they are involved throughout the process | D | F | Designers came prepared to meetings | D | F | Designers were not judgmental | U | F | Meetings were not intimidating | U | F | Input was treated as confidential | U | O | Users agreed too quickly with designers | D | O | UI designers might have led the users | D | O | Users were unaware of implementation constraints during task model activities | U | O | UI designer didn't mediate one meeting effectively | D/U | F | Individual meetings first with users allowed them to give their opinion openly | D/U | F | Group meetings with users facilitated reaching agreement | D | O | Users from one group attended a meeting in larger numbers than another user group | U | O | Conflicts were brought out in the open | D | F | Users asked about their area of expertise | D | F | Designers chose expert users to go first | D | F | Design representations acted as focus for communication | D | F | Users came up with ideas for notations | D | F | Whiteboard provided a useful focus | D/U | F | Some users were active during meetings | D | O | Some users were passive during meetings | D | O | One user wanted to work at his own pace | D | F | A hard copy of task model used to get input from more users | D | O | One user had a negative attitude towards paper prototyping | D/U | O | Hard to judge interaction issues with paper prototypes | D/U | F | First user negotiated task model notation | D | O | Subsequent users had to accept the notation | R | F | Some users found the notation useful | U | O | Some users found the notation confusing | U | O | Users did not always have enough time to assimilate and understand the models | D | O | The notation did not capture all aspects of the task | U | O | Some users misunderstood the notation | D |
|---|
Focus points
In order to facilitate input to the design process, particularly the task modelling
activities, the designers asked the users specific questions about their own area of
expertise (their task and role in the department). They selected the first user for the
task modelling carefully, choosing an individual who had wide ranging knowledge
about different roles and jobs in the department. The users were also asked to
comment on the information already received about other work areas. The designers
used various design media, such as a whiteboard and post-it notes, as a focus for
communication in design meetings. Further focus points were provided by giving
the users access to design artefacts that had previously been produced. For example,
the users that were involved in the paper prototyping sessions had access to the task
models as a reminder of previous design decisions. In some cases, the users
themselves came up with other solutions that would help to focus their attention and
increase their input. For example, one user came prepared with a check-list to the
paper prototyping session, and another asked for a hard copy of the task model, so
he could take it home with him and think about it some more.
Design representations, media and tools
A number of design representations were used at different times to capture design
information and to facilitate user input. The user interface designer anticipated that a
whiteboard (on which the task models were constructed) would be useful medium,
because it would provide an easy focus, and would allow users to cooperate in
constructing and modifying the task models.
Some users were very active at the whiteboard and during the paper prototyping sessions, whereas others were more passive in their participation. One user explained that he preferred to take a copy home with him, so he could work on it at his own pace. This further inspired the user to suggest that this hard copy could also be distributed to other users to get more feedback.
The designers felt that the paper prototypes could be constructed rapidly and were easy to change. One user commented that they were useful to get a feeling for the screen designs. Another user that was involved in the paper prototyping had a negative attitude to the paper prototypes, and therefore contributed much less to the session. This user preferred working with software prototypes, and said it was too hard to judge interaction just using a paper-based representation of the user interface.
The notation used to describe the task model was negotiated with the first user who participated in the task modelling activity. As a consequence, subsequent users had to accept this notation as a given. Some found the notation a useful way of visualising the problem, whereas others found it confusing. According to the application designer, some users did not really understand the model, possibly because it was rather large and required some time to assimilate and understand. The users were not always given sufficient time to do this at the beginning of design meetings. One user felt that the task model notation did not capture all aspects of the task. There was also evidence that some users misunderstood aspects of the design representations, making incorrect assumptions that had to be corrected later.
Motivate all stakeholders
All stakeholders (users, managers, designers) need to be made aware of the potential
benefits of user involvement and motivated to involve users. We need to appreciate
that motives vary from one individual to another, so that an argument to convince
management may not carry much weight with the users.
Select a representative cross-section of users
Although it sounds obvious, it is worth emphasising the importance of selecting a
truly representative cross-section of users when some selection has to be made. This
means selecting not just users from different work areas, or those who appear to
know the most, but people of varying levels of seniority, expertise and service
conditions. This is complicated by the fact that designers may not have completely
understood the organisational context at the point in time when a selection has to be
made.
Involve a champion for the cause of user involvement
As was evident in this study, a champion of user involvement can play a major role
in influencing the design process, motivating people and organising the design
activities accordingly.
Organise meetings effectively
It is important to be clear not just about the time and location of meetings, but also
about their purpose. Knowing the purpose can help motivate the users to attend.
Poor organisation results in meetings either failing to happen or starting off in a bad
atmosphere. It is better to contact the users individually, preferably in person or by
phone, when arranging the meetings.
Ensure active management buy-in
It is not sufficient for management to simply agree to user involvement. Management
should promote the importance of user involvement, making sure that users are
aware of what is happening and can take time out of their normal work to contribute.
Don't expect the users to be designers
Different expertise should be valued, with each participant contributing their special
knowledge to the design process. In particular, it is unrealistic to expect users to have
design skills. Designers should be largely responsible for contributing design
expertise, while users can contribute information about their work and organisation.
Follow user involvement through to the end
User involvement lost momentum in this case study, and therefore some of the initial
benefits were lost along the way. Users need to see that their contributions have been
noted and, where appropriate, have influenced the design.
Be flexible
No design technique is going to be suitable for all participants in any design context.
It should be possible to adjust the approach to accommodate individual differences
and preferences of the users (e.g. skills, work method).
Facilitate later involvement through earlier involvement
It is important not to exclude users too soon by selecting a subset to be involved in
the design activities. Further, the inclusion of Ônew� users at later stages should be
facilitated through appropriate education.
Educate users about the whole design process
Users should be educated about what will happen in the process as a whole, which
decisions will be made when and what the consequences of these decisions may be.
Organise both individual and group meetings
Users should be given the opportunity to voice their opinions in individual meetings
where their views may be aired openly, but it is also useful to resolve differences of
opinion with other users through group meetings.
Based on this and other studies (e.g. [5]), it is becoming clear that involving users in an effective way in design is a complex problem. While some of our results might appear obvious at first sight, they point to the fact that the problem is not just a matter of developing the appropriate design techniques. Rather, it involves working on a wider change in design practice, where the design process, the social process of design and the way design is managed within organisations have to be studied as a whole. The results of the study reported here relate to all these different aspects and show the importance of understanding more about different perspectives (designers, users, organisations) in order to bring about this change. In this study, the user interface designer tried to utilise various techniques to facilitate user-designer collaboration, some of which were more successful than others. Although basic hurdles to user involvement, such as management consent, were largely overcome, there were still other factors, such as user motivation, that prevented users from contributing effectively.
The results of the study also paint a rich picture of the complex trade-offs that have to be considered. For example, in selecting a subset of users because of time or budget constraints, it is hard to ensure that a representative set is chosen. Furthermore, individual differences between users make it difficult to choose and use design techniques and representations that are optimum for all users and lead to useful input to the design process. Compromises have to be made in considering wide ranging issues such as how users should be contacted, which users should be selected to participate, how to keep the users motivated and what kind of techniques and design representations are best suited to the design context.
This study has examined the obstacles and facilitators associated with user involvement in a specific design context: the development of a bespoke system. It is to be expected that other factors will prevail in different situations. For example, in a product development context additional problems arise because the users are often unknown, or inaccessible, at the outset, or because commercial advantage requires confidentiality. While it is easier to know, to contact and to select the users in a custom-built development context [6], it is problematic to ensure that any selection of users is representative until the designers have acquired a thorough understanding of the design problem and organisational context. In this study they did not appreciate certain differences in users' jobs and roles until the end of the project.
The results presented in this paper are qualitative in nature, relying largely on the perceptions of the users and designers. These are of interest in themselves, in that they reflect the contrasting concerns of those involved in a design project and these concerns need to be addressed to facilitate effective user participation. The findings will be counterbalanced by our current work in which we are conducting further analyses of the data to determine how the users' contributions were actually incorporated into the design, and how effective these contributions were in terms of their impact on the usability of the implemented system.
The work reported here is novel in two main respects. Firstly, unlike previous studies, we have focused on both the obstacles and the facilitators to user involvement, allowing us to highlight both the strengths and weaknesses of design techniques in this respect. Secondly, we compare and contrast the views of the designers, users and ourselves, as observers, as to the obstacles and facilitators they perceive. The results indicate that a careful examination of the relations and trade- offs between these factors is necessary to fully understand how user involvement might be enhanced in the future so as to contribute to the design of effective and usable systems.
|
|
