(PDF) Effects-Driven IT Development: A Strategy for Sustained Participatory Design and Implementation

Roskilde University Effects-Driven IT Development A Strategy for Sustained Participatory Design and Implementation Hertzum, Morten; Simonsen, Jesper Published in: Proceedings of the 11th Biennial Conference on Participatory Design: Participation – the challenge Publication date: 2010 Document Version Early version, also known as pre-print Citation for published version (APA): Hertzum, M., & Simonsen, J. (2010). Effects-Driven IT Development: A Strategy for Sustained Participatory Design and Implementation. In K. Bødker, T. Bratteteig, D. Loi, & T. Robertson (Eds.), Proceedings of the 11th Biennial Conference on Participatory Design: Participation – the challenge (pp. 61-70). Association for Computing Machinery. http://www.pdc2010.org/ General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain. • You may freely distribute the URL identifying the publication in the public portal. Take down policy If you believe that this document breaches copyright please contact

providing details, and we will remove access to the work immediately and investigate your claim. Download date: 10. Dec. 2021 Effects-Driven IT Development: An Instrument for Supporting Sustained Participatory Design Morten Hertzum Computer Science/User-Driven IT Innovation Roskilde University, Denmark University St. 1, Bldg. 43-2, 4000 Roskilde

Jesper Simonsen Computer Science/User-Driven IT Innovation Roskilde University, Denmark University St. 1, Bldg. 43-2, 4000 Roskilde

change, and (c) embrace an overall technology-driven organizational change process (Simonsen & Hertzum, 2008). ABSTRACT We present effects-driven IT development as an instrument for pursuing and reinforcing Participatory Design (PD) when it is applied in commercial information technology (IT) projects. Effects-driven IT development supports the management of a sustained PD process throughout design and organizational implementation. The focus is on the effects to be achieved by users through their adoption and use of a system. The overall idea is to (a) specify the purpose of a system as effects that are both measurable and meaningful to the users, and (b) evaluate the absence or presence of these effects during real use of the system. Effects are formulated in a user-oriented terminology, and they can be evaluated and revised with users in an iterative and incremental systems-development process that involves pilot implementations. In this paper we investigate the design, pilot implementation, and effects assessment of an electronic patient record. Effects concerning, among other things, clinicians’ mental workload were specified and measured, but apart from the planned changes associated with these effects the pilot implementation also gave rise to emergent, opportunitybased, and curtailed changes. We discuss our experiences regarding conditions for making the specification of effects and their real-use evaluation central activities in IT projects. In this paper, we present effects-driven IT development, which is an instrument supporting a proactive strategy for sustained participatory design and implementation of large IT projects aiming at major changes in work practices and work organization. We use the term ‘instrument’ to emphasize that this is not a coherent method on its own but rather an approach that can be applied to supplement and enhance existing systems-development methods. Effects-driven IT development entails a sustained focus on the effects to be achieved by users through their adoption and use of a system. Effects may be about any aspect of the match between system and organization, including aspects such as effectiveness, efficiency, and satisfaction (ISO 9241, 1998) but also usefulness, which is often crucial to the integration of a system into organizational work practices. The overall idea is that specification and formative evaluation of the effects desired from a system will provide users and developers – customer and vendor – with a means for working systematically with the design and organizational implementation of the system. Our research on effects-driven IT development was initiated in 2004 and currently involves the authors and five Ph.D.s. Our collaboration includes two vendors and three out of five healthcare regions in Denmark. Keywords Effects-driven IT development, sustained participatory design, organizational implementation, effects specification, prototyping, pilot implementation, formative evaluation. An organization’s investment in new IT derives from a desire for organizational change. The process of designing and implementing IT should therefore be conducted with a focus on achieving the desired change, i.e. combining the IT project with organizational-change programs as described by Markus (2004). We distinguish between four types of technology-driven organizational change, the former three originally proposed by Orlikowski and Hofman (1997): planned or anticipated, emergent, opportunity-based, and curtailed change. Planned change denotes the desired change that is planned ahead and occurs as intended during implementation. It is, however, impossible to plan and predict all changes that occur when introducing new IT in a work context. Work is situated (Suchman, 2007) in the sense that the course of the work process depends on the material and social circumstances at hand. Thus “[u]nanticipated use of computer artefacts reflects the fact that work itself is undetermined until realized in situ” (Robinson, 1993, p.189). Unanticipated change can be divided into emergent and opportunitybased change (Orlikowski & Hofman, 1997). Emergent change is defined as local and spontaneous change, not originally anticipated nor intended. Such change does not INTRODUCTION Lately, the PD community has been encouraged to devise PD strategies that can be applied throughout design and organizational implementation. PD should extend the iterative approach (Simonsen & Hertzum, 2008), engage in commercial, large-scale information-systems development (Shapiro, 2005), and embrace implementation as ‘co-realization’ and ‘design-in-use’ of new IT (Hartswood et al., 2008). New PD strategies need to (a) include fully integrated systems exposed to real work practices, (b) evaluate planned as well as unanticipated Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. PDC’10, 29-NOV-2010, Sydney, Australia. Copyright 2010 ACM 978-1-4503-0131-2/10/0011…$10.00. 61 involve deliberate actions but grows out of practice. Opportunity-based change is purposefully introduced to take advantage of unexpected opportunities, events, or breakdowns that occur after the introduction of a new information system. Finally, we supplement Orlikowski and Hofman (1997) by including curtailed change to emphasize that change processes may fail to produce the intended effects. of IT projects: The benefits specification aims at informing the IT project, but the assessment of the benefits is mainly seen as input for future projects (Ward & Daniel, 2006, pp. 113-118) – i.e. benefits management involves summative evaluations (Harlen & James, 1997). Effectsdriven IT development involves frequent evaluations of the effects obtained from using mature prototypes implemented and tested in real use. This way, effects-driven IT development aims to support an iterative development approach where the assessment of effects during pilot use is fed back into ongoing design and implementation activities. Thus, the evaluations are formative (Harlen & James, 1997). While both management and end-users take part in the specification of effects, the measurement and assessment of the effects focus on the end-users’ experiences when using the system in their work. In this paper, we explore, refine, and describe effectsdriven IT development based on an empirical study in which the clinical-process module of an electronic patient record (EPR) system was iteratively designed and pilot implemented at a hospital unit as part of the activities involved in the project tender and bid for a large EPR contract of great importance to both customer and vendor. A clinical-process module supports clinical documentation and decision making and comprises the clinicians’ continual documentation of their observations, treatment, and care of the patients. If any type of change is to result from the introduction of an IT system, then the IT system must be implemented and used. Embracing and evaluating design and implementation introduce two levels of iterative processes (see Figure 1): iterative prototyping and what we – in order to distinguish it from customary final implementations – denote pilot implementations. Iterative prototyping is the process of creating, in advance of the implementation, a working model (the prototype) that exhibits essential features of the final system and using this prototype to test aspects of the design, illustrate ideas or features, and gather early feedback. Pilot implementation involves using and evaluating a more mature but still unfinished pilot system (Rzevski, 1984) in a restricted manner. During the pilot period the system is used in its intended work environment and using real data. The evaluation is formative in the sense that its results are to inform the subsequent design and implementation of the system (Berg et al., 2003; Granlien & Hertzum, 2009; Hamilton & Chervany, 1981). The empirical study evaluated a fully functional EPR module with complete patient records. The study had a tight timeframe, which is a prerequisite for using effects as an active means of managing IT projects, and involved the use of the EPR module for actual clinical work, which is necessary to become aware of relevant effects and to evaluate them. In the following, we first describe effectsdriven IT development. Then, we describe the method of our empirical work and the stroke unit at which it took place. Our results comprise examples of effects in relation to the four types of change process. We discuss our experiences and the conditions for effects-driven IT development and conclude by outlining some implications. EFFECTS-DRIVEN IT DEVELOPMENT Effects-driven IT development attempts to provide a sustained focus on the effects to be achieved by users through their adoption and use of a system. Simply put, the overall idea is to capture the purpose of a system in terms of effects that are both measurable and meaningful to the users, and to systematically evaluate whether these effects are attained during real use of the system. A sustained focus on effects accentuates that the functionality of a system is merely a means to an end, but it also entails that effects must not only be specified but also evaluated in the course of the development process. That is, effectsdriven IT development blurs the distinction between design and organizational implementation – between design and use. This focus is summarized in our definition of effects, adopted from Ottersten and Balic (2007): Effects = system quality × system adoption Figure 1. Effects-driven IT development. Pilot implementation constitutes a formative evaluation of planned, emergent, opportunity-based, or curtailed technology-driven organizational change. Effects-driven IT development may be compared to benefits management (Ward & Daniel, 2006). Benefits management is recognized as an instrument that supports a focus on deriving business benefit from IT projects. The idea is to (a) specify the IT project’s initial investment objectives and (b) refine these objectives into benefits, changes, and the needed IT functionality. There are, however, important differences between effects-driven IT development and benefits management. Benefits management focuses mainly on benefits specification and pays less attention to the organization of the subsequent parts The sustained participatory design process outlined in Figure 1 is adopted from Simonsen and Hertzum (2008) and emphasizes the evaluation of IT systems through exposing them to real work. The starting point is the planned and intended changes. Planned changes are specified, in terms of desired effects expected to manifest from using the system. A pilot of the system is then implemented and tried out under conditions as close as possible to real use – a process sometimes referred to as a 62 pilot study or pilot implementation (Glass, 1997; Rzevski, 1984; Turner, 2005). Actual use of the system allows for measuring the planned effects, for the emergent and opportunity-based changes to occur, and for the identification of curtailed effects. Our empirical study investigates whether and how these two critical activities can be performed. At the same time, the two critical activities capture how a sustained focus on effects adds to related approaches in PD and usercentred design (UCD). PD and UCD techniques such as diagnostic mapping, future workshops, mock-ups, and exploratory prototyping focus mostly on the early stages of technical implementation and do not involve evaluation of whether identified user needs are subsequently satisfied by the developed system. Usability evaluation, a widespread UCD technique, is commonly performed on set tasks and test data and with a focus on usability problems rather than usage effects. A UCD technique particularly related to effects-driven IT development is usability specifications (Good et al., 1986; Whiteside et al., 1988). A usability specification gives the worst, planned, best, and present levels of user performance for a specified set of tasks. In giving values defining the different levels of performance, usability specifications specify a set of effects and provide for a process alternating between design and evaluation until the effects have been attained. For the rather narrowly scoped tasks mostly associated with usability specifications it has not been considered a problem to obtain precise performance measurements, but for usage effects that involve the establishment of new organizational procedures, collaborative practices, and individual competences reliable measures are difficult to obtain (Hamilton & Chervany, 1981). Concrete examples of planned effects may include (see also Table 1 below): (a) The physician can complete the medical ward round without an escorting nurse, thereby making the clinical work more cost effective. (b) A reduction in clinicians’ mental workload at the daily team conference, thereby reducing the risk of errors in their assessments of patient status. Effects will often form a hierarchy where higher-level effects specify why effects at lower levels are desirable and lower-level effects specify how effects at higher levels can be attained. For example, national healthcare policies may state political effects, which influence individual hospitals’ choice of strategic effects, which in turn are reflected in effects directly concerning different aspects of the clinical work. While these examples relate to healthcare (the domain of our research program), the idea of effects-driven IT development is generally applicable to IT projects. The primary focus of effects-driven IT development will typically be on direct effects on the users’ work. System success is critically dependent on the users’ participation, support of, and attitude toward the system and thereby on whether they agree with the sought-for effects and can relate them to their work. Also, the effects on the users’ work can be specified most precisely, whereas effects at political and strategic levels are more indirect and thereby subject to additional sources of ambiguity. Specification of effects has also been suggested by Leveson (2000) but as an analytic device; that is, without measuring whether the specified effects are actually achieved. Conversely, feedback from users based on their actual use of (parts of) a system is available in incremental development and delivery (Sommerville, 2004; Steinberg & Palmer, 2004). However, incremental development and delivery does not involve specification and measurement of usage effects as a means of systematically evaluating whether a system provides desired effects. An exception is results-driven incrementalism (Fichman & Moses, 1999), which has a lot in common with effects-driven IT development. Finally, our work on effects-driven IT development has been inspired by performance-based procurement (Connell et al., 1995) and benefits management (Ward & Daniel, 2006). Working systematically with effects involves two critical activities: Specification of desired effects. Akin to Vicente’s (1999) view of purposes as relatively permanent properties of work domains, it is our contention that effects are more stable than functional requirements because effects are higher level and far fewer. If a focus on effects is to provide a framework within which different designs can be explored, it must, however, be possible to specify effects. This involves identifying, formulating, and prioritizing effects as well as devising methods for their measurement. We suggest that this is done in collaboration with users following a PD approach such as the MUST method (Bødker et al., 2004). AN EMPIRICAL STUDY To investigate effects-driven IT development empirically we entered into collaboration with a vendor and a customer and conducted an evaluation concerning how desired effects can be specified and measured. The evaluation involved close collaboration between four partners: the vendor organization CSC Scandihealth, the customer organization Region Zealand (one of five healthcare regions in Denmark), the evaluation site, which was the stroke unit at Roskilde Hospital, and the researchers (i.e., the authors). We chose an action-research approach (Whyte, 1991) because the study involved devising appropriate ways of specifying and measuring effects in addition to using them for specifying and measuring the effects of the EPR system, and because the active participation of all four partners was necessary to devise appropri- Formative evaluation of effects. Effects-driven IT development presupposes that it is feasible to use the presence or absence of effects as an active means of managing IT projects. For this to work it must be possible to demonstrate effects within the timeframe of IT projects. This involves setting up and conducting evaluations to measure effects of system usage during real work – and this must be done while the system is being developed, not after it has been finalized. We contend that this can be accomplished by, for example, developing and configuring systems based on standardized and flexible development platforms (e.g., HL7 (www.hl7.org) and XML (www.w3.org/XML)) and by using Wizard-of-Oz techniques (Maulsby et al., 1993). 63 ate ways of specifying and measuring effects in relation to the EPR system. the stroke unit experienced the system as if all transactions were fully IT supported. The four partners met for five full-day PD workshops to analyze clinical needs, design an EPR system, and specify desired effects and their measurement. Main parts of the EPR system were designed through up to three iterative events progressing from mock-ups on flip-over charts, through non-interactive prototypes in PowerPoint to running prototypes using XML-based templates loaded into CSC’s clinical framework based on the Oracle Healthcare Transaction Base (HTB). In parallel, a number of effects related to the clinical practice was identified, specified, and prioritized. During the workshops CSC focused mainly on identifying how the system could provide what the clinicians wanted, and the region and the clinicians from the stroke unit focused mainly on articulating and refining what they wanted in response to CSC’s questions and design suggestions. The researchers sought to elicit the effects implicit in the clinicians’ statements about their requirements toward the system. A main vehicle for doing this was asking why questions. This led to the gradual identification and formulation of a set of candidate effects. While the formulation of the effects was a process mostly involving the researchers and the clinicians, the final prioritization of which effects to measure in the evaluation was a joint activity and thereby ensured all four partners’ commitment to the prioritized effects. The prioritized effects converged on situations pertaining to the formation of an overview of patient status and the coordination of clinical activities. Consequently, the set of effects selected for measurement concerned the team conferences, ward rounds, and nursing handovers, see the next section. During the trial period we observed the clinical work and measured the prioritized effects. Prior to the trial period we had similarly observed and made measurements of the clinicians’ use of paper records. We also made nine interviews with clinicians. SETTING THE SCENE: THE STROKE UNIT The stroke unit is part of the neurological ward of Roskilde Hospital, a medium-sized Danish hospital. Stroke is a leading cause of death and chronic disability in most industrialized countries (Sarti et al., 2000). The stroke unit is an in-patient clinic with nine beds and treats approximately 850 patients a year. The clinical staff comprises physicians, nurses, and therapists. On any shift one physician is in charge of the medical treatment of the patients and one nurse is the leader of a team of 2-4 nurses and auxiliary nurses. During day shifts the group of therapists includes occupational therapists, physiotherapists, speech therapists, and neuropsychologists. Two central aspects of the work at the stroke unit are the clinicians’ continual creation and recreation of an overview of the status of the individual patients and the coordination among the clinicians, within as well as across staff groups. Overview and coordination are particularly prominent in relation to three activities: Team conference. Every morning on weekdays physicians, nurses, and therapists meet for about 15 minutes to quickly walk through the admitted patients. The team conference is intended to provide an overview of the patients’ status informed by all three staff groups and serve as a forum for cross-group coordination. In addition to a status, given by the nurse team leader, an overview of current plans is available on a whiteboard or, during the trial use of electronic records, a screen projected on the wall. The terse format makes the team conferences predominantly oral. After the last workshop, CSC undertook the technical development and implementation of the EPR system, which comprised 243 screens and involved real-time integration with other systems (e.g., a patient-administrative system, a drug-administration module, and various laboratory systems). Data about five years of patients at the hospital were migrated to the system to achieve a realistic data load. The clinicians at the stroke unit received an introduction to the evaluation and about half a day of training in the use of the EPR system and in working according to some revised, EPR-supported patient trajectories. Ward round. After the team conference the chief physician starts his or her ward round, which consists of medically assessing each patient and adjusting the treatment and care accordingly. In doing this the physician consults the patient records, sees the patient, and often seeks additional information from nurses and therapists. As there usually is no time for a nurse to escort the physician during the ward round, information exchange and coordination is obtained through the patient record and by ad hoc communication. Due to frequent interruptions the ward round stretches over a period of 3-6 hours. The evaluation involved a trial period of five days during which the EPR system replaced all paper records at the stroke unit. During the trial period the EPR system was available on all computers in the stroke unit, including the portable computers physicians bring to the patients’ bedside during ward rounds and hand-held devices for recording measurements such as blood pressure. Further, the system was projected on the wall and thus visible to all clinicians during team conferences and nursing handovers. To simulate a fully integrated EPR system, a ‘back office’ was established and staffed 24 hours a day. Patient-record entries that involved paper transactions with other wards were simulated using a Wizard-of-Oz technique: The back office continuously monitored the system, identified such entries, mailed them in the normal fashion, waited for the results to arrive, and immediately typed them into the EPR system. Thus, the clinicians at Nursing handover. At the start of every nursing shift the nurses and auxiliary nurses meet for about 45 minutes to walk through the admitted patients and coordinate activities. The walkthrough is led by the nurse team leader and based entirely on reading the patient records; no nurses from the previous shift are present. The nurse team leader gives an oral overview of each patient based on the patient records; the other nurses listen and, during the trial use of electronic records, view the projected screens. 64 RESULTS: FOUR CHANGE PROCESSES among others, mental workload, which was measured by the NASA task load index (TLX, Hart & Staveland, 1988). TLX ratings were made by each clinician participating in a team conference, ward round, or nursing handover and consisted of assigning a rating between 0 (low) and 100 (high) to each of the six TLX subscales: mental demand, physical demand, temporal demand, effort, performance, and frustration. In terms of using specification and measurement of effects for managing change processes, the evaluation of the EPR system gave different results for planned, emergent, opportunity-based, and curtailed changes (Table 1). Planned Change Many methods assign primacy to changes that are planned ahead of time and subsequently occur as intended. For example, usability specifications (Good et al., 1986; Whiteside et al., 1988) are presented as a stable set of performance goals that provides structure and guidance to iterative design processes. While this involves a risk of presuming that all changes can be anticipated and thereby disregarding other types of change, planned changes are important because they provide for working systematically toward achieving desired effects. The EPR evaluation yielded positive effects of the EPR system for all three clinical activities involved in the measurements (Hertzum & Simonsen, 2008). Most prominently, improvements in mental workload when using the EPR system instead of paper records were obtained for two of the three clinical activities. For the team conferences mental workload was significantly lower on five of the six TLX subscales. For the ward rounds the chief physician’s mental workload was significantly reduced, corroborating the results from the team conferences. For the nursing handovers mental workload neither decreased nor increased. At nursing handovers, the use of the EPR system gave rise to a planned decrease in the number of missing pieces of information and the number of messages to pass on to other clinicians after the nursing handovers. In the EPR evaluation the initial focus was mainly on planned changes, which were measured as differences between the prior use of paper records and the use of the EPR system during the trial period. The established practice of using paper records formed the baseline for measuring the effects of the EPR system. Baseline measurements were performed about a month before the trial period and involved six team conferences, four ward rounds, and five nursing handovers. During the trial period all clinicians at the stroke unit used the EPR system instead of paper records. To safeguard against misunderstandings, which might have entailed risk to patient health, the clinicians were supported by ‘shadows’ who knew the EPR system well and were present 24 hours a day. The shadows were personnel from CSC and Region Zealand, most of whom with a clinical background, and they could help the clinicians if they needed any assistance in operating the EPR system. Measurements similar to those performed during the use of paper records were performed at five team conferences, three ward rounds, and five nursing handovers. The measurements involved all effects specified at the workshops and comprised, IT enabled change Emergent Change The changes that occurred while the EPR system was in trial use were, however, not restricted to those planned ahead of the trial period (Simonsen & Hertzum, 2010). Some changes emerged spontaneously as a result of the ways in which the clinicians changed their work practices in face of the EPR system. These emergent changes became visible because they were in contrast to the work practices we had encountered when observing the clinicians’ use of paper records. During the observations of nursing handovers prior to the trial period patient records were seldom seen by clinicians other than the nurse team leader. Rather, the nurse team leader scanned a patient’s paper record and read key in- Example effect Assessment method Better overview of patients Mental workload/TLX Better coordination Counting # missing pieces of information, and # messages to pass on From oral reporting to collective reading of EPR Ethnographically inspired observation Collective investigation of the EPR Ethnographically inspired observation Sharing nursing observations during the team conference Ethnographically inspired observation Motivation for increased structuring of the nursing record Focus-group interview with nurses Improved NIP recordings Record audit (paper and EPR) Better medical-treatment and nursing plans Rating scale Planned/anticipated Emergent Opportunity-based Curtailed Table 1. Examples of different types of effect and the way they were identified and assessed. The assessments included 6/5 team conferences, 4/3 ward rounds, and 5/5 nursing handovers before/after implementation of the EPR system. Hertzum and Simonsen (2008) elaborate on the planned/anticipated changes. Simonsen and Hertzum (2010) elaborate on the emergent and opportunity-based changes. 65 formation out loud. Such oral reporting was an established practice but implied that the nurse team leader constituted a gatekeeper controlling access to the information in the paper record. In contrast, the electronic records were visible to everybody during the trial period because the screens of the EPR system were projected on the wall during nursing handovers (and team conferences). As a result the nurses engaged in a process of collective reading. The content of the electronic records was inspected by the group of nurses and they collectively participated in interpreting the status and condition of the patients, guided by the nurse team leader. The nurse team leader navigated the EPR system and read selected passages aloud to draw attention to them as well as to set a shared flow in their reading, enabling her to smoothly negotiate when to wait for a moment, when to scroll down, when to open windows with more detailed information, and so forth. This change in the nurses’ work practice emerged during the trial period, and the nurses experienced this new way of working as a strengthening of their professional role. For an elaborated description of this emergent change we refer to the ethnography given in Simonsen and Hertzum (2010). During the first days of the trial period the nurses experienced how the information on the EPR screen used at the team conferences set the agenda for the discussion at these conferences. As a result, the nurses proposed extending this screen with an extra panel containing selected nursing observations of relevance to the team conference. This change was approved by the chief physician and technically implemented halfway through the trial period. After this opportunity-based change of the EPR system, important observations made by nurses at their handovers could be selected for presentation at the following team conference, and these selected observations became more salient to the group of clinicians in their process of forming an overview of the status of the patients. During the last half of the trial period we observed how the nurses’ entries at the team-conference screen were often brought up by clinicians other than the nurses and contributed to a smoother flow in the interdisciplinary exchanges of information. Effects-driven IT development includes identifying hitherto unrealized opportunities that may be candidates for inclusion in the set of prioritized effects. The example also shows how flexible development tools can make it possible to perform many kinds of modification quickly and easily. Collective reading and interpretation is a strong candidate for an effect that will be prioritized by the nurses in their future work with EPR systems. It exemplifies that unanticipated but desirable effects may emerge when systems are tried in real use. Therefore, effects-driven IT development must remain alert to new effects that grow out of practice and should be incorporated in the set of prioritized effects. Initiatives to incorporate an emergent effect among the prioritized effects may come from the customer to ensure that the newly recognized effect persists, or it may come from the vendor as an argument for additional payment or for use in future project bids. Curtailed Change Stroke is one of eight diseases included in the National Indicator Project (NIP), which is a Danish medical database providing a scientific basis for monitoring the treatment of selected diseases. An improvement of the quality of the NIP reportings was prioritized as an effect to be achieved from using the EPR system. This effect was considered easy to obtain, for two reasons. First, using paper records the clinicians often forgot to record NIP data and a medical secretary spent considerable time collecting at least some of these data after patients were discharged. Thus, the baseline quality of the NIP reportings was perceived as rather low. Second, many of the data to be included in the NIP reportings were already recorded in other parts of the patient record and could thus be collected automatically by the EPR system; the remaining NIP data could be collected by including fields for entering them into the EPR system. It was perceived that highquality NIP data could be collected at little extra cost to the clinicians. However, the quality of the NIP reportings did not improve. It turned out that many of the data recorded in other parts of the patient record did not fully meet the requirements for NIP reportings, and that the recording of NIP data involves that specific staff groups (e.g., a nurse rather than a physician, or vice versa) make the recordings at specific times. This presupposes elaborate work procedures, which were neither in place nor supported by notifications generated by the EPR system. Opportunity-Based Change Opportunity-based change differs from planned change by not being planned ahead of time and from emergent change by not emerging spontaneously. Rather, opportunity-based changes are purposefully introduced during the change process in response to unexpected opportunities (Orlikowski & Hofman, 1997). The decision to purposefully introduce these changes indicates that they are considered desirable, and their existence verifies that it is impossible to anticipate and plan all desired changes ahead of time. At the PD workshops prior to the trial period the clinicians from the stroke unit discussed possibilities for having the EPR system support their interdisciplinary work by making it easier for them to become aware of information recorded by subgroups of clinician. The system could, for example, increase the physicians’ awareness of the nurses’ work. In the end these discussions were however not specified in an effect. As it turned out, the EPR screen made for the team conferences was mostly designed by the chief physician whereas the nurses and therapists had considerably less influence on its design. This probably reflected that in practice the team conferences to a large extent serve to provide the chief physician with an interdisciplinary overview of the patients. The reason for the failure to improve the NIP reportings appears to be an under-appreciation of the complexity of the required technical as well as organizational implementation. While the failure of the organizational implementation may in the case of the EPR system be due to the short trial period, Granlien et al. (2008) studied the adoption and use of an electronic medication record about three years after its deployment and found that no system 66 such as scenarios (e.g., Rosson & Carroll, 2002) and usability evaluation (e.g., Dumas & Redish, 1999) involve empirical work to gain an understanding of user needs and system usability, they generally occur either in a setting separated from users’ real work or without specified performance targets such as effects. facility was consistently adopted by more than 67% of the surveyed hospital wards, and that no mandated work procedure involving the system was consistently adopted by more than 48% of wards. The considerable gap between actual and mandated use three years after deployment suggests that it may be misconstrued to expect that a long period of use will lead to a gradual closing of such gaps. Rather than gradual, the adoption process may be discontinuous and characterized by a relatively brief period for exploring and developing new work practices, which thereafter tend to stick (Tyre & Orlikowski, 1994). This provides a candidate explanation for the persistence of the gap in the adoption of the electronic medication record and, in general, entails considerable risk of curtailed change. Effects-driven IT development entails that specification and evaluation of effects are incorporated in IT projects. The EPR evaluation suggests that this can be done but also illustrates that the IT-project context constrains the ways in which evaluations of effects can be made. Three such constraints stand out. First, the timing of evaluations is a trade-off between, on the one hand, evaluating after short periods of use to acknowledge project deadlines, save resources, and reduce diffusion of ineffective systems and, on the other hand, evaluating after longer periods of use to allow system errors to be corrected, users to gain proficiency, work practices to stabilize, use situations to reach their true level of heterogeneity, and longterm effects to emerge. The EPR evaluation exemplifies that effects-driven IT development may be confined to short trial periods. Thus, the consequences of various learning effects become critical to the interpretation of measurements, and little research has examined learning curves in, for example, healthcare technologies (Ramsay et al., 2000). While it is encouraging that improvements could be measured after using the EPR system for only five days, longer trial periods are desirable for the reasons listed above as well as to get beyond the goodwill that can be invested in trying something new for a restricted period of time. Curtailed change occurs when organizational implementation stops short of delivering the effects that might have been achieved had the system been more fully adopted. To avoid curtailed change it appears necessary to iteratively follow up on whether effects are achieved and, if not, intervene to reopen the process of exploring and developing new work practices. DISCUSSION Below we discuss our experiences with effects-driven IT development regarding (a) the conditions given by the ITproject context in which effects are specified and measured, (b) the amount of resources required to specify and measure effects, and (c) the perspectives for effectsdriven IT development. Specifying and Evaluating Effects within IT Projects Our empirical study suggests that the customer and vendor were able to specify and work with effects in their analysis of user needs and whether they were met by the EPR system. The process of specifying effects proceeded in parallel with the design of the EPR system. This appeared to be a workable approach, and it provided possibilities for deriving prototype functionality from identified effects as well as for deriving effects from the ongoing work on prototype functionality. In contrast, the California Franchise Tax Board’s approach to performancebased procurement involves considerable up-front work to specify the effects prior to actual development (Connell et al., 1995). While specification of user needs is part of many systems-development techniques (e.g., Bødker et al., 2004; Rosson & Carroll, 2002; Vicente, 1999), it is noteworthy in relation to effects-driven IT development that few of these techniques follow up with measurements of whether specified needs are achieved. Second, in starting to use a new IT system, users are not simply replacing one tool with another while everything else remains unchanged. Systems are accompanied by changes in individual users’ tasks, in collective work practices, and in required competences, status, and organizational structures. Thus, the effects that can be evaluated are a result of multiple, interrelated factors including social and organizational factors. Effects-driven IT development insists on the primacy of the effects and thereby on ensuring that IT projects do not become dissociated from the process of organizationally implementing the systems. The system and its organizational implementation are seen as a unit, and attempts at linking effects to either technological or organizational causes are considered dubious. For both vendor and customer this will often imply a stronger focus on the activities undertaken during organizational implementation (Markus, 2004), and customer, vendor, or both may be reluctant to extend their collaboration to also include these activities. The measurements of whether the specified effects were achieved during the trial period incorporated experiences from real use of the EPR system into the systemsdevelopment process. Thereby, the measurements went beyond evaluation of the technical implementation of the EPR system and also involved its organizational implementation. Furthermore, while we measured a set of effects prioritized ahead of the trial period, additional effects and opportunities emerged during the trial period. It is unlikely that these additional effects and opportunities would have emerged unless the EPR system had been evaluated in real clinical work. While UCD techniques Third, effects-driven IT development involves a balancing of the benefits of evaluating effects during real use against the confounds introduced by the necessity of special precautions to safeguard against unacceptable errors. While evaluating effects during real use increases validity and the possibility of unanticipated discoveries, special precautions may reduce validity. For safety-critical systems it may not be acceptable to leave users to trial and error when they encounter situations not covered by training, and the IT-project context will often preclude that evaluations are postponed until special precautions are no 67 longer necessary. Thus, either users must have ready access to support, or evaluations must move to laboratory settings. In the EPR evaluation the clinicians were supported by shadows and certain parts of the EPR system were simulated by a back office. These precautions were necessary as troubles and misunderstandings in using the system might entail risk to patient health, but they were considered clearly preferable to a laboratory evaluation. the clinicians quickly become capable of doing many things without the support of shadows and because the tasks of the back office become routine. This is important because a longer trial period will allow a greater variety of effects to materialize and settle. Perspectives for Effects-Driven IT Development The premise of effects-driven IT development is to establish a partnership in which customer and vendor share the responsibility of providing IT systems and associated work practices that yield specified usage effects. If such partnerships are based on demonstrating specified, measurable effects, customers can focus on conceptual proposals defining the problem and on desired outcomes in terms of specified effects, as opposed to more narrowly conceived usability issues or a detailed functional specification. This does not require detailed insight into technical issues. Participation can be encouraged and changes in work organization and work practices related to the deployment of IT may become easier to implement because users can be presented with descriptions of the effects they will obtain and will recognize these effects as beneficial to their work. Further, a partnership with the vendor can support long-term efforts to accomplish substantial changes in an incremental manner. Correspondingly, vendors can enhance their business area from IT systems to complete business solutions including organizational implementation and change management. Thus, a broader range of vendors’ expertise is appreciated and valued. A partnership with the customer including specialists among the users will support the vendor in devising solutions that deliver desired effects and in attaining long-term customer relationships. In addition, documentation of the usage effects obtained from a vendor’s solutions may strengthen the marketing effect toward other customers. Resources Expended The viability of effects-driven IT development depends, among other things, on the resources required to specify and evaluate effects. Table 2 shows the person hours expended on the EPR evaluation. A total of 4249.5 hours were spent by the four partners in the course of their collaboration, which lasted five months. This includes the technical development and configuration of the EPR system for use at the stroke unit and the development of interfaces to several clinical systems. Such activities are indispensable for conducting realistic evaluations. The five-day trial period consumed 19% of the hours spent on the evaluation, corresponding to about 23 weeks of work. The labor intensity of the trial period was due to the 24hour-a-day nature of hospital work, the back office, the shadows, and the numerous evaluation-related activities running in parallel. It is noteworthy that only 9% of the person hours were spent by the clinicians at the stroke unit, and mostly through their participation in the preparation phase. The EPR evaluation consumed many resources. A prime reason for this is the sophistication required from the EPR system because it affects all groups of clinician at the stroke unit, is used repeatedly by the clinicians during their shifts, handles information pertinent to their work, and concerns a domain in which mistakes may have severe consequences. Relative to the budget of a full future development and deployment of the EPR system at the hospital (expected to be beyond US$ 20 million), the EPR evaluation was, however, a minor expense. Further, we believe three sources of resource optimization suggest themselves. First, the loading of real-world patient data consumed 864 of the 1996 hours CSC spent on preparations. Less comprehensive loading of data may suffice. Second, the extent of the preparations included that large parts of the system were developed for the evaluation. Fewer resources will be needed for preparations in projects that to a larger extent consist of reusing extant functionality, for example an evaluation of the EPR system at the stroke unit of another hospital. Third, the resources needed for the trial period may not increase linearly with an extension of the trial period beyond five days, because Activity CSC Preparations Effects-driven IT development supports a sustained PD process and a more permeable boundary between vendor and customer, especially during the organizational implementation of systems. This involves that vendors must be granted influence on the nature, extent, and managerial enforcement of organizational implementation. Customers, on their part, must be able and willing to engage in collaboration on these kinds of condition. We are currently investigating effects-driven IT development as an instrument for managing the development process, but our long-term goal is that contract fulfillment should involve whether specified effects are achieved. Mechling (1999) find that whereas practitioners, at least on the customer side, are very optimistic about the potential of performance contracting there is at the same time very little real-world experience to learn from and great uncertainty Region Zealand Stroke unit Researchers 1996 527.4 237.5 240 64 0 65 71 Trial period 534 141.6 70 58 Other 197 0 0 48 Total 2791 669 372.5 417 Training and paper-record measurements Table 2. Person hours spent by the four partners in the EPR evaluation. 68 about how to proceed. While we share this uncertainty, we also find that effects may provide a vehicle for contractually specifying IT projects in a way that gives PD, organizational usability, and its evaluation a central role in systems development. gain proficiency in the use of the system and for new work practices to stabilize. This emphasizes the general issue of fitting pilot implementations to the timeframe of IT projects. Effects-driven IT development challenges the traditional division of responsibilities, where the vendor is paid for design and technical implementation while the customer is responsible for organizational implementation, including the attainment of the effects desired from using the system. If vendor and customer are to share the responsibility of providing usage effects, a new contractual foundation for IT projects is needed, in which contract fulfillment is determined on the basis of proven utility value and measured effects. At present, our research has barely touched upon this issue, but it might be critical to the adoption of effects-driven IT development in commercial IT projects. CONCLUSION Effects-driven IT development makes specification of effects and formative evaluation conducted during real use central activities of IT projects. This incorporates a sustained PD approach throughout technical and organizational implementation. While the results of our empirical work is promising as regards the possibilities of specifying and measuring effects, further work is required to elaborate and evaluate many aspects of effects-driven IT development. The EPR evaluation reported in this paper demonstrates that effects from planned change can be specified and evaluated in close collaboration with the clinical staff and in parallel with the development of the EPR system. Evaluations can be conducted using recognized methods such as TLX or, alternatively, the technology acceptance model (TAM, Davis, 1989) or health care empowerment questionnaire (HCEQ, Gagnon et al., 2006). Specific effects might be evaluated using specialized questionnaires though this increases the ways in which the results of the evaluation can be interpreted. ACKNOWLEDGEMENTS We are grateful for the inspiring partnership with CSC Scandihealth and Region Zealand throughout this project. Special thanks are due to the clinicians at the stroke unit for their enthusiasm and willingness to participate in the project in spite of their busy schedules. The authors were neither financially nor otherwise related with the customer, vendor, or stroke unit, apart from the professional relations that evolved in the course of the study. Our empirical study shows that four types of change process need to be considered in working with effects: planned, emergent, opportunity-based, and curtailed, where the latter three only occur during real use of the system. The impact of evaluating the system during a pilot implementation is immense. In the EPR evaluation the system was only in use for five days; however, 38% (183 out of 482) of the user’s design ideas were reported during this period. Emergent, opportunity-based, and curtailed change needs to be systematically and efficiently identified and analyzed. This implies a shift in the role of ethnography-based evaluation from a role describing existing work practices or the situation after a complete implementation (known from CSCW and STS research) to a formative role involving pilot implementations of how new IT is appropriated by users (Simonsen, 2009). REFERENCES Berg, M., Aarts, J., and Lei, J. v. d. ICT in health care: Sociotechnical approaches. Methods of Information in Medicine, 2003, 42(4): 297-301. Bødker, K., Kensing, F., and Simonsen, J. Participatory IT design: Designing for business and workplace realities. MIT Press, Cambridge, MA. 2004. Connell, K., Andal, D., and Brown, C. L. Performance based procurement. Another model for California. California Franchise Tax Board, Sacramento, CA. 1995. Davis, F. D. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 1989, 13(3): 319-340. Dumas, J. S., and Redish, J. C. A practical guide to usability testing. Revised edition. Intellect Books, Exeter, UK. 1999. Fichman, R. G., and Moses, S. A. An incremental process for software implementation. Sloan Management Review, 1999, 40(2): 39-52. Gagnon, M., Hébert, R., Dubé, M., and Dubois, M.-F. Development and validation of an instrument measuring individual empowerment in relation to personal health care: The Health Care Empowerment Questionnaire (HCEQ). American Journal of Health Promotion, 2006, 20(6): 429-435. Glass, R. L. Pilot studies: What, why, and how. Journal of Systems and Software, 1997, 36(1): 85-97. Good, M., Spine, T. M., Whiteside, J., and George, P. User-derived impact analysis as a tool for usability engineering. In Proc. CHI'86, pp. 241-246. ACM Press, New York, 1986. Granlien, M. F., Hertzum, M., and Gudmundsen, J. The gap between actual and mandated use of an electronic medication record three years after deployment. In S. The introduction of pilot implementations entails several challenges. Conducting a pilot implementation involves balancing careful planning with engaging in a process characterized by a high degree of uncertainty. This requires careful preparations and a high level of readiness, especially during the initial part of the pilot implementation, to correct or alleviate critical system errors, handle immediate demands for system re-configuration, and accommodate needs for adapting the organization to the system. Pilot implementations are not a well researched subject, and more research is needed, including how learning objectives can be appropriately integrated in situations where the involved users also need to get their daily job done: What extra precautions and costs are needed to balance learning objectives against competing demands for stable day-to-day operation and production rate? The EPR evaluation involved a five-day pilot implementation, which was too short for the clinicians to 69 K. Andersen, G. O. Klein, S. Schulz, J. Arts, and M. C. Mazzoleni (eds), Proc. MIE'2008, pp. 419-424. IOS Press, Amsterdam, 2008. Granlien, M. S., and Hertzum, M. Implementing new ways of working: Interventions and their effect on the use of an electronic medication record. In Proc. GROUP'2009, pp. 321-330. ACM Press, New York, 2009. Hamilton, S., and Chervany, N. L. Evaluating information system effectiveness - Part I: Comparing evaluation approaches. MIS Quarterly, 1981, 5(3): 55-69. Harlen, W., and James, M. Assessment and learning: Differences and relationships between formative and summative assessment. Assessment in Education: Principles, Policy & Practice, 1997, 4(3): 365-379. Hart, S. G., and Staveland, L. E. Development of NASATLX (task load index): Results of empirical and theoretical research. In P. A. Hancock and N. Meshkati (eds), Human Mental Workload, pp. 139-183. NorthHolland, Amsterdam, 1988. Hartswood, M., Procter, R., Slack, R., Voss, A., Büscher, M., Rouncefield, M., and Rouchy, P. Co-realization: Toward a principled synthesis of ethnomethodology and participatory design. In M. S. Ackerman, C. A. Halverson, T. D. Erickson, and W. A. Kellogg (eds), Resources, Co-Evolution and artifacts, pp. 59-94. Springer, London, 2008. Hertzum, M., and Simonsen, J. Positive effects of electronic patient records on three clinical activities. International Journal of Medical Informatics, 2008, 77(12): 809-817. ISO 9241 Ergonomic requirements for office work with visual display terminals (VDTs) - Part 11: Guidance on usability. International Standard Organization, Geneva, CH. 1998. Leveson, N. G. Intent specifications: An approach to building human-centered specifications. IEEE Transactions on Software Engineering, 2000, 26(1): 15-35. Markus, M. L. Technochange management: Using IT to drive organizational change. Journal of Information Technology, 2004, 19(1): 4-20. Maulsby, D., Greenberg, S., and Mander, R. Prototyping an intelligent agent through Wizard of Oz. In Proc. INTERCHI '93, pp. 277-284. ACM Press, New York, 1993. Mechling, J. Better funding for government IT: Views from the front line. Journal of the American Society for Information Science, 1999, 50(4): 305-313. Orlikowski, W. J., and Hofman, J. D. An improvisational model for change management: The case of groupware technologies. Sloan Management Review, 1997, 38(2): 11-22. Ottersten, I., and Balic, M. Effect managing IT. Liber, Køge. 2007. Ramsay, C. R., Grant, A. M., Wallace, S. A., Garthwaite, P. H., Monk, A. F., and Russell, I. T. Assessment of the learning curve in health technologies: A systematic review. International Journal of Technology Assessment in Health Care, 2000, 16(4): 1095-1108. Robinson, M. Design for unanticipated use... In G. De Michelis, C. Simone, and K. Schmidt (eds), Proc. ECSCW'93, pp. 187-202. Springer, Berlin, 1993. Rosson, M. B., and Carroll, J. M. Usability engineering: Scenario-based development of human-computer interaction. Morgan Kaufmann, San Francisco, CA. 2002. Rzevski, G. Prototypes versus pilot systems: Strategies for evolutionary information system development. In R. Budde, K. Kuhlenkamp, L. Mathiassen, and L. Zullighoven (eds), Approaches to Prototyping: Proceedings on the Working Conference on Prototyping, pp. 356-367. Springer, Heidelberg, 1984. Sarti, C., Rastenyte, D., Cepaitis, Z., and Tuomilehto, J. International trends in mortality from stroke, 1968 to 1994. Stroke, 2000, 31(7): 1588-1601. Shapiro, D. Participatory design: The will to succeed. In Proc. AARHUS'2005 Conference on Critical Computing, pp. 29-38. ACM Press, New York, 2005. Simonsen, J. The role of ethnography in the design and implementation of IT systems. Design Principles and Practices, an International Journal, 2009, 3(3): 251264. Simonsen, J., and Hertzum, M. Participative design and the challenges of large-scale systems: Extending the iterative PD approach. In J. Simonsen, T. Robertson, and D. Hakken (eds), Proc. PDC'2008, pp. 1-10. ACM Press, New York, 2008. Simonsen, J., and Hertzum, M. Iterative participatory design. In J. Simonsen, J. O. Bærenholdt, M. Büscher, and J. D. Scheuer (eds), Design Research: Synergies from Interdisciplinary Perspectives, pp. 16-32. Routledge, London, 2010. Sommerville, I. Software engineering. Seventh edition. Addison Wesley, San Francisco, CA. 2004. Steinberg, D. H., and Palmer, D. W. Extreme software engineering. Pearson, Upper Saddle River, NJ. 2004. Suchman, L. A. Human-machine reconfigurations: Plans and situated action, 2nd edition. Cambridge University Press, Cambridge, UK. 2007. Turner, J. R. The role of pilot studies in reducing risk on projects and programmes. International Journal of Project Management, 2005, 23(1): 1-6. Tyre, M. J., and Orlikowski, W. J. Windows of opportunity: Temporal patterns of technological adaptation in organizations. Organization Science, 1994, 5(1): 98118. Vicente, K. J. Cognitive work analysis: Toward safe, productive, and healthy computer-based work. Erlbaum, Mahwah, NJ. 1999. Ward, J., and Daniel, E. Benefits management: Delivering value from IS & IT investments. Wiley, Chichester, UK. 2006. Whiteside, J., Bennett, J., and Holtzblatt, K. Usability engineering: Our experience and evolution. In M. Helander (ed), Handbook of Human-Computer Interaction, pp. 791-817. Elsevier, Amsterdam, 1988. Whyte, W. F. (ed). Participatory action research. Sage, Newbury Park, CA. 1991. 70