AUC Academic Conference 'From Virtual to Reality' The University of
Queensland 1996AUC Academic Conference 'From Virtual to Reality' The University of
Queensland 1996
Back to Home Page ...... by Presenter ...... by Paper Title ...... by Keyword
Paper Title:
VisMap:
A CD for Exploring Evaluation Issues in Computer-Facilitated LearningPresenter:
Paul Fritze, The University of Melbourne
Authors:
Paul Fritze
Educational Multimedia Designer, The University of MelbourneDr Carmel McNaught
Senior Lecturer, La Trobe University
Keywords: Evaluation, Professional Development
Faculty area: Academic Development
This paper needs to be viewed at full screen size. Please click on the 'size button' at the top right of this window in the title bar, or just drag the browsers window larger using the 'scale window' icon at the bottom right of this window.
Abstract
The VisMap program and its implementation in an interactive CD-ROM are designed firstly, to facilitate the process of evaluation of educational multimedia and secondly, to provide a rich resource for professional development courses in which issues of evaluation are covered. As an evaluation instrument, it has been used in post-trial interviews to stimulate recall of the learning experience and to highlight critical incidents. As a teaching resource, it provides an integrated view of different evaluation techniques as applied to a particular multimedia project illustrating the complexity, pitfalls and essentially individual nature of the student learning experience. The program can be used in lecture presentations, workshop exercises and classroom discussions to promote a deeper understanding of many important issues concerning computer-facilitated learning.
Introduction
We are experiencing a upsurge of innovations in teaching and learning fostered by rapid development in computer-based technologies. Corresponding attention to the evaluation of these products is needed if the significant effort and expense of production is to be justified in terms of improvement in the quality of learning, cost of production and development of staff skills. To what extent this occurs is open to question. Laurillard (1993) refers to research and development projects on educational media paying "...quantities of hard cash for development, lip service to evaluation...". Investigating the validity of this statement, Alexander & Hedberg (1994) examined reports of technology-based learning projects, drawing attention to the inadequacies of the studies in their lack of focus on both evaluation and implementation. It is also the view of the authors that, for a variety of reasons, many computer-facilitated learning (CFL) development projects maintain a simplistic approach to evaluation in order to satisfy, at best, its most basic purposes. The cost of inadequate evaluation to both students and institutions can be large, given the high level of investment and the increasingly competitive nature of the tertiary sector. While recognising the conflicting pressures on academic staff, we feel that there exists a real need to provide further support and professional development in this area. This can, in part, be facilitated by applying the strengths of computer-based media itself.
Purposes of CFL evaluation
To clarify the role of evaluation in CFL, three fundamental purposes of evaluation of importance to the faculty developer can be identified. These are to facilitate the development of CFL materials, to report on their effectiveness in meeting the learning need and, in a more open-ended manner, to refine and enhance the developers' understandings of the interaction between the process of development and implementation and the reality of the actual learning experience. This increase in understanding will initially be made by the participants in the development but should flow on to others. The categories of evaluation are compared in table 1. The first two purposes are clearly associated with formative and summative evaluation, while the third we have labelled 'educative evaluation'. A fourth possible use of evaluation, educational research, considers generalised models of learning in a more rigorous manner, and is beyond the direct interest of most faculty developers.
Table 1. The purposes of CFL evaluation
Formative evaluation is undertaken during specifically during development, to test and refine the emerging product in respect to the educational intentions and anticipated delivery. Summative evaluation then reports on the actual learning outcomes of the implementation, validating the quality of teaching and identifying areas requiring revision. Although often carried out in a rather ad hoc manner, both formative and summative evaluation appear to be generally recognised as having clear purposes, consequences and associated responsibilities for the developer. If we accept, however, that the development process is, in fact, a learning experience for the developers, then there is also a need for feedback and reflection on that performance for that learning to be effective. The performance in this case is expressed through the impact of the design on learning as reflected by the actual use by students. This is likely to be different to the experience gained in formative trials and will bring up significant issues that occurs at a level far deeper that is reported by summative evaluation.
Evaluation for this educative purpose depends on and extends the participants' prior personal understandings of teaching, learning and media development. While certainly utilising the data of formative and summative evaluation, it extends the focus to better reflect the reality of the implementation. This requires a level of detail appropriate to the complex interaction between the learning situation, design and production issues, social effects and learning outcomes.
We believe that educative evaluation, the reflection path necessary for developers to refine their own understandings of materials design and learning, is only informally recognised despite its role in improving the ultimate quality of developments in the longer term. Consider how many CFL projects are never fully implemented or disappear with out trace after only brief use. The institution must continue to invest large amounts into further CFL developments in the belief that the increase in expertise will eventually support the production of more effective, efficient and competitive learning systems. We would argue that, in practice, the value of these experiences, whether successful or otherwise, is often not fully exploited and what understandings have been gained often remain with individuals of the development team. A mechanism for sharing these understandings more widely is needed.
VisMap evaluation project
The VisMap project is designed to support the processes and understanding of a variety of evaluation approaches. It focuses on one particular CFL project, providing:
instruments for gathering formative and educative evaluation data from student trials,
a tool for analysis and presentation of evaluation data, and
a CD-ROM resource for use in professional development.
Ariadne's Thread tutorial
The evaluation study is based on the tutorial, 'Ariadne's Thread', a HyperCard program designed to reduce the load on Library staff and lecturers by increasing the library skills of first year Arts students (Jones & Fritze, 1996). It was developed by the Interactive Multimedia Learning Unit (now the Multimedia Education Unit) of The University of Melbourne in conjunction with University librarians and academic staff from the Faculty of Arts. The main areas of focus are use of the catalogue system and interpretation of reading list citations. The program involves a fairly typical tutorial structure involving a variety of topics consisting of several screens employing a combination of information and interactive questions.
Figure 1. Topic structure of Ariadne's Thread
The program was developed and trialed during 1994 and scheduled computer laboratory sessions organised by library staff were first offered to students on a voluntary basis in 1995. Detailed formative evaluation involving collection of computer audit trails (Fritze, 1994), pre- and post-questionnaires and informal interviews were used to successively refine the program over this period.
Evaluation data collection
The VisMap software is written in HyperCard although it is in the process of translation to Oracle Media Objects. The evaluation data methodology is summarised in figure 2 which indicates the three functions of the software, the forms of evaluation data and the settings under which they were gathered.
Figure 2. Summary of evaluation data methodology
Some of the techniques are worth elaborating upon:
Learning settings for gathering evaluation data
Evaluation data comes from three learning settings, the actual computer laboratory tutorials run in 1995 and volunteer trials in a specially set up multimedia laboratory, involving either single students, or pairs operating in a 'think-aloud' evaluation setting (Gomoll, 1990). In the last case, the pair of students are encouraged to verbalise their impressions of the experience as they work through the tutorial. There are clear differences between the ways in which students work in a natural educational setting where their focus is on learning content material, and trial settings where students are also conscious of giving feedback about the design of the software.
Visual mapping
Visual mappings of the computer audit trails (figures 3, 5 and 6) illustrate each student session and provide a focus for the post-trial discussion. The maps also provide the overview from which video sequences, reflections and audit trails can be indexed. The technique of visual mapping of audit trails has been described elsewhere (Fritze, 1994).
Post-trial interview
Immediately following the single student trials and think-aloud evaluation sessions, students were invited to reflect on their experience of working through the instructional material. These were undertaken in the same location using the VisMap software as an instrument of evaluation. A visual mapping of the audit trails was automatically processed and provided a visual summary of all data logged by the computer including assessment data and audit trails. A display of static tutorial screen images was also available for reference. Open-ended discussion then centred around the representation of this information which greatly stimulated recall of the experience. Students explained the reasons why they moved in a particular sequence, why they spent time on particular screens, etc.
Figure 3. VisMap as an evaluation instrument to focus post-trial interview
Screen and student videos
Computer screen activity was recorded by taking using the video output of a Mac 8100AV, a high resolution video monitor, video tape recorder and microphone arranged to pick up conversations and mouse clicks (figure 4). A second recorder and camera captured images of the students. Recordings were later converted to QuickTime movies.
Figure 4. Collection of data
Analysis of data
Figure 5 shows how the think-aloud video, video of screen activity, audit trails and post-interview audio recordings can be linked together. As the user clicks on one screen in the visual map, the program displays the video of students discussing their work on the screen, a video of the screen and the audit trail at that point. Interesting events or critical incidents can be noted by the installation of comment balloons. Evaluator's comments, think-aloud videos or audios of post-tutorial interviews can be used in this analysis.
Figure 5. VisMap used in analysis mode to link extracts from video and audio records with mapping of audit trails
Professional development
As a teaching resource, the VisMap CD-ROM provides a integrated view of different evaluation techniques. It can be used in lecture presentations, workshop exercises and classroom discussions to promote a deeper understanding of many important issues concerning computer-facilitated learning. The comment balloons produced during the exploration and analysis of the audit trail maps can now be used by other developers. When a user clicks on one of the balloons, the program displays the experience the students had when working through this section of the tutorial. They can examine either the evaluator's comments, students' in-trial comments or their reflective post-trial comments.
Figure 6. VisMap as used in a professional development setting with comments linked to the various interesting events on the map
Discussion
The rich, in-depth, personalised data from student trials is clearly portrayed in VisMap in a way which focuses on the student perspective. One can work through and reflect on students' actual experiences.
The experience of this project has clearly illustrated the fact that students behave differently in different learning settings. No single evaluation method is sufficient in itself-the concept of triangulation is clearly illustrated in this project. Student trial data is necessarily somewhat artificial and this data needs to be juxtaposed with data from the real world where students work naturally in an educational setting. The audit trail data from actual educational use can be mapped in VisMap and compared with the few samples of full student trial data. This provides another valuable focus for discussion in professional development sessions.
The process of gathering full student trial data is clearly time consuming and cannot be undertaken to any significant extent by many faculty multimedia developers. The production of a resource that can be used by the wider developer community can make a significant contribution to the understanding of the student learning perspective central to issues of multimedia design and development.
Back to Home Page ...... by Presenter ...... by Paper Title ...... by Keyword
References
Alexander, S., & Hedberg, J. G. (1994). Evaluating technology based-learning: Which model? pp. 233-244 in K. Beattie, C. McNaught & S. Wills (Eds.). Interactive multimedia in university education: Designing for change in teaching and learning. Amsterdam: Elsevier.
Fritze, P. (1994). A visual approach to the evaluation of computer-based learning materials. pp. 273-285 in K. Beattie, C. McNaught & S. Wills (Eds.). Interactive multimedia in university education: Designing for change in teaching and learning. Amsterdam: Elsevier.
Gomoll, K. (1990). Some techniques for observing users. pp. 85-90 in B. Laurel (Ed.). The art of human-computer interface design. Reading, Massachusetts: Addison-Wesley.
Jones, D. & Fritze, P. (1996). Ariadne's Thread. An interactive computer-assisted training programme teaching undergraduates how to maximise their use of the Library OPAC and how to understand the basics of bibliography. pp. 477-484 in Proceedings of the Victorian Association for Library Automation (VALA) 1996 Biennial Conference, Melbourne, 30 January - 1 February.
Laurillard, D. (1993). Rethinking university education: A framework for the effective use of educational technology. London: Routledge.
Mr Paul Fritze
Educational Multimedia Designer
Centre for the Study of Higher Education
The University of Melbournephone +61 3 9344 7757
fax +61 3 9344 7576email P.Fritze@cshe.unimelb.edu.au
Dr Carmel McNaught
Senior Lecturer
Academic Development Unit
La Trobe UniversityBack to Home Page ...... by Presenter ...... by Paper Title ...... by Keyword
