Rich Laverty       Frank Schindler     Bob Medearis
619-553-2918       619-553-2845        619-553-6377
laverty@nosc.mil   fschindl@nosc.mil   medearis@nosc.mil

Do not distribute to DTIC or other data depositories.
Distribution authorized to DOD components only; premature dissemination (date). Other requests shall be referred to Naval Command, Control and Ocean Surveillance Center (NCCOSC), RDT&E Division, San Diego, California 92152-5000.

---

1. Preliminary Information

---

1.1 Report Title

1.2 Notices

1. The University does not provide any licenses to the Government for commercial, off-the-shelf software that have been be used as part of the overall effort.
2. The University reserves the right to develop other products and own them, even if they use ideas implemented in the delivered software.

1.3 Abstract

1.4 Table of Contents; List of Tables, Figures and Illustrations

1.4.1 Contents

1. TECHNICAL REPORT
2. 1.Preliminary Information
   1. 1.1Report Title
   2. 1.2Notices
   3. 1.3Abstract
   4. 1.4Table of Contents; List of Tables, Figures and Illustrations
      1. 1.4.1Contents
      2. 1.4.2Tables and Figures
   5. 1.5List of Symbols, Abbreviations, and Acronyms
   6. 1.6Preface and Acknowledgements
3. 2.Body of the Report
   1. 2.1Summary
   2. 2.2Introduction
   3. 2.3Methods, Assumptions, and Procedures
      1. 2.3.1Overview of Belvedere
         1. 2.3.1.1Cognitive Support
         2. 2.3.1.2Collaborative Support
         3. 2.3.1.3Evaluative Support
         4. 2.3.1.4Other Features
      2. 2.3.2Architecture
         1. 2.3.2.1Concepts of Application
            1. 2.3.2.1.1Critical Inquiry in Science
            2. 2.3.2.1.2Generality
         2. 2.3.2.2Interface Presentation
            1. 2.3.2.2.1A Graphical Interface for Critical Inquiry
            2. 2.3.2.2.2Relations to Concepts of Applications
            3. 2.3.2.2.3Comments on the Analysis
         3. 2.3.2.3Concepts of Operations
            1. 2.3.2.3.1Supporting Collaborative Coached Critical Inquiry
            2. 2.3.2.3.2Relations to Other Levels
            3. 2.3.2.3.3Interoperability and Reusability
         4. 2.3.2.4Abstract Implementation
            1. 2.3.2.4.1Relations to Concepts of Operations
            2. 2.3.2.4.2Interoperability and Reusability Issues
         5. 2.3.2.5Resource Layer
         6. 2.3.2.6Architectural Lessons
      3. 2.3.3Design of Computer Coaches
         1. 2.3.3.1Introduction
         2. 2.3.3.2Pedagogical Constraints on Advice
         3. 2.3.3.3Syntactic Advice Strategies
         4. 2.3.3.4Consistency-Based Advice Strategies
            1. 2.3.3.4.1Consistency and Transitivity
            2. 2.3.3.4.2Formative Experiments
            3. 2.3.3.4.3 "Snippets"
            4. 2.3.3.4.4Implemented Expert Coach
            5. 2.3.3.4.5Reimplementation in Java
         5. 2.3.3.5Coaching Status and Future Directions
      4. 2.3.4Design of Classroom Implementation
         1. 2.3.4.1Classroom Environment
         2. 2.3.4.2Teacher's Role
         3. 2.3.4.3Curriculum Materials
         4. 2.3.4.4Student Activities
         5. 2.3.4.5Supporting Peer Evaluation with Performance-based Assessment
      5. 2.3.5Collaborative Activities Funded
         1. 2.3.5.1Belz and Luckham: Architecture Abstraction Hierarchy Reference Model
         2. 2.3.5.2Forbus and Koedinger: Integrated Feasibility Demonstration of Science Learning Spaces
            1. 2.3.5.2.1Demonstration Architecture
            2. 2.3.5.2.2Lessons Learned from the Integration Demonstration
         3. 2.3.5.3Workshop: Architectures and Methods for Designing Cost-Effective and Reusable ITS
   4. 2.4Results
      1. 2.4.1Highlights of Project
         1. 2.4.1.1Cognitive/Education Highlights
         2. 2.4.1.2Technology Highlights
         3. 2.4.1.3Implementation Status and Platforms
            1. 2.4.1.3.1Belvedere
            2. 2.4.1.3.2Collaborative Database Server and Connection Manager
            3. 2.4.1.3.3Coach ("Idea Generator")
         4. 2.4.1.4Maturity and Deployability
         5. 2.4.1.5Openness and Interoperability
            1. 2.4.1.5.1Points of Interoperability:
            2. 2.4.1.5.2Limitations on Interoperability:
         6. 2.4.1.6(Re)Applicability
         7. 2.4.1.7Venues and Collaborations
         8. 2.4.1.8Future Research
            1. 2.4.1.8.1Technology Research
            2. 2.4.1.8.2Cognitive/Education Research
      2. 2.4.2Results of Independent Evaluations
         1. 2.4.2.1Evaluation Methodology
         2. 2.4.2.2Summary of Evaluation Results
   5. 2.5Conclusions
   6. 2.6Recommendations
      1. 2.6.1Follow-on Work for Belvedere
         1. 2.6.1.1Port Server to NT
         2. 2.6.1.2Refine and Extend the Belvedere Interface as Guided by User Feedback
         3. 2.6.1.3Future Work for Coaching
         4. 2.6.1.4Cognitively Motivated Design of Challenge Activities and Materials
         5. 2.6.1.5Realistic Classroom Implementation
         6. 2.6.1.6Assessment Rubrics
         7. 2.6.1.7Commercial Deployment
      2. 2.6.2Additional Research
         1. 2.6.2.1Designing open, interoperable educational software.
         2. 2.6.2.2Representational devices as "epistemic forms" for collaborative learning.
         3. 2.6.2.3Software participation in reflective learning interactions.
         4. 2.6.2.4Examining the cost-benefit tradeoff between knowledge engineering and coaching functionality.
         5. 2.6.2.5Designing hypermedia structures to scaffold critical inquiry skills.
         6. 2.6.2.6Scaffolding learning from simulations and visualizations.
         7. 2.6.2.7Comparing scientists' and learners' inquiry skills.
         8. 2.6.2.8Realistic school implementation of advanced educational technology.
4. 3.End Matter
   1. 3.1Appendices
   2. 3.2Bibliography

1.4.2 Tables and Figures

FIGURE 1. BELVEDERE INQUIRY DIAGRAM AND ADVICE

FIGURE 2. THE BELVEDERE INTERFACE

FIGURE 3: ABSTRACT IMPLEMENTATION LAYER

FIGURE 4. EXAMPLE ADVICE CONSISTENCY PATH.

FIGURE 5. "SNIPPETS" MAY BE REFERENCED BY CLICKING ON DOCUMENT ICONS.

FIGURE 6. WEB-BASED MATERIALS FOR CHALLENGE PROBLEM

FIGURE 7. SAMPLE ASSESSMENT RUBRICS

FIGURE 8. ARCHITECTURE REFERENCE MODEL.

FIGURE 9. SCIENCE LEARNING SPACE DEMONSTRATION.

FIGURE 10. LEARNING SPACE DEMONSTRATION ARCHITECTURE.

1.5 List of Symbols, Abbreviations, and Acronyms

AAAS

CAETI

Computer Aided Education and Training Initiative

CGI

Common Gateway Interface

CSCL

Computer Supported Collaborative Learning

DoD

Department of Defense

DoDDS

Department of Defense Dependent Schools

DoDEA

Department of Defense Education Activity

HTML

HyperText Markup Language

HTTP

HyperText Transfer Protocol

ITS

Intelligent Tutoring System

LRDC

Learning Research and Development Center

MOO

Object Oriented MUD (see)

MUD

Multi-User Domain

NSES

National Science Education Standards

SGML

Standard Generalized Markup Language

URL

Universal Resource Locator

WYSIWIS

What you see is what I see

WWW

World Wide Web

1.6 Preface and Acknowledgements

Project PIs are indebted to Kirstie Bellman for her visionary leadership, to Neil Jacobstein for his facilitation as cluster leader, to Gary Bridgewater for the many ways in which he coordinated our work, to Bill Bewley, Sue Latzko, and their colleagues at ISX for coordination of site visits, to Lila Cheville for helping us coordinate with the DoDEA curriculum, to Lynne Gilfilian for evaluation, and to Frank Belz for useful discussions about architecture. The Belvedere client was programmed by Kim Harrigal and Dan Jones; the collaborative database server by Dan Jones; and automated coaching by Dan Suthers and Joe Toth. Science materials were prepared by Eva Toth and Arlene Weiner. On-line help and user interface testing was performed by John Connelly. Sandy Katz contributed to evaluation planning. Dan Suthers was technical lead and project manager.

---

2. Body of the Report

---

2.1 Summary

However we went much further than handing raw technology to the schools - we also provided comprehensive support for implementation and integration in the classroom and curriculum. Situated in a well-known institute for the study of school learning, we are sensitive to the many demands and opportunities that school reform places on teachers and schools. Already experienced with classrooms, we listened carefully to DoDEA curriculum supervisors and teachers, developed curriculum materials keyed to the DoDDS curriculum standards and objectives, and modified materials and software in the development process . Our "science challenge" activities are designed to match and enrich the DoDDS curriculum, and are based on standards such as the National Research Council's NSES, LRDC's New Standards, and the AAAS Benchmarks for Science Literacy. We also provided students and teachers with assessment rubrics which serve to scaffold student activities and guide peer review, as well as help the teacher assess nontraditional learning objectives.

2.2 Introduction

An important form is the intelligent tutoring system (ITS). Quite a number have been built, at least in prototype form. Most have emphasized student modeling and providing feedback in the form of statements about knowledge deemed to be missing from the student. Several labs, notably John Anderson's, have pursued a scheme called model tracing, in which an intelligent critic intervenes whenever the student deviates from an "expert" approach in solving a set problem [Anderson & Pelletier, 1991; Anderson, et al. 1995] A different approach is associated with the work in Roger Schank's lab, in which cases are indexed according to situational properties of impasses or breakdowns that occurred within them. When a case becomes relevant because it matches a current realized or potential breakdown, the system offers to tell the student a "story" that might be helpful. Finally, our coached apprenticeship scheme provides all six affordances, but, at the outset of this project, only for technical jobs and only in purely computer-human interactions, rather than collaborative interactions of students with each other and with teachers. Consequently, we set out to adapt our approach to school curriculum. Consistent with research that shows that students learn better when they actively pursue understanding rather than passively receiving knowledge [Chi et al 1989, Resnick & Chi 1988, Webb 1989] the classroom teacher is now being urged to become a "guide on the side" rather than "the sage on the stage." New roles have also been recommended for tutoring systems that parallel the teacher's new role in "decentered" classrooms [Chan & Baskin, 1988; Clancey, 1992; Roschelle, 1994]. Hence work was addressed towards tutoring systems that augment the learning processes of students engaged in collaborative critical inquiry [O'Neill & Gomez, 1994; Slavin 1990].

A second form is the exploratory environment , such as a simulation of friction-free movement of objects. Exploratory environments provide many opportunities that might stimulate learning-producing activity among student peers and with teachers, but such systems characteristically lack any kind of advisor to explain, coach, and scaffold learning. They also tend to lack any structure for facilitating an explicit learning agenda. Still, we borrow from the exploratory microworlds form the notion of having kits that students can use to configure environments from which they might learn something.

A third form is systems to support authoring and, more generally, communications . Students routinely use word processing in schools now, for example, and modest beginnings of groupware for learning can be found. In the most innovative experiments, students author multimedia "publications," an approach we will be promoting as well. A variety of low-bandwidth interaction forums, MUDs, for example, are also being used in experimental learning research efforts and by some innovative teachers. However, the tools for developing substantial interactions and for publishing substantial bodies of work are just beginning to be developed to support learning activities. We planned to focus on a piece of this problem, tools to support (scaffold) the expansion from simple, narrative communications to communications of arguments, especially comparisons of alternative theories and debates over alternative public policies. In such dialectical activities, communications are more complex, there is more need for diagrammatic tools and other intelligent support for students, and there is great need to quickly recast subsets of discussion in different forms. For this reason, we saw the development of authoring, advising, and interacting capabilities for diagrammatic representations of arguments to be of central importance to stimulating a new level of high-order thinking and interaction over networks.

A fourth form is technology to support collaborative learning. There are at least three motivations driving the growing interest in this approach. The first is a practical one: having more than one student use instructional software is a cost saver. Also, there are seldom enough computer workstations for every student. The second (and perhaps primary) motivation is a response to the abundant research showing that collaborative learning correlates with a wide range of positive outcomes. Within non-computer-based, classroom and laboratory settings, collaborative learning has been shown to correlate with greater learning, increased productivity, more time on task, transfer of knowledge to related tasks, higher motivation, and heightened sense of competence [Johnson & Johnson, 1989; Rysavy & Sales, 1991; Sharan, 1980; Slavin, 1990]. Similarly, Webb's [1987] review of the research on collaborative use of instructional software suggests that it is at least as effective as individual use, and is sometimes superior [e.g., Johnson et al., 1985; Justen et al. 1990]. However, there is also widely recognized room for improvement. Collaborative learning does not work for all learners, and the results of instructional outcome studies are mixed [Klein & Pridemore, in press, Webb, 1987]. Fruitful student interactions are simply not a given. We can not expect learning gains to happen just because students are sitting together, at a desk or at a computer workstation. In Brown and Palincsar's [1989] words: "Social interactions do not always create new learning; peer interactions vary enormously; only some teaching environments actually create ideal learning experiences." (p. 397) Thus, the third motivation to develop CSCL systems is to improve the effectiveness of collaborative learning as an instructional format: i.e., to support peer interactions so as to increase learning gains.

These cognitive and social barriers to fruitful collaborative interactions raise an important challenge for teachers and developers of CSCL environments: to identify the features of collaborative learning situations that potentiate learning and to build learning environments which contain these features. At the outset of this work, we had been addressing this objective in two projects by designing and implementing tools which support question-asking, explanation, and critical discussion - i.e., the "knowledge articulation" activities which underlie social learning - particularly, for collaboration across networked machines. In the Belvedere project, our goal is to support students learning to engage in critical discussion of competing scientific theories. This work has resulted in a graphical argumentation environment in which students articulate and compare alternate theories and their associated arguments, and change them in response to new evidence or criticism [Suthers et al. , 1995; Suthers & Weiner, 1995]. The Sherlock troubleshooting environment for a complex electronics device contains tools to support peer critique of student solution traces. [Lesgold et al., 1992; Katz et al., 1993; Katz & Lesgold, 1994]. Students can step through a peer's solution, and select from a menu of "troubleshooting standards" those standards that the student did not follow, at a particular step of his solution. The software also has the capability to attach a menu of questions to domain objects, such as components in circuit diagrams.

Several system developers have been building instructional software which is specially tailored for collaborative use - i.e., CSCL systems [McManus & Aiken 1993; Newman 1991; Scardamalia et al. 1992; Whitelock et al. 1991]. However, none of these systems, to our knowledge, were being built upon a foundation of empirical research on collaborative cognition in the target domain of instruction. We believe that it is critical to ground the development of CSCL systems in research which addresses issues such as the following: What types of knowledge do students seek from their peers while collaborating on problem-solving activities? Are there any patterns in the types of knowledge that students can/can not readily explain to their peers? To what extent do students ask the right questions at the right time; i.e., do they know what they need to know in order to overcome an impasse during problem solving? What is the nature of human tutors' support for students, during collaborative problem solving and individual or collaborative critique of peer solutions? For example, what is the content and structure of human tutors' explanations for particular types of questions? Is human coaching primarily directive or question-driven? To address these questions, we had been working with small groups of students engaged in analysis of scientific debates in the Belvedere environment. We had also begun to carry out observational studies of students working together on Sherlock problems, and critiquing traces of peer solutions, with system coaching suppressed but a human tutor available to ask questions to when they are stuck (i.e., unable to help each other). This work prepared us to inform the development of a computer model of support for peer collaboration during particular types of activities (e.g., peer critiquing, collaborative problem solving).

A major motivation was the concern that knowledge-based educational software, such as intelligent tutoring systems, have historically been large, self-contained programs with specialized platform requirements. We saw that to make these technologies viable, we must be able add component functionality incrementally, and enable systems to interoperate with commercial software and internet resources [Brusilovsky, et al. 1996; Ritter & Koedinger, 1995; Roschelle & Kaput, 1995]. We knew that to reduce the cost of materials prepared by developers, and to enable greater collaboration between users, representations of educational materials should be shareable between diverse applications across the internet. Interoperability and reuse considerations suggested a "lowest common denominator" approach, yet we did not want to limit support for more advanced functionality such as domain-specific coaching. We also saw a great need to leverage productive software engineering technology, standard off-the-shelf commodities, and standard information packaging protocols in order to make educational technology more affordable. We had refined our software engineering capabilities toward this end, and planned to leverage SGML and especially its World-Wide Web expression in our work.

Thus, we started in a good position to develop and evaluate intelligent network-based technology for supporting critical discussion, problem solving, and use of on-line information resources, within both declarative work environments, such as texts and arguments, and procedural resources, such as simulations and problem-solving task environments.

2.3 Methods, Assumptions, and Procedures

2.3.1 Overview of Belvedere

• A collaborative inquiry database that students use to keep track of their inquiry process, including a statement of the problem, hypotheses that have been proposed and evidence offered for and against them, as well as references to information resources and experimental records.
• A Java-based "inquiry diagram" interface to this database, which helps students visualize the important ideas in a debate as concrete objects that can be pointed to, linked to other objects, and discussed;.
• HTML-based interfaces to the database, to enable access when Java tools are not available and to support tabular and textual views on the record of inquiry.
• A coach that is designed to stimulate students' contributions to the critical inquiry process.

• HTML-based reference materials that are structured to scaffold the critical inquiry process, and annotated to support coaching.

We use a diagrammatic interface for cognitive, collaborative, and evaluative reasons. First, the cognitive: concrete representations of abstractions turn conceptual tasks into perceptual tasks. Thus the diagrams help students "see" and internalize these abstractions and keep track of them while working on complex issues. Second, the collaborative: diagrams support collaboration by providing a shared context and reference point. Third, the evaluative: student-constructed diagrams provide the teacher and the computer with a basis for assessing students' understanding of inquiry in general and of a topic area in particular. These three reasons are discussed further below.

2.3.1.1 Cognitive Support

2.3.1.2 Collaborative Support

2.3.1.3 Evaluative Support

2.3.1.4 Other Features

2.3.2 Architecture

Each of the following sections begins with a general characterization of the corresponding level of description, followed by an informal description of our application or architecture at that level, and a summary of mappings to other levels of description. At each level we discuss reusability and interoperability concerns, and the advantages and disadvantages of our design.

2.3.2.1 Concepts of Application

2.3.2.1.1 Critical Inquiry in Science

1. Familiarizing oneself with a field of study
2. Identifying a problem of interest
3. Proposing hypotheses (or solutions)
4. Identifying and seeking evidence that bears on those hypotheses (or solutions)
5. Drawing conclusions based on the evidence found
6. Summarizing and reporting the inquiry to others
7. Evaluating the status of the inquiry, with repeat at any of the steps above
8. Discussing and coordinating the doing of 1-8 with others.

1. Obtaining solicited and unsolicited guidance on how to conduct critical inquiry

1. Lack of motivation.
2. Limited knowledge of scientific domains.
3. Inability to recognize abstract relationships implicit in scientific theories and arguments about them.
4. Difficulty keeping track of a complex debate.

1. Lack of scientific argumentation criteria, and associated biases, e.g., confirmation bias.

2.3.2.1.2 Generality

2.3.2.2 Interface Presentation

2.3.2.2.1 A Graphical Interface for Critical Inquiry

Belvedere is designed to be used in conjunction with materials presented in a Web browser. The materials are segmented into units at a granularity which a subject matter expert chooses for his or her own inquiry diagrams. "Reference This" buttons in the Web pages enable students to send "references" to these segments into the Belvedere "in-box" (upper right of Figure 2) from where they may be dragged into the inquiry diagram as needed. The small icons in the upper left of each shape indicate that hyperlinks can be followed back to the original document.

2.3.2.2.2 Relations to Concepts of Applications

Summarizing from Suthers et al. [1995] and Suthers & Weiner [1995], here is how the interface is designed to address barriers to learning critical inquiry:

1. Lack of motivation: Belvedere is designed to support collaborative problem solving, providing peer motivation and engaging activities [O'Neill & Gomez 1994; Scardamalia & Bereiter 1991; Slavin 1990]. Support for collaboration includes networked WYSIWYS, the chat facility, and the diagram itself, which helps students switch between working independently and working together without losing track of what they are doing.
2. Limited knowledge of scientific domains: This is addressed in part through on-line materials, and in part through "expert coaches" which can coach based on the knowledge of a particular domain.
3. Inability to recognize abstract relationships and arguments: Belvedere's diagrammatic representations reify these relationships and make weaknesses and points where further contributions can be made salient [Smolensky et al 1987; Streitz et al 1989].
4. Difficulty keeping track of a complex debate: This is partially addressed by the concrete visual representation, which help students keep track of main points and pending issues.

1. Lack of scientific argumentation criteria, and associated biases: This is addressed by Belvedere's coach.

1. Familiarizing oneself with a field of study: Browsing the Web materials.
2. Identifying a problem of interest: Starting a new inquiry diagram, labeled by a problem statement.
3. Proposing hypotheses: Either selecting the "hypothesis" icon and typing in a statement of a hypothesis, or using a "reference this" button to bring a reference to an existing hypothesis into the diagram.
4. Identifying and seeking evidence that bears on those hypotheses: The coach helps users identify when evidence is needed. The Web materials themselves along with hands-on activities suggested in those materials provide some sources of evidence. Evidence is recorded as for A3, except the "Data" icon is used.
5. Drawing conclusions: Belvedere provides a facility for changing and viewing the relative "strength" of the different statements. The coach provides some guidance, but further support is needed here.
6. Summarizing and reporting the inquiry to others: Currently support is inadequate. Users can print their inquiry diagrams, or convert them into HTML tables that summarize the evidence for and against each major hypothesis.
7. Evaluating the status of the inquiry, with repeat at any of the steps above: The coach provides some local guidance. Also we provide an outline of phases of activity and a "Guide" menu to help students through these phases.
8. Discussing and coordinating the doing of 1-8 with others: If not in co-located, users can interact via the Chat window.

1. Obtaining solicited and unsolicited guidance: The coach provides both.

2.3.2.2.3 Comments on the Analysis

2.3.2.3 Concepts of Operations

2.3.2.3.1 Supporting Collaborative Coached Critical Inquiry

Inquiry Diagrams. Inquiry diagrams consist of a problem statement, and a collection of statements and relationships between them. The operations abstract communications between the Belvedere interface and a persistent object store. Some of these are New Inquiry Diagram (A2), Open Inquiry Diagram (A2), Add Statement (A3, A4), Add Relationship (A3, A4), Update Statement (A5, A7), and Delete Statement or Relationship (A5) (we retain a complete history of all objects that existed).

Information Search. Accomplished by Get Page (A1) and Send Reference (A3, A4), invoked via the Web browser.

Discussion with Others. A8 is accomplished by Send Message.

Advice Services. Objects include requests, replies, and interruptions; all in support of A9. The client can Request Advice; and the coach can Send Advice, which consists of the advice text and a list of the objects that the advice text refers to. The coach can also send an Interruption, which is a request to perform an interface action that notifies the user that advice is available.

Some important Concepts of Application activities are not supported by this model. These include performing data analysis and visualizations (A4, A5), asking the coach specific questions (A9), and abstracting summaries of the inquiry (A6). Extensions are being planned to address these concerns.

2.3.2.3.2 Relations to Other Levels

2.3.2.3.3 Interoperability and Reusability

2.3.2.4 Abstract Implementation

1. Browsing (Get Page)
   1. Client request (HTTP)
   2. Server reply (HTML with embedded Java)
   3. Access logging. (When implemented, Tracker will notify Coaches.)
2. Referencing On-line Materials (Send Reference)
   1. Java applet sends reference to server (data embedded in CGI GET)
   2. Reference sent via socket in application specific protocol
3. Application Requests and Updates (New Inquiry Diagram, Open Inquiry Diagram, Add Statement, Add Relationship, Update Statement, Delete Statement)
   1. Request or update sent to Session Server (data embedded in CGI GET)
   2. Server queries or updates Database (SQL requests)
   3. Database replies with results or return code
   4. Reply sent to client (response to CGI GET. User was able to continue working before reply received.)
4. Updates Propagated to Other Clients (WYSIWIS for events generated in #3)
   1. Update sent to Session Server (subset of 3a)
   2. Connection Manager informed of update (TCP socket; application specific protocol)
   3. Connection Manager formats message and informs all clients that are using the same workspace (TCP socket; application specific protocol)
5. Coaching (Request Advice Send Advice)
   1. Update or advice request sent to Session Server (data embedded in CGI GET)
   2. Update or advice request sent to Coach dispatcher (TCP socket; application specific protocol)
   3. Coach queries Database if needed to determine state (SQL, read-only)
   4. Database replies
   5. Coach sends client advice, if requested. Coach sends interrupt when update activated high priority advice (TCP socket; application specific protocol)
6. Chat Facility (Send Message)
   1. User's comment sent to Session Server (data embedded in CGI GET)
   2. Session Server sends comment to Connection Manager (TCP socket; application specific protocol)
   3. Connection Manager forwards to users in same workgroup (TCP socket; application specific protocol)

2.3.2.4.1 Relations to Concepts of Operations

2.3.2.4.2 Interoperability and Reusability Issues

The above design is limited in several ways. Some of the protocols are application specific. This is probably unavoidable; although some reuse may be facilitated by shared ontologies at the Concepts of Operations level. Under new PTI funding, we have begin another cycle of redesign to enable delivery using other databases and other server class machines. Prototype versions of RMI and CORBA server interfaces have been implemented and are currently undergoing testing and evaluation. Our new design will also greatly simplify the addition of new types of clients. (We plan to add clients that manipulate influence diagrams, causal loop diagrams, and concept maps.) Under the new design the protocols are data-driven, so that only minor modifications to the Session Manager (and no other existing components) are required to add a new type of client. Each client would load a data type table into the Database.

We are attempting to generalize the abstract implementation architecture to be configurable for any learning application that requires networked collaboration, coaching, and multimedia. Adaptive multimedia [Brusilovsky et al, 1996] could be included with scripts that automatically generate HTML pages from the database to meet user's needs. We have designed and implemented a prototype of this adaptive hypermedia extension but have not incorporated into our released system. Student modeling facilities would be improved by informing the Coach of which materials students have examined via the Tracker.

2.3.2.5 Resource Layer

Client platforms: Any platform supporting Java applications and Netscape 2.0 or better. We have tested on Mac OS, Solaris, Windows'95 and Windows NT. Installations in our lab and in DoDDS schools were on PowerMacintosh 8100 series and various Pentium platforms. Applications shown in the shaded box in Figure 3 must be running at the same IP location.

Server platforms: A Unix server was required. The redesigned version will deliver on Windows NT and other server class machines. The server was installed on a Sparcstation 20 MP in our lab and on Netras in each of the 4 DoDDS schools. The server components shown in shaded boxes must reside at the same location as others in the box.

Network requirements: With the possible exception of images embedded in HTML materials, all messages are small, so communication load is low. A 28.8 connection was adequate. Installations were all 10BaseT.

2.3.2.6 Architectural Lessons

2.3.3 Design of Computer Coaches

2.3.3.1 Introduction

In this section we discuss two methods of advice generation that we have implemented. Syntactic advice strategies make suggestions based solely on the syntactic structure of students' inquiry diagrams. Consistency-based advice strategies use a simple knowledge base of consistency relations between information units to identify information that may challenge or corroborate relationships postulated by the students. Before describing these advice giving methods, we first briefly describe the design constraints under which we operated.

2.3.3.2 Pedagogical Constraints on Advice

Maintain the student-initiated character of Belvedere's environment. Belvedere encourages reflection by allowing students to see their argumentation as an object. They can point to different parts of it and focus on areas that need attention. They can engage in a process of construction and revision, reciprocally explaining and confronting each other. An advisor should not intervene prematurely in their thinking process. It should be discreet, offering advice on request. Students should feel free to discard an advisor's suggestions when they believe them to be irrelevant or inappropriate.

Address certain parts of the task that are critical to the desired cognitive skill. Research on "confirmation bias" and hypothesis driven search suggests that students are likely to be concerned with the process of constructing an argument for a favored theory they are supporting, sometimes overlooking or discounting discrepant data [Klayman & Ha 1987; Chinn & Brewer 1993]. Also, they may not consider alternate explanations of the data they are using. An advisor should address these problems. For example, it should offer information that the student may not have sought, including information that is discrepant with the student's theory.

Be applicable to problems constructed by outside experts and teachers. The advisor should be able to give useful advice based on a simple knowledge base that an expert or a teacher might construct. So far Belvedere has been used to construct arguments in domains as different as theory of evolution, contrasting theories of mountain formation, cause of the Cretaceous extinctions, whether HIV causes AIDS, and theories in social psychology. It is not feasible to develop for each a representation of the knowledge needed to deal with the argumentation students potentially could engage in. We are instead interested in a general approach, applicable to all the cases, in which the knowledge base can be constructed by a teacher.

2.3.3.3 Syntactic Advice Strategies

The advice applicable to a given inquiry diagram is often more than a student can be expected to absorb and respond to at one time. When more than one instance of advice is applicable, a preference-based quicksort algorithm is used, following a mechanism used by Suthers [1993] for selecting between alternate explanations. Advice instances are sorted in priority order, and the highest priority advice is given. Objects that bind to variables in the patterns are highlighted in yellow when the advice is given, so the user can easily identify what the advice is about. If further advice is requested before the diagram changes, subsequent advice instances on the sorted list are used without reanalysis. We are investigating preferences that take into account factors such as prior advice that has been given, how that advice has been responded to, how recently the object of advice was constructed, and various categorical attributes of the applicable advice.

We believe that the most important kind of advice is that which stimulates and scaffolds constructive activity on the part of the students. To give this kind of "open world" advice, our first step was to identify partial argument patterns in the inquiry diagram the students had constructed so far and indicate how the student could complete these patterns. For example, the advisor might find theoretical claims that have no empirical support and suggest that support be sought (hypothesis-lacks-empirical-evidence), or it might find competing theories that are both supported by the same empirical evidence, and ask if there is a discriminating piece of evidence (discriminating-evidence-needed in Table 1).

 (def-advice 'HYPOTHESIS-LACKS-EMPIRICAL-EVIDENCE
  :query '(retrieve (?h) (and (hypothesis ?h) (No-Evidencep ?h)))
  :advice ("Can you find data that are for or against this hypothesis?
A scientific hypothesis is put forward to explain observed data. Data
that a hypothesis explains or predicts count *for* it. Data that are
inconsistent with the hypothesis count *against* it.")

  :subsequent-advice ("Can you find some data for or against this
hypothesis?")
  :advice-types '(incompleteness))

(def-advice 'DISCRIMINATING-EVIDENCE-NEEDED
  :query '(remove-duplicate-binding-sets 
           (retrieve (?h1 ?h2)
		(and (hypothesis ?h1) (hypothesis ?h2)
		     (:not (:same-as ?h1 ?h2))
		     (Exists-Consistent-DataP ?h1)
		     (Exists-Consistent-DataP ?h2)
		     (fail (Consistent-HypoP ?h1 ?h2))
		     (Identical-EvidenceP ?h1 ?h2))))
  :advice ("These hypotheses are supported by the same data. When this
happens, scientists look for more data as a \"tie breaker\" -- especially
data that is *against* one hypothesis. Can you produce some data that
would \"rule out\" one of the hypotheses?")
  :subsequent-advice ("Can you produce some data that might support just
one of the hypotheses?")
  :advice-types '(incompleteness evaluative))

(def-advice 'CONFIRMATION-BIAS
  :query '(retrieve (?h) (and 
		           (hypothesis ?h)
                           (Exists-Multiple-Consistent-DataP ?h)
                           (Multiply-LinkedP ?h)
                           (fail (Exists-Inconsistent-DataP ?h))))
  :advice
  ("You've done a nice job of finding data that is consistent with this
hypothesis. However, in science we must consider whether there is any
evidence *against* our hypothesis as well as evidence for it. Otherwise
we risk fooling ourselves into believing a false hypothesis. Is there
any evidence against this hypothesis?")
  :subsequent-advice ("Don't forget to look for evidence against this
hypothesis!")
  :advice-types '(cognitive-bias))

(def-advice 'ALTERNATE-HYPOTHESIS
  :query '(retrieve (?h) (hypothesis ?h))
  :lisp-test '(lambda (client query-result) 
           (declare (ignore client)) 
           (= (length query-result) 1))
  :advice ("Scientists consider many hypotheses to get the best
explanation of the data they are interested in. If they don't compare
their favorite idea to other ideas, somebody else will! Is there another
hypothesis that you could consider?")
  :subsequent-advice ("Is there another hypothesis that you could
consider?")
  :advice-types '(cognitive-bias incompleteness))

(def-advice 'SWALLOW-DOES-NOT-A-SUMMER-MAKE
  :query '(retrieve (?d ?h)
             (and (data ?d) (hypothesis ?h)
                  (Consistent-HypoP ?d ?h)
                  (fail (Exists-Multiple-Consistent-DataP ?h))
                  (fail (Exists-Inconsistent-DataP ?h))))
  :advice-template 
  ("Strong hypotheses and theories usually have a lot of data to support
them. However, this hypothesis has only one consistent data item. It
looks rather weak. Can you find more data for this hypothesis? Can you
find data that is against it?")
  :subsequent-advice
  ("This hypothesis has only one consistent data item. Could you find
more data for (or against) this hypothesis?")
  :advice-types '(evaluative incompleteness))

The syntactic advisor also responds to illegal and incoherent constructions. "Illegal" constructions are those that use elements of the diagrammatic language in a manner inconsistent with their intended semantics. For example, a "support" link should not be used between data. "Incoherent" constructions are those in which the elements are each used legally, but in combinations that are semantically problematic. Examples include a loop of "support" links, or a datum that both supports and undermines the same claim. Some other advice given includes:

1. Suggests that a theory or hypothesis be formulated when none is present in the inquiry diagram.
2. Asking whether the students can find empirical evidence relevant to a hypothesis that has no relations to empirical observations, or show that it predicts or explains an observed phenomenon.
3. Asking whether the students can say how an unconnected statement relates to the rest of the diagram.
4. Asks whether there is another theory that provides an alternate explanation for the empirical data when only one theory or hypothesis is involved in the inquiry diagram;.
5. Asks whether data can be found to discriminate between two theories that have identical support.
6. Asking whether a "data" box that supports one hypothesis could possibly support another. ("Data" is a kind of "Empirical Observation." This is an instance of support-competitor.)
7. The "one swallow does not a summer make" rule suggests that a good hypothesis explains more than one datum, and asks whether the student can find more data.

2.3.3.4 Consistency-Based Advice Strategies

The consistency-based advisor is our first step in this direction. It is intended to offer specific information that the student may not discover on her own. It makes two assumptions: students construct their inquiry diagrams from existing units of text, and these units are annotated with relationships recording whether they are consistent or inconsistent with each other, based on expert judgment. The advisor searches the latter "consistency graph" to find paths between units that students have used in their inquiry diagrams, and selects other units found along those paths which are brought to the students' attention. Our claim is that this enables us to point out information that is relevant at a given point in the inquiry process without needing to pay the cost of a more complete semantic model of that information.

2.3.3.4.1 Consistency and Transitivity

1. A&B is consistent with each of A and B individually.
2. If C is inconsistent with A or inconsistent with B then C is inconsistent with A&B.

When the teacher or domain expert defines the task and the information needed, she can draw an inquiry diagram in Belvedere. The diagram is easily transformed into a set of consistency relations between pairs of texts, following the rules described above. These relations become the "expert model" or knowledge base for the consistency-based advisor.

During a student session, students can drag texts authored by the expert into their diagrams, and express argumentation relationships between them. The links in the student diagram are interpreted as consistency relationships in the manner described above. The advisor can then compare consistency relations defined by the students with consistency relations defined by the expert, and look for inconsistencies and other possible advice. The comparison is based on a graph-search algorithm, which has been implemented and tested as reported below. It searches in the expert's consistency graph for paths between nodes that have been related in the students' graph, constrained by the following rules:

1. Only the shortest path between two nodes is considered.
2. A "positive path" crosses only consistency links.
3. A "negative path" ends with an inconsistency link.
4. A path can cross only one inconsistency link. An inconsistency link ends a path.
5. Conflicts between positive and negative paths are resolved in favor of the negative path.

Rules 2 to 4 are used to maintain the consistency of the path. Once a negative link is crossed, it is plausible that the two nodes at the end of the path are inconsistent. We cannot extend the path any farther, because whatever conclusion is drawn from there is quite arbitrary. For example, suppose that A is inconsistent with B, and B is inconsistent with C. We can't conclude that A is inconsistent with C: they could be consistent components of an argument against B. On the other hand, we can't assume that A is consistent with C: they could be arguing against B based on incompatible assumptions. Thus, Rule 4 forces the search to stop when a conflict is reached. Rule 5 is introduced to address the confirmation bias.

2.3.3.4.2 Formative Experiments

Figure 4. Example advice consistency path.

2.3.3.4.3 "Snippets"

Figure 5. "Snippets" may be referenced by clicking on document icons.

We have presently encoded extensive domains of documents on two topics into HTML-based materials databases: What caused mass extinctions? and What advice should be given to a person with a family history of a genetic disorder? The Mass Extinctions database, for example, contains about one hundred snippets. A student or teacher would select a pre-configured "snippet" from an HTML document while browsing the materials database using the Netscape browsing tool. Once such a snippet has been selected, a Java-based "applet" is activated that notifies the Belvedere client tool that the snippet has been selected. A visual icon known as the "In-Box" in the Client is highlighted, indicating that the material from this snippet can be relocated into the inquiry diagram. A list of such selections is maintained in this In-Box. Students also have the option to discard snippets present on the In-Box list. A dialogue box is again provided in which the student can check the snippet summary and modify its type, if necessary. When the student retrieves the snippet from the In-Box, the student-written label (summary) appears in the appropriate hypothesis, data, or principle shape as defined by the student.

2.3.3.4.4 Implemented Expert Coach

The student cannot see the expert diagram during a session. The expert diagram is thought of as a "read-only" entity and is configured by the teacher before an inquiry session begins. The student diagram is dynamic; each time a change occurs in a student diagram, the change is noted by the expert coach and the LOOM knowledge base is updated with the new information.

As students construct an inquiry diagram, they may include snippets from the materials database. The expert coach is utilized only when a student assigns a relationship between two snippets with a "for", "against", or "and" link. The expert coach can only provide advice about what it knows. Thus, if non-snippets are introduced into the student diagram the expert coach will be unaware of their existence, and it will have no advice about non-snippets created by the students. In that case, the argument coach still subjects the non-snippet objects to analysis and can respond. Meta-rules manage the advice from two different coaches before the advice is passed on to the student. As advice is generated from each coach, it is maintained on a list which is then subject to a recursive multi-keyed "preference" sort. For instance, expert advice is provided first, multiple instances of the same advice are reduced to single instances, and argument advice is sorted according to type, ranging from "getting started" (e.g., the inquiry diagram is empty) to "advanced" (e.g., a swallow does not a summer make). In this fashion, we envision future arbitration schemes to manage several such knowledge sources.

The expert coach has been implemented with a best-first heuristic search to determine the optimal path from the start node to the goal node in the expert diagram using the cost function

f(n) = g(n) + h(n)

1. The start and goal statements in the student diagram must be snippets and must also exist in the expert diagram.
2. The shortest path between the start and goal is always considered.
3. If the student has connected two snippets with a "for" link, and there is a path from the start to the goal in the expert inquiry diagram with an "against" link in it, provide feedback about the contents of the two statements connected by the "against" link. The "against" link in the path closest to the start node is always preferred.
4. If the student has connected two snippets with an "against" link, and there is a path from the start to the goal in the expert inquiry diagram that consists entirely of "for" links, provide feedback about the contents of each statement in that path.
5. If either of the statements found in a path in the expert diagram is the start or the goal do not provide feedback about it.
6. If a student has indicated a "for" link between the start and goal and the expert diagram has a direct (i.e., no intervening links) "against" link between the start and goal , ask the student to consider the ramifications of an "against" link.
7. If a student has indicated an "against" link between the start and goal and the expert diagram has a direct (i.e., no intervening links) "for" link between the start and goal , ask the student to consider the ramifications of a "for" link.

2.3.3.4.5 Reimplementation in Java

2.3.3.5 Coaching Status and Future Directions

Although we have selected an appropriate level of representation, the snippet, to allow the student to access domain-relevant material, we are considering the pedagogical value of both a finer and a coarser grain size. A finer grain would reduce ambiguity and increase the accuracy of feedback. On the other hand, a coarser grain, i.e., at the level of a normal paragraph, or of a typical Web document, would enable quicker authoring of the Web-based materials described earlier. Currently the expert's specification of the relations is a major bottleneck for complex domains. The model of coaching with a larger grain size would be an "FYI" coach, which would function like a research librarian forwarding new information to those likely to be interested in it. It would still be possible to specify "for" and "against" relations in a general sense, just as a paper can give evidence for or against a particular view. However, coarse-grained representation has obvious limitations. For example, it is important for students to learn that one can often extract evidence for a view from a context that is generally unfavorable. Indeed, scientific papers are obliged to take note of divergent views and limitations. We are also considering exposing the student to sub-graphs of the expert diagram. We are exploring models of learning and cognitive/perceptual mapping for the novice and expert, regarding the information realized in the diagrams the web-based materials [e.g., Petre 1993].

2.3.4 Design of Classroom Implementation

Consistent with these views, pilot studies with Belvedere [Suthers & Weiner; 1995] indicated that there was a need to structure the roles and activities of students working with Belvedere (see also [Waugh & Levin, 1988]). With DoDEA teacher colleagues, we developed a classroom implementation methodology focused on collaborative problem solving by small groups of students. The methodology calls for changes in the classroom environment, teacher's role, curriculum materials, student activities, and assessment methodology. Students work in teams to investigate real-world "challenge problems," designed to match and enrich the DoDDS curriculum with attention to National Science Education Standards [National Academy of Sciences, 1996]. The teams plan their investigation, perform hands-on experiments, analyze their results, and report their conclusions to others. Our classroom activity plans provide teachers with specific guidance on how to manage these activities with different levels of computer resources. Teachers and students are provided with assessment instruments designed as an integral part of the curriculum. Assessment rubrics are given to the students at the beginning of their project as criteria to guide their activities. They guide peer review, as well as helping the teacher assess nontraditional learning objectives. In this section we describe and comment on this methodology as it was carried out in our most exemplary case, in a Wurzburgh general science class.

2.3.4.1 Classroom Environment

2.3.4.2 Teacher's Role

2.3.4.3 Curriculum Materials

Figure 6. Web-based Materials for Challenge Problem

As shown in Figure 6, two menus are provided with the web based materials. A domain independent menu (left side) guides students through five phases of inquiry, providing suggestions on how to conduct scientific inquiry and how to use the Belvedere software in this process. Another menu (bottom) provides domain specific links organized in a manner relevant to the phases of inquiry. For example, students are provided with a link to a glossary of terms; access to simplified versions of articles on scientists' hypotheses, methodology, and field reports; and a link to experiments involving both hands-on manipulatives and computer simulations. The Web-pages contain "reference" icons resembling text pages (two are seen in Figure 6, one preceding each paragraph of text), which enable students to send text found on these pages into the inquiry diagram's "in-box."

2.3.4.4 Student Activities

What you learn  How you learn it   How you tell how well you learned           
to do                                                                          
                                   Poor    The inquiry diagram contains one     
                                           appropriate hypothesis and no        
                                           related data.                        
To formulate    Create Belvedere   Fair    The inquiry diagram shows one        
and revise      inquiry diagrams           appropriate hypothesis and one       
scientific      that record                data supporting it.                  
explanations,   different          Good    The inquiry diagram shows one        
and to use      hypotheses about           hypothesis with the use of           
evidence to     a problem,                 evidence for it as well as against   
                                           it.                                  
develop a       different data     Good    The inquiry diagram shows several    
logical         that can help              hypotheses each connected to         
argument.       you decide                 multiple pieces of data.             
                between                                                         
                the hypotheses,    Great   The inquiry diagram shows multiple   
                and the                    hypotheses with the use of           
                relationships              evidence for as well as against      
                between the data           each of these hypotheses.            
                and hypotheses.                                                 
                                   Great   The inquiry diagram indicates        
                                           additional information the student   
                                           would look for to support or to      
                                           refute explanations.                 
                                   Poor    The inquiry diagram only contains    
                                           information that is drawn from       
                                           personal experience or speculation.  
To develop a    Find out what      Good    The inquiry diagram contains         
model that      specialists in             references to information from       
integrates      different                  only one discipline, for example     
concepts from   disciplines                Geology, Physics, Chemistry, or      
                think of the               Biology.                             
                problem.                                                        
multiple        Look for           Good    The information in the inquiry       
domains with    information from           diagrams come from one kind of       
different kinds different                  resource, for example only from      
of data.        resources, such            experiments, field observations,     
                as on-line and             or articles.                         
                library            Great   The inquiry diagram contains         
                articles,                  references to information from       
                experiments you            multiple disciplines such as         
                do, and field              Geology, Physics, Chemistry,         
                observations.              Biology.                             
                                   Great   The information in the inquiry       
                                           diagrams are drawn from multiple     
                                           resources, such as experiments,      
                                           field observations, and articles.    

2.3.4.5 Supporting Peer Evaluation with Performance-based Assessment

2.3.5 Collaborative Activities Funded

2.3.5.1 Belz and Luckham: Architecture Abstraction Hierarchy Reference Model

                 Application Model                   
               Abstract System Model                 
    Interaction Model      Conceptual Model           
           Abstract Implementation Model             
                  Resource Model                     

Application Model:

This level describes the application domain and identifying concepts used to define the domain and to organize work in the domain. This level is described largely in its own terms (as practitioners view it), and includes theories and analysis of the domain, educational objectives (or other task objectives), task analysis, and identification of the objectives and tasks which the software is expected to support.

Abstract System Model:

This level describes the user's (and designer's) interactive and conceptual model of the system; the "high level" architecture of the system [Brooks], defining the overall effect of the system on the user, its function and its character. This level is divided into two distinct aspects to distinguish two essentially different sets of issues; for humans, things and models of them have not only an immediate, perceivable, manipulable part but also an indirect, understandable part.

Interaction Model:

A view describing the perceptual characteristics of the system, portraying the system and its functionality to the user, along with the corresponding facilities of the system to interact with the user. This level addresses all perceptual concepts pertinent to the system and defines the perceivable entities chosen to make up the interface and interactions available to the user with these entities. It is expected that this level will reify or otherwise reflect the pertinent perceptual and interactive aspects and components of the Application Model.

Conceptual Model:

A view describing the inherent functionality and character[7] of the system; the way that the user conceives of the system. It also delineates the means by which the software provides for modeling the application and application domain, i.e. the software strategy. It does this, for example, in terms of the style or type of system, along with the user "visible" classes of objects and the operations that can be performed on them. The Conceptual Model parallels the desired subset of a given Application Model, defining abstractly the automation strategy (theory) chosen to realize the system. It is the way the user conceives of the system and how it operates.

Abstract Implementation Model:

This level describes the structural framework of the implementation, delineating architectural elements and communication between these elements, including software modules such as interpreters, databases, event managers, etc., and data and control flow between them. This level describes the high level implementation architecture[8] of the system. The Interaction Model and Conceptual Model entities and interactions are allocated (mapped) to events and representations in portions of the Abstract Implementation Model during the modularization part of the software design process.

Resource Model:

This level describes the resources required to actually implement the system according to the Abstract Implementation Model. This level enables explicit description of implementation and performance issues (and models) used in designing the performance (and other non functional characteristics) of the system. This level addresses implementation issues that affect design trade-offs, and therefore can profitably be addressed prior to implementing the system. Issues considered here include physical architecture, location of services, algorithm complexity, resource constraints, bandwidth, latency, etc.

2.3.5.2 Forbus and Koedinger: Integrated Feasibility Demonstration of Science Learning Spaces

2.3.5.2.1 Demonstration Architecture

Figure 10. Learning Space Demonstration Architecture.

Subsequently, we added the Belvedere system and Argumentation Coach (left side of Figure 10). Belvedere's communication architecture (abstracted in Figure 10 as the "BORBI" but described more fully in section 2.3.2.4) is itself capable of supporting component-based composition of functionality. However, we decided to use the MOO due to its prior use in the first demonstration. Integration of the Belvedere subsystem into the MOO required the addition of one translator component (the other small box in the figure): no modification to Belvedere itself was required. The translator watched the MOO for Hypothesis and Simulation Run objects sent by the Simulation Interface. When seen, these were converted to Belvedere Hypothesis and Data objects and placed in the user's "in-box" for consideration.

2.3.5.2.2 Lessons Learned from the Integration Demonstration

This experience illustrates some tradeoffs and limitations of a purely "plug and play" approach to component based systems, while also showing that there is hope given further research. We showed not only how to reduce the effort required to "hook up" diverse components, but also the value of sharing semantics between applications. Information objects created with empirical and theoretical identities in one application (Active Illustrations) retained that identity in how they were treated in another application (Belvedere and its Argumentation Coach). Furthermore, a third Tutor Agent treated these objects as having the same semantics in both situations. Consistent treatment of the learner's representations by different software agents reinforces the semantics that we want learners to reflect upon and manipulate. Perhaps most significantly, the contextual semantics of these objects accumulate as they are used: an object, viewed in one context (e.g., evidence maps), "stands for" the learning interactions centered on it in another context (e.g., the simulation) Critical to this accumulation of contextual semantics is persistence of identity. Special attention was required to ensure that the learner "sees" the thing that shows up in Belvedere as the same object she constructed in Active Illustrations.

2.3.5.3 Workshop: Architectures and Methods for Designing Cost-Effective and Reusable ITS

2.4 Results

2.4.1 Highlights of Project

2.4.1.1 Cognitive/Education Highlights

• Belvedere scaffolds critical inquiry with "inquiry diagrams," automated coaching and cognitively motivated organization of materials and activities.
• Supports collaborative learning through shared visual reification of ideas and relationships under discussion, and chat facility for distant collaboration situations.
• Science "challenge" activities are designed to match and enrich the DoDDS curriculum and textbook.
• Challenge activities are supported by on-line materials, hands-on activities and simulations, both individual and collaborative work, interaction with classes at other sites, and peer evaluation based on critical inquiry rubrics we supply.

2.4.1.2 Technology Highlights

• Supports proximal, distal, synchronous, or asynchronous collaboration through shared, persistent diagrammatic workspaces and chat facility. Workspaces and chat visibility are organized by workgroups.
• Database representation of discussions can support multiple media representations, for example, diagrams of arguments and HTML tables summarizing evidence.
• Automated "idea generator" (coach) provides users with suggestions on how to improve their arguments and other lines of investigation to consider. Includes both general but "weak" and stronger but domain-specific coaching heuristics.
• Reusable client-server architecture delivers advanced educational technology on small platforms by putting "heavyweight" functionality and shared resources on server. Clients need only interface and networking software (Java and Netscape).

2.4.1.3 Implementation Status and Platforms

2.4.1.3.1 Belvedere

2.4.1.3.2 Collaborative Database Server and Connection Manager

2.4.1.3.3 Coach ("Idea Generator")

2.4.1.4 Maturity and Deployability

• Usability studies have been conducted in our lab.
• Belvedere client software is available to anyone on request, and has been requested by and sent to 23 sites to date.
• Complete client/server system deployed in 5 classrooms in DoDDS schools, and is available for other deployment opportunities.

2.4.1.5 Openness and Interoperability

2.4.1.5.1 Points of Interoperability:

• Most communications use standard HTTP and SQL protocols.
• Web-based materials can be imported into Belvedere client in one of two ways:
  • Java applets embedded in web pages send objects to client's "in box" (imports semantics but special preparation of web pages required).
  • Cut and paste supported by OS (imports text only but works on anything).
• Diagrams are stored in database server as objects and relations that can be used to generate other document types. For example,
  • we currently generate HTML tables that summarize the diagrams, and
  • we plan to generate a format readable by word processors.
• Database can be extended to accommodate other object types, thereby supporting other specialized applications with identical infrastructure.
• Additional or alternate coach servers can easily be added with CGI scripts that form the communication link.
• Web-based materials link to MS Word and other standard tools.

2.4.1.5.2 Limitations on Interoperability:

• Object protocol between Belvedere client and server has idiosyncratic syntax for objects.
• Protocol involving Coach (via TCP/IP) has idiosyncratic syntax for events and advice.
• Would like to place hyperlinks (URLs) in diagrams and link to Netscape, but cannot yet contact running browser on Solaris and Windows, so a new one starts.

2.4.1.6 (Re)Applicability

Extended configuration to be available early 1997 , including inquiry diagrams, concept maps, causal loop diagrams, influence diagrams, and plan diagrams, will extend applicability to other phases of critical inquiry as well as other applications beyond education, including risk assessment, planning, and job skills analysis.

2.4.1.7 Venues and Collaborations

• Visited DoDDS schools 5/96, 9/96 and 10/96. Installed Belvedere client on Teacher machines
• Belvedere client distributed to 23 sites in first few weeks of public release.
• Collaborating with Belz (TRW) and Luckham (Stanford) on hierarchical modeling of architectures, using the Advlearn architecture as an example.
• Collaborated with Forbus and Koedinger on Integration Feasibility Demonstration.

2.4.1.8 Future Research

2.4.1.8.1 Technology Research

• Abstract the Advlearn architecture to be suitable for reuse in other application areas, and develop a hierarchical refinement methodology for working from educational objectives through a conceptual model of the domain to instantiation of an appropriate system in the architecture.
• Investigate how the utility of (a) diagrammatic reification of one's ideas and (b) automated coaching based on diagrams generalizes to other diagram types and their associated activities.
• Develop interface and advice-giving tools for helping users compare two representations and reconcile the multiple views they represent.
• Support user-constructed simulations of causal diagrams, and coach users on planning interesting experiments with these simulations.
• Investigate the extent to which natural language technology (including part of speech tagging and shallow "understanding") can be used to answer student questions, help automate the comparison of two diagrams, etc.
• Address the knowledge-based systems authoring bottleneck by seeking middle ground between knowledge engineering and writing uninterpreted text. Investigate use of NL technology here as well.
• Investigate the cost-benefit tradeoff between various types of coaching and the knowledge-engineering required to support them.

2.4.1.8.2 Cognitive/Education Research

• Continue to investigate how to coach critical inquiry, in particular getting students to pay attention to discrepant evidence.
• Continue to investigate the interaction between coaching and collaboration. How can we effectively coach collaboration? How can we coach critical inquiry in a collaborative situation?
• Begin investigation into coaching the use of simulations, as an example of the more general problem of coaching the design of effective experiments for testing one's ideas.
• Investigate the value of multiple representations and techniques for coordinating those representations in concept development and refinement.

2.4.2 Results of Independent Evaluations

2.4.2.1 Evaluation Methodology

2.4.2.2 Summary of Evaluation Results

Observations of student activity show that students were engaged and on task during the collaborative problems solving situations presented to them by the Belvedere comprehensive approach. Teachers indicated that the approach enhanced students ability to engage in collaborative tasks.

"Classroom observations of teachers and students using Belvedere show that it is being used to support cooperative problem solving, with students working in groups of 2 to 4 students. Students appeared to be engaged and on task. Teachers report that it is easy to use, and they find that it enhances students ability to engage in cooperative work, and to address scientific hypothesis testing in an organized and analytical way."[9]

"Students report that working with Belvedere makes it easier for them to organize and review the arguments for and against a specific scientific hypothesis. They also report that they find it easy to integrate work in Belvedere with work in other applications like Word and Excel and Web Browsers. Students using Belvedere generated artifacts that demonstrated integration of the knowledge representation maps generated using Belvedere with text and graphic information taken from a variety of resources, including the Internet, and with numerical data generated as a result of classroom activities."

"Data collected on the efficacy of staff development for teachers using Belvedere indicated that they were very satisfied with the training provided, and believed that they were well prepared to integrate use of the Belvedere software into their classrooms. The staff development provided for Belvedere compared very favorably with that provided by other application developers in the CAETI program.

2.5 Conclusions

Many of our conclusions are to be found throughout this report, especially in Section 2.3.2.6 (Architectural Lessons), 2.3.3.5 (Coaching Status and Future Directions), and 2.3.5.2.2 (Lessons Learned from the Integration Demonstration). Other conclusions are best reported in the form of recommendations for future work.

2.6 Recommendations

2.6.1 Follow-on Work for Belvedere

2.6.1.1 Port Server to NT

• Redesign of the "session manager":
  • An improved, uniform and extensible communication protocol based on existing standards.
  • Designed to hide the choice of database from other system components.
  • Implemented entirely in Java for portability and ease of maintenance.
• Redesign of the critical inquiry "coach":
  • Port from Lisp to Java, eliminating the need for Lisp license and large memory image.
  • Redesigned to use the communication protocol mentioned above.
• Port to NT:
  • Choose an NT-based database.
  • Modify the server's session manager to interface with the new database.
  • Perform any other changes needed to port the server to the NT-platform.
  • Test and fine-tune the server on the NT platform.

2.6.1.2 Refine and Extend the Belvedere Interface as Guided by User Feedback

• Stronger integration between Belvedere and other tools, such as Netscape, MS Word, and Excel. (Currently, Netscape documents can be referenced in Belvedere.)
  • Ability to follow hyperlink from Belvedere to Netscape without starting a new copy of Netscape (the current situation).
  • Embed links to MS Word documents and Excel spreadsheets directly in Belvedere.
  • Be able to translate Belvedere inquiry records into an outline form readable in MS Word.
  • Integration of inquiry diagrams and free-form textual discussion tools such as Netscape newsgroups.
  • Integrate multimedia in Belvedere diagrams.
• Provide alternate "diagram types" with custom "palettes" of one or more of the following types:
  • Concept mapping: students diagram relationships between concepts, using relations and concepts of their own choosing.
  • Causal loop diagrams: students model natural processes under study.
  • Mathematical models: students construct diagrammatic representations of mathematical relationships.
  • Planning inquiry: students diagram plans for inquiry activities, such as lab experiments.
  • Language Arts: students construct diagrams appropriate to their writing goal (such as "persuade," "report," "critique," and convert these diagrams into an outline for writing.
• Extensions of "idea generator" for above diagram types. The coaching software underlying the "light bulb" in Belvedere's interface would be extended to give advice on concept mapping, causal loop diagrams, and/or planning inquiry or writing.
• Simulations based on causal loop diagrams. Students would run simulations to see the implications of their causal models, particularly those containing positive and negative feedback loops. Simulations compiled into Java could be embedded in Web pages for others to try.
• Provide for linking between different diagrams, for example, enabling a hypothesis in an inquiry diagram to "open up" into a causal model.

2.6.1.3 Future Work for Coaching

2.6.1.4 Cognitively Motivated Design of Challenge Activities and Materials

• A library of Web-based authoring templates for particular genres of challenge materials, indexed for easy selection. Using an appropriate template, a teacher or other non-professional author could use basic authoring tools to quickly develop new curricular materials.
• A set of more abstract design principles. We would describe the cognitive and social implications of the various design choices available to developers of on-line materials, hands-on activities, and role structures for group activities.
• These templates and design principles will be made available to teachers and others in the DoDEA community on the DoDDS NT servers, and will be shared with publishing partners as appropriate relationships are established.

2.6.1.5 Realistic Classroom Implementation

2.6.1.6 Assessment Rubrics

2.6.1.7 Commercial Deployment

The more familiar of the two approaches is technology transfer to a commercial entity which would market and support a product derived from research software such as Belvedere. One barrier towards such technology transfer is the lack of regular contact between researchers and relevant commercial entities. Research labs should be facilitated in finding appropriate commercialization options.

An alternate, non-proprietary approach to long term support of educational software is being developed. This approach is known variously as the Educational Object Economy (NSF) and the Object Economy Model (Apple Computer). Reusable, platform independent software objects are shared and maintained in object repositories. The Repository is based on an innovative licensing scheme which provides software for free, provided that improved versions are made available in the repository under the same license, or that a royalty-paying license be negotiated (royalties fund the repository). We realize that this model is too experimental at this point to be viewed as a primary delivery mechanism. However, considerations such as (1) the large number of specialized topics addressed in education, (2) the need for students to examine the objects of their study under multiple representations using a variety of tools, and (3) the propensity for teachers and schools to adopt materials to their own way of doing things suggest that the traditional commercial economic model will not support the diversity of functionality needed for education. Hence an EOE model should be supported.

2.6.2 Additional Research

2.6.2.1 Designing open, interoperable educational software.

One involves the design of communication architectures for composing systems out of separately developed components. As suggested in this document, key issues lie in semantic interoperability. We recommend a testbed in which learning applications are composed of existing software that fills specific pedagogical needs. The research issues would be to investigate how to get adequate semantic coupling to support the pedagogical needs while minimizing changes needed to the components. Research and development could examine the roles of various solutions, including ontologies, translators, and alternate communication infrastructures. The work should be concerned with both software-level coupling (e.g., agents interpret objects from other components in an appropriate manner) and human-level coupling (e.g., objects retain their identity in the eye of the user as they move between components).

Another line of work involves the development of semantic annotations that can be embedded in more conventional materials, yet support advanced functionality. Much interest has been generated recently in the area of "metadata" for digital learning objects. Such efforts provide part of the groundwork for our vision, but are limited in three ways. First, the granularity of such efforts tends to be coarser than would be required for a semantic model of learner-constructed representational artifacts -- the dominant model is annotation of entire objects or documents. Second, metadata efforts do not go far enough to provide machine interpretable content semantics. Shared ontologies could eventually fill this need. (Metadata efforts are not to be faulted: there is an immediate need for other aspects of metadata that cannot be delayed while the research community develops ontologies.) The third limitation of metadata is also a limitation of any purely formal semantics for learner-constructed representations: it does not ensure that semantics will accrue for the user of the materials. Future work may be required in providing objects with persistence of appearance as well as of behavior before learners will perceive that which moves between applications as a single thing accumulating contextual semantics.

2.6.2.2 Representational devices as "epistemic forms" for collaborative learning.

These kinds of design considerations are critical, yet are insufficiently studied. We recommend a series of studies that vary features of representational systems, such as whether epistemological distinctions (theoretical claims versus empirical observations) must be attached to statements, and whether evidential relations between statements are represented as first class objects. Dependent variables include coding of discussion for certain discourse features shown by other literature to be correlated with positive learning outcomes, as well as direct measures of individual learning outcomes in both subject matter knowledge and inquiry skills.

2.6.2.3 Software participation in reflective learning interactions.

A related line of work might first begin with observations of peer coaching to answer questions such as: What support for learning do peer groups offer their members, and what support is lacking from peer groups that must be addressed by some mixture of human and automated mentoring? How can we recognize opportunities for coaching effective collaboration?

2.6.2.4 Examining the cost-benefit tradeoff between knowledge engineering and coaching functionality.

2.6.2.5 Designing hypermedia structures to scaffold critical inquiry skills.

2.6.2.6 Scaffolding learning from simulations and visualizations.

2.6.2.7 Comparing scientists' and learners' inquiry skills.

2.6.2.8 Realistic school implementation of advanced educational technology.

Designing teachers' mental models of our systems. While conducting teacher development workshops, we found ourselves engaged in cognitively demanding, rapid translation of our rich mental model of the software into a model that would be useful to teachers. We now believe that we need to design a mental model of the system oriented towards its classroom implementation, and to do so as part of the software design process rather than after the fact. This need fits nicely with user-oriented layers of a hierarchical architecture reference model that that developed in collaboration with external colleagues Frank Belz and Tom Wheeler.

Large scale, distributed evaluation. In order to conduct our research in the context of large scale school implementations, we need ways to understand what is going on during possibly concurrent use in a number of geographically distributed classrooms. Methodologies for analyzing large sets of interaction data will be needed to augment what can be observed in person. Our networked technology provides an opportunity for distributed data collection and developing new evaluation methodologies.

---

3. End Matter

---

3.1 Appendices

• Server software
• Coach software
• Client software: Windows, Solaris and MacOS versions.
• Hypertext science challenge materials

3.2 Bibliography

Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995).

Cognitive tutors: Lessons learned. The Journal of the Learning Sciences, 4 (2) 167-207.

Anderson, J. R. & Pelletier, R. (1991).

A development system for model-tracing tutors. In Proceedings of the International Conference of the Learning Sciences , 1-8. Evanston, IL.

Bates, R. & MacGregor, R. (1987).

The loom knowledge representation language. ADA 183415 RS-87-188, Information Sciences Institute, University of Southern California, Marina del Ray, CA, 1987.

Brown, A.L., & Palincsar, A.S. (1989).

Guided, cooperative learning and individual knowledge acquisition. In L. Resnick (Ed.), Knowing, Learning, and Instruction: Essays in Honor of Robert Glaser. Hillsdale, NJ: Lawrence Erlbaum Associates.

Brusilovsky, P., Schwarz, E, & Weber, G. (1996).

ELM-ART: An intelligent tutoring system on world wide web. ITS'96, Third International Conference on Intelligent Tutoring Systems Monteal, June 1996, pp. 261-269.

Chan, T.W., & Baskin, A.B.(1988).

Studying with the prince: The computer as a learning companion. In Proceedings of the International Conference on Intelligent Tutoring Systems, Montreal, pages 194-200, 1988.

Chi, M.T.H., Bassok, M., Lewis, M., Reimann, P., & Glaser, R. (1989).

Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13 :145-182, 1989.

Chinn, C. & Brewer, W. (1993).

Factors that influence how people respond to anomalous data. In Proc. 15th Annual Conf. of the Cognitive Science Society, pages 318--323, Hillsdale, NJ: Lawrence Erlbaum, 1993.

Clancey, W.J. (1992).

Guidon-manage revised: A socio-technical systems approach. In Intelligent Tutoring Systems, pages 21-36, 1992.

Cohen, P.R. & Feigenbaum, E.A. (1989).

The Handbook of Artificial Intelligence, Volume 1. Addison-Wesley, New York, 1989.

Collins, A., Brown, J. S., & Newman, S. (1989).

Cognitive Apprenticeship: Teaching the craft of reading, writing, and mathematics. Tech. Report No. 403, Palo Alto, CA: Institute for Research on Learning, 1989.

Collins, A. & Ferguson, W. (1993).

Epistemic Forms and Epistemic Games: Structures and Strategies to Guide Inquiry. Educational Psychologist, 28 (1), 25-42.

Cummins, J. (1988).

From the inner city to the global village: The microcomputer as a catalyst for collaborative interchange. Language, Culture and Curriculum, 1 (1), 1-13, 1988.

Forbus, K. (1997).

Using qualitative physics to create articulate educational software. IEEE Expert , 12 (3), May/June 1997.

Friedman-Hill, E. (1997).

Jess, the Java Expert System Shell. Sandia National Laboratories, Livermore, CA, 1997. http://herzberg.ca.sandia.gov/.

Ikeda, M., Hoppe, U., & Mizoguchi, R. (1995).

Ontological Issues of CSCL Systems Design. AI-Ed 95, the 7th World Conference on Artificial Intelligence in Education., August 16-19, 1995, Washington DC, pp.242-249.

Johnson, D. & Johnson, R. (1989).

Cooperation and Competition: Theory and Research. Interaction Book Company, 1989.

Johnson, R.T., Johnson, D.W., & Stanne, M.B. (1985).

Effects of cooperative, competitive, and individualistic goal structures on computer- assisted instruction. Journal of Educational Psychology , 77 (6), 668- 677.

Justen, III, J.E., Waldrop, P.B., & Adams, II, (1990, July).

Effects of paired versus individual user computer- assisted instruction and type of feedback on student achievement. Educational Technology, 30 (7) , 51- 53.

Katz, S., Lesgold, A., Eggan, G., & Gordin, M. (1993).

Modeling the student in Sherlock II. Journal of Artificial Intelligence in Education (Special issue on student modeling, G. McCalla & J. Greer, eds.), 3 , 495-518.

Katz, S., & Lesgold, A. (1994).

Implementing post-problem reflection within Coached Practice Environments. In P. Brusilovsky, S. Dikareva, J. Greer, and V. Petrushin (Eds.), Proceedings of the East-West International Conference on Computer Technologies in Education (pp. 125-30), Crimea, Ukraine.

Klayman, J. & Ha, Y.-W. (1987).

Confirmation, disconfirmation, and information in hypothesis testing. Psychological Review, 94 :211--228, 1987.

Klein, J.D., & Pridemore, D.R. (in press).

Effects of cooperative learning and need for affiliation on performance, time on task, and satisfaction. Educational Technology, Research & Development, 40 (4).

Kuhn, D. (1993).

Science as argument: Implications for teaching and learning scientific thinking skills. Science Education, 77 :319-337, 1993.

Lesgold, A. M., Lajoie, S. P., Bunzo, M., & Eggan, G. (1992).

SHERLOCK: A coached practice environment for an electronics troubleshooting job. In J. Larkin & R. Chabay (Eds.), Computer assisted instruction and intelligent tutoring systems: Shared issues and complementary approaches (pp. 201-238). Hillsdale, NJ: Lawrence Erlbaum Associates.

Luckham, D.C., Vera, J., and Belz, F. (1997).

Towards an abstraction hierarchy for CAETI architectures, and possible applications. Stanford University Computer Systems Laboratory Technical Report No. CSL-TR-97-727.

McManus, M.M., & Aiken, R.M. (1993).

The group leader paradigm in an intelligent collaborative learning system. In P. Brna, S. Ohlsson, and H. Pain (Eds.), Proceedings of the World Conference on Artificial Intelligence in Education (pp. 243-256). Edinburgh, Scotland, August, 1993.

Murray, T. (1996).

Having it all, maybe: Design tradeoffs in ITS authoring tools. ITS'96, Third International Conference on Intelligent Tutoring Systems , Monteal, June 1996, pp. 93-101.

National Academy of Sciences (1996).

National Science Education Standards. National Academy Press, 1996.

NCEE (1995).

New Standards: Performance Standards. National Center of Education and the Economy, 1995.

Newman, D. (1991).

Computer support for school work. In L. Birnbaum (Ed.), Proceedings of the 1991 Conference on the Learning Science (pp. 344-350). Charlottesville, VA: Association for the Advancement of Computing in Education.

O'Neill, D. K., & Gomez, L. M. (1994).

The collaboratory notebook: A distributed knowledge-building environment for project-enhanced learning. In Proceedings of Ed-Media '94, Vancouver, BC.

Paolucci, M., Suthers, D., & Weiner, A. (1996).

Automated advice-giving strategies for scientific inquiry. Intelligent Tutoring Systems, 3rd International Conference, Montreal, June 12-14, 1996.

Resnick, L. & Chi, M.T.H. (1988).

Cognitive psychology and science learning. In M.Druger, editor, Science for the Fun of it: A Guide to Informal Science Education, pages 24-31. National Science Teachers Association, 1988.

Ritter, S. & Koedinger, K. R. (1995).

Towards lightweight tutoring agents. In Proceedings of the World Conference on Artificial Intelligence in Education, AACE: Charlottesville, VA.

Ritter, S. & Koedinger, K. R. (1997).

An architecture for plug-in tutoring agents. In Journal of Artificial Intelligence in Education , 7 (3/4), 315-347. Charlottesville, VA: Association for the Advancement of Computing in Education.

Roschelle., J. (1994).

Designing for cognitive communication: Epistemic fidelity or mediating collaborative inquiry? The Arachnet Electronic Journal on Virtual Culture, 2 (2), 1994.

Roschelle, J. & Kaput, J. (1995).

Educational software architecture and systemic impact: The promise of component software. Presented at AERA Annual Meeting, San Francisco, April 19, 1995.

Rysavy, D.M. & Sales, G.C. (1991).

Cooperative learning in computer-based instruction. Educational Technology Research & Development , 39 (2), 70-79.

Scardamalia, M., & Bereiter, C. (1991).

Higher levels of agency for children in knowledge building: A challenge for the design of new knowledge media.The Journal of the Learning Sciences, 1 (1), 37--68.

Scardamalia, M., Bereiter, C., Brett, C., Burtis, P.J., Calhoun, C., & Smith Lea, N. (1992).

Educational applications of a networked communal database. Interactive Learning Environments , 2 (1), 45-71.

Scofield, J. W. (1996).

Computers and Classroom Culture. Cambridge: Cambridge University Press. 1996.

Sharan, S. (1980).

Cooperative learning in small groups: Recent methods and effects on achievement, attitudes, and ethnic relations. Review of Educational Research , 50 , 241-272

Slavin, R. E. (1990).

Cooperative learning: Theory, research, and practice . Englewood Cliffs, NJ: Prentice-Hall.

Smolensky, P., Fox, B., King, R., & Lewis, C. (1987).

Computer-aided reasoned discourse, or, how to argue with a computer. In R. Guindon (Ed.), Cognitive science and its applications for human-computer interaction (pp. 109-162). Hillsdale, NJ: Erlbaum.

Streitz, N. A., Hannemann, J., & Thuring, M. (1989).

From ideas and arguments to hyperdocuments: Traveling through activity spaces. In Hypertext '89 Proceedings, Pittsburgh, PA (pp. 343--364). New York: ACM.

Suthers, D. (1993).

Preferences for model selection in explanation. In Proc. 13th International Joint Conference on Artificial Intelligence (IJCAI-93), pages 1208--1213, Chambery, France, August 1993.

Suthers, D. & Jones, D. (1997).

An architecture for intelligent collaborative educational systems. AI-Ed 97, the 8th World Conference on Artificial Intelligence in Education, Kobe Japan, August 20-22, 1997.

Suthers, D., Toth, E., and Weiner, A. (1997).

An Integrated Approach to Implementing Collaborative Inquiry in the Classroom. Computer Supported Collaborative Learning (CSCL'97), Toronto, December, 1997.

Suthers, D., Weiner, A., Connelly, A. and Paolucci, M. (1995).

Belvedere: Engaging students in critical discussion of science and public policy issues.AI-Ed 95, the 7th World Conference on Artificial Intelligence in Education., August 16-19, 1995, Washington DC

Suthers, D. and Weiner, A. (1995).

Groupware for developing critical discussion skills. CSCL '95, Computer Supported Cooperative Learning, Bloomington, Indiana, October 17-20, 1995.

Thomason, R.H. (1992).

Netl and subsequent path-based inheritance theories. Computers Math. Applic., 23 (2-5):179--204, 1992.

Toth, J., Suthers, D., and Weiner, A. (1997).

Providing expert advice in the domain of collaborative scientific inquiry. To appear in AI&ED97.

Waugh, M.L. & Levin, J. (1988).

Telescience activities: Educational uses of electronic networks. The Journal of Computers in Mathematics and Science Teaching , 8 (2): 29-33, 1988.

Webb, N. (1989).

Peer interaction and learning in small groups. International Journal of Education Research, 13 :21-40, 1989.

Whitelock, D., Taylor, J., O'Shea, T., & Scanion, E. (1991).

How students construct a shared understanding of collisions in Newtonian mechanics. In L. Birnbaum (Ed.), Proceedings of the 1991 Conference on the Learning Sciences (pp. 430-441). Charlottesville, VA: Association for the Advancement of Computing in Education.

Contents

1. TECHNICAL REPORT
2. 1.Preliminary Information
   1. 1.1Report Title
   2. 1.2Notices
   3. 1.3Abstract
   4. 1.4Table of Contents; List of Tables, Figures and Illustrations
      1. 1.4.1Contents
      2. 1.4.2Tables and Figures
   5. 1.5List of Symbols, Abbreviations, and Acronyms
   6. 1.6Preface and Acknowledgements
3. 2.Body of the Report
   1. 2.1Summary
   2. 2.2Introduction
   3. 2.3Methods, Assumptions, and Procedures
      1. 2.3.1Overview of Belvedere
         1. 2.3.1.1Cognitive Support
         2. 2.3.1.2Collaborative Support
         3. 2.3.1.3Evaluative Support
         4. 2.3.1.4Other Features
      2. 2.3.2Architecture
         1. 2.3.2.1Concepts of Application
            1. 2.3.2.1.1Critical Inquiry in Science
            2. 2.3.2.1.2Generality
         2. 2.3.2.2Interface Presentation
            1. 2.3.2.2.1A Graphical Interface for Critical Inquiry
            2. 2.3.2.2.2Relations to Concepts of Applications
            3. 2.3.2.2.3Comments on the Analysis
         3. 2.3.2.3Concepts of Operations
            1. 2.3.2.3.1Supporting Collaborative Coached Critical Inquiry
            2. 2.3.2.3.2Relations to Other Levels
            3. 2.3.2.3.3Interoperability and Reusability
         4. 2.3.2.4Abstract Implementation
            1. 2.3.2.4.1Relations to Concepts of Operations
            2. 2.3.2.4.2Interoperability and Reusability Issues
         5. 2.3.2.5Resource Layer
         6. 2.3.2.6Architectural Lessons
      3. 2.3.3Design of Computer Coaches
         1. 2.3.3.1Introduction
         2. 2.3.3.2Pedagogical Constraints on Advice
         3. 2.3.3.3Syntactic Advice Strategies
         4. 2.3.3.4Consistency-Based Advice Strategies
            1. 2.3.3.4.1Consistency and Transitivity
            2. 2.3.3.4.2Formative Experiments
            3. 2.3.3.4.3 "Snippets"
            4. 2.3.3.4.4Implemented Expert Coach
            5. 2.3.3.4.5Reimplementation in Java
         5. 2.3.3.5Coaching Status and Future Directions
      4. 2.3.4Design of Classroom Implementation
         1. 2.3.4.1Classroom Environment
         2. 2.3.4.2Teacher's Role
         3. 2.3.4.3Curriculum Materials
         4. 2.3.4.4Student Activities
         5. 2.3.4.5Supporting Peer Evaluation with Performance-based Assessment
      5. 2.3.5Collaborative Activities Funded
         1. 2.3.5.1Belz and Luckham: Architecture Abstraction Hierarchy Reference Model
         2. 2.3.5.2Forbus and Koedinger: Integrated Feasibility Demonstration of Science Learning Spaces
            1. 2.3.5.2.1Demonstration Architecture
            2. 2.3.5.2.2Lessons Learned from the Integration Demonstration
         3. 2.3.5.3Workshop: Architectures and Methods for Designing Cost-Effective and Reusable ITS
   4. 2.4Results
      1. 2.4.1Highlights of Project
         1. 2.4.1.1Cognitive/Education Highlights
         2. 2.4.1.2Technology Highlights
         3. 2.4.1.3Implementation Status and Platforms
            1. 2.4.1.3.1Belvedere
            2. 2.4.1.3.2Collaborative Database Server and Connection Manager
            3. 2.4.1.3.3Coach ("Idea Generator")
         4. 2.4.1.4Maturity and Deployability
         5. 2.4.1.5Openness and Interoperability
            1. 2.4.1.5.1Points of Interoperability:
            2. 2.4.1.5.2Limitations on Interoperability:
         6. 2.4.1.6(Re)Applicability
         7. 2.4.1.7Venues and Collaborations
         8. 2.4.1.8Future Research
            1. 2.4.1.8.1Technology Research
            2. 2.4.1.8.2Cognitive/Education Research
      2. 2.4.2Results of Independent Evaluations
         1. 2.4.2.1Evaluation Methodology
         2. 2.4.2.2Summary of Evaluation Results
   5. 2.5Conclusions
   6. 2.6Recommendations
      1. 2.6.1Follow-on Work for Belvedere
         1. 2.6.1.1Port Server to NT
         2. 2.6.1.2Refine and Extend the Belvedere Interface as Guided by User Feedback
         3. 2.6.1.3Future Work for Coaching
         4. 2.6.1.4Cognitively Motivated Design of Challenge Activities and Materials
         5. 2.6.1.5Realistic Classroom Implementation
         6. 2.6.1.6Assessment Rubrics
         7. 2.6.1.7Commercial Deployment
      2. 2.6.2Additional Research
         1. 2.6.2.1Designing open, interoperable educational software.
         2. 2.6.2.2Representational devices as "epistemic forms" for collaborative learning.
         3. 2.6.2.3Software participation in reflective learning interactions.
         4. 2.6.2.4Examining the cost-benefit tradeoff between knowledge engineering and coaching functionality.
         5. 2.6.2.5Designing hypermedia structures to scaffold critical inquiry skills.
         6. 2.6.2.6Scaffolding learning from simulations and visualizations.
         7. 2.6.2.7Comparing scientists' and learners' inquiry skills.
         8. 2.6.2.8Realistic school implementation of advanced educational technology.
4. 3.End Matter
   1. 3.1Appendices
   2. 3.2Bibliography