|
|
Bringing ITS to the Marketplace: A
Successful Experiment in Minimalist Design
Presented by:
Kim Adolphe and Patrick Brackett
Gemini Performance Solutions Inc.
Calgary, Alberta, Canada
Dr. Marlene Jones and Carl Gutwin
Alberta Research Council
Calgary, Alberta, Canada
Abstract: Intelligent Tutoring Systems
(ITS) have proven to be effective tools for teaching and
training. However, ITSs have not become common in
industrial and organizational settings, in part because
their complexity has proven difficult to manage outside of
the research lab. Minimalist ITSs are an attempt to bridge
the gap between research and practical application; they
simplify research techniques while striving to maintain as
much pedagogic intelligence as possible. This paper
describes one such system, SWIFT, that is an example of how
a minimalist ITS can be delivered as a commercial product.
We outline some of the issues facing designers of a
minimalist system, and describe the ways that research
techniques have been incorporated into four modules of
SWIFT: adaptive testing, course planning, guidance, and
diagnosis.
AI Techniques
Years of extensive research and development has been
undertaken into applying Artificial Intelligence (AI)
techniques to training. The results have proven that more
effective training can be achieved through AI. However, the
application of AI techniques to teaching and training has
not become common in industrial and organizational
settings, in part because their complexity has proven
difficult to manage outside of the research lab. SWIFT is
the first commercial application of its kind to
successfully bridge the gap between research and practical
application. This has been accomplished by simplifying the
research techniques while striving to maintain as much of
the pedagogic intelligence as possible. SWIFT is a
minimalist Intelligent Tutoring System (ITS) that uses AI
techniques from ITS research in its adaptive testing
facility, instructional planner, diagnosis, and situation
recognition and guidance. An AI technique called pattern
recognition is also applied to the short answer question
type in SWIFT.
Minimalist Design in SWIFT
The following sections describe the approaches that we
have used to make; the most of the resources available to
SWIFT. In general, our strategies have taken three paths:
first, we have found ways to minimize ITS techniques
without compromising too much of their power; second, we
have found additional mechanisms to make our solutions more
robust; and third, we have taken advantage of the abilities
of learners and our knowledge of the eventual user
population.
Knowledge Representation
A defining feature of an ITS is a semantic
representation of the instructional domain, where the
concepts are encoded in data structures that allow the
system to reason about the course. A minimalist ITS must
also employ semantic representation, for an understanding
of the concepts in the domain is the basis of much of a
system's intelligent behavior. However, the detail and
sophistication of the representation can vary. In SWIFT, we
have implemented a presentation scheme that allows us to
reason about the domain, but does not contain as much
detail about specific concepts as might be found in a full
ITS. SWIFT courses are stored in a hierarchical structure
that divides the instructional material into smaller and
smaller pieces, much as a book does with chapters, sections
and subsections. A course has three levels: the first
contains a set of topics, which are divided at the second
level into sets of modules, which are divided at the third
level into concepts. A semantic representation of the
course also allows the specification of dependencies
between concepts. The current version of SWIFT allows for
prerequisite and sequence links between individual concept
objects.
Adaptive Testing
ITSs gather information about a learner's progress by
observing them as they interact with the learning
environment. Many minimalist systems use exercises,
quizzes, and exams as the setting for these observations
since the range of possible inference about the learner can
be more easily constrained. Since many organizations
(corporate and otherwise) also require that a training
system provides concrete records of progress, we have
chosen to use formative and summative testing as our means
for observing the learner in SWIFT.
One of the problems with traditional exams is that they
are of fixed length; a learner must complete a long series
of questions in order for the system to determine how well
they know a subject. This characteristic can cause
frustration for both novices and experts, who may know
after a few questions that the subject matter is either
bewildering or trivial. Aside from giving the learner
greater control over exams - in that they are never forced
to take a test - our primary strategy for tackling the
problem of fixed-length exams is adaptive testing. Adaptive
testing allows exams to be significantly shorter than
traditional tests, without losing any predictive power
about a learner's master of the material. The approach that
is implemented in SWIFT is based on the work of [Welch
& Frick, 1993]. The algorithm uses Bayes' theorem to
estimate the probability that the learner is a master or
non-master of the material after each test question is
answered. In SWIFT, novices (non-masters) and experts
(masters) can be determined in as few as five questions.
This graph represents the outcomes of a study that was
completed comparing classroom instruction to four
Intelligent Tutoring Systems. The study looked at (1)
whether Tutors engender more effective and efficient
learning in relation to traditional formats and (2) do they
reduce the range of learning outcomes measures where a
majority of individuals are elevated to high performance
levels. The Tutors underwent systematic, controlled
evaluations: a) Lisp tutor (Anderson Farrell & Sauers,
1984); b)Smithtown (Shute & Glaser, in press); Sherlock
(Lesgold, Lajoie, Bunzo & Eggan, 1990); and d) The
Pascal ITS (Bonar, Cunningham, Beatty & Weil, 1988).
Instructional Planning (ALE)
Instructional planning in SWIFT is based on two
information sources: the results of an adaptive pretest,
and the learner's choice of one or more instructional
goals. Each goal specifies which topics and modules of the
course are to be included in the learner's path;
performance on the pretest then indicates whether concepts
within those sections are already known and need not be
included. Our approach to instructional planning is
effective, but is relatively simple compared to some ITSs
(e.g. [Becht, 1990]) because of SWIFT's less-sophisticated
domain representation. Since our simpler approach weakens
SWIFT's planning to a degree, we have found other ways of
ensuring that appropriate instruction is always available
to the learner.
Since we knew that the target population for SWIFT is
composed largely of learners that are cooperative and
motivated, we were able to view instructional planning as a
human-computer problem rather than just a computational
one. One of the ways we involve the learner is by providing
tools that allow them to monitor their path through the
course, and to take control if desired. SWIFT provides an
easy to navigate course map that displays the entire course
and allows the learner to navigate to any topic of
interest. This course map and non-linear approach also
ensures that SWIFT is effective for other learning purposes
such as, review or just-in-time training. This approach
also improves instructional planning by making use of the
knowledge of both parties: learners can improve upon or
customize the system's course plan if they wish; the
recommended path, which is adequate in most cases, provides
support for learners who do not wish to venture out on
their own. If they do, and get lost as a result, SWIFT can
take them back to where they should be with the click of a
button.
Diagnosis
Diagnosis modules attempt to understand problems and
misconceptions in a student's knowledge of the domain (e.g.
[Johnson & Soloway, 1985], [McCalla & Greer, 1990]
). Although any student action may be considered, diagnosis
is commonly applied to a learner's answers to test or
exercise questions. Diagnosis entails drawing conclusions
about the learner's knowledge based on features in their
answers; good diagnosis allows systems to provide
appropriate feedback and remediation as well as simple
indication of whether an answer is right or wrong.
Diagnosis can require significant inferencing power and
domain knowledge, which are not the strengths of minimalist
systems. An alternative to a fully knowledge-based approach
is to detail a number of categories, or cases, of typical
errors and misconceptions. Using a case-based approach
transforms the inference problem to one of classifications,
but effective classification can also be difficult to
achieve. One problem occurs in specifying the answers that
belong to a particular class. The obvious method is to
encode every answer. However, this technique implies that
any variation of an answer, even those that do not change
its essential parts, must also be included. This can be a
daunting task for any but the most trivial of exercises.
Our approach to this problem allows a course designer to
concentrate on the qualitative differences in the possible
answers to a question, rather than on syntactic variations.
Our case-based diagnosis subsystem uses regular
expressions, constructs that allow a designer to specify a
large number of possible variations with a single answer
pattern. The system can examine and evaluate any short
textual answers for which cases have been designed. The
course designer specifies patterns for classes of correct
and incorrect answers, and can annotate each class with
appropriate feedback and remediation information.
This strategy still requires that the course designer
understands the kinds of difficulties that learners can
have in a particular area, and how each problem can be
manifested in answers to questions. However, we have
provided a framework for structuring and using that
pedagogic knowledge that is both powerful and efficient
enough to be used in a minimal system.
Situation Recognition and Guidance
SWIFT has more and more become a learner-controlled
system, both by design and by necessity. In a self-directed
environment, the task of the intelligent tutoring system
shifts from tutoring and control to guidance and support.
WE have been forced to find and implement mechanisms for
supporting learners as they explore the system on their
own.
We have developed a subsystem within SWIFT that can
provide guidance on pedagogic issues according to the
specific situation that the learner is in, and can also
encourage the learner to initiate certain learning
behaviours. Many strategies exist for assisting
self-directed learning that promote metacognition and more
effective learning behavior (eg. [Derry & Murphy,
1986], [Derry, 1992], [Pressley et al., 1989], [Shuell,
1992], [Winne, 1992]). Examples of effective learning
behavior include positive self-talk, note-taking or
highlighting, summation, imagery, question-generation, and
review of learning objectives.
SWIFT's guide watches system events and monitors a
learner's location, history, and current knowledge. When
particular kinds of situations occur, the guide can decide
to deliver advice to the learner. For example, if a student
turned their attention to a new section of course material,
the guide might suggest that they test their knowledge of
the current section before going on. The guide is
implemented as a rule-based system, and the above example
would involve a rule such as "if the learner has not
demonstrated mastery in the concepts of the current module,
and the learner requests a move to a new module, the system
will suggest that the learner take a module test for the
current module." The guide's advice is presented in a popup
dialogue box.
The rule-based guidance system provides SWIFT with a
generalized architecture for presenting useful information.
We are able to give the learner pedagogic guidance in a
wide variety of situations, but the architecture can also
be used to give information about any situation, such as
tips on using SWIFT to its fullest capacity.
References
[Brecht, 1990] Brecht (Wasson), B.
(1990) Determining the Focus of Instruction: Content
Planning for Intelligent Tutoring Systems, Unpublished
doctoral dissertation, University of Saskatchewan.
[Derry & Murphy, 1986] Derry,
S., and Murphy, D.A. (1986), Designing Systems that Train
Learning Ability: From Theory to Practice. Review of
Educational Research , 56(1), 1 - 39.
[Derry, 1992] Derry, S. (1992)
Metacognitive Models of Learning and Instructional System
Design. In Adaptive Learning Environments: Foundations
and Frontier, M. Jones and P. Winne ed. Springer-Verlag.
257-286.
[Johnson & Soloway, 1985]
Johnson, W. L. and Soloway, E. 1985 PROUST: An Automatic
Debugger for Pascal Programs. Byte, 10 (4), 179-190.
[Pressley et al., 1989] Pressley,
M. Johnson, C. Symons, S., McGoldrick, J., Kurita, J.,
(1989), Strategies That Improve Children's Memory and
Comprehension of Text. The Elementary School Journal,
90(1), 3-32.
[McCall & Greer 1994] McCalla,
G. I. And Greer, J.E. (1994), Granularity-Based Diagnosis
and Belief Revision in Student Models, In Student
Modelling: The Key to Individualized Knowledge-Based
Instruction, J. Greer and G. McCalla ed, Springer-Verlag,
, 39-62
[Winne, 1992] Winne, P. (1992)
State-of-the-art Instructional Computing Systems that
Afford Instruction and Bootstrap Research. In Adaptive
Learning Environments: Foundations and Frontiers, M.
Jones and P. Winne ed., Berlin:Springer-Verlag, 349-380.
[Shuell, 1992] Shuell, T. J. (1992)
Designing Instructional Computing Systems for Meaningful
Learning. In Adaptive Learning Environments: Foundations
and Frontiers, M. Jones and P. Winne ed.,
Berlin:Springer-Verlag, 19-54.
|
