Computational models of tutorial dialogue

ITSs help students master a certain topic. Research on the next generation of ITSs explores NL as one of the keys to bridging the gap between current ITSs and human tutors. Two issues are receiving the most attention: what specifically makes human tutoring effective; and whether an NL interface can be really effective, given the current state of the art in NLP. We are currently working on both issues. We apply empirical methods to dialogues that have been tagged for features that correlate, individually or in subsets, with tutorial methods and functions. We use the empirical findings in the development of a computational model of tutorial dialogue, envisioned as a multi-layered planning system.

We gratefully acknowledge our sponsors, the Office of Naval Research and the National Science Foundation.

The DIAG-NLP project (2000-2004) was among the first to show that a NL interaction improves learning. We added Natural Language Generation (NLG) capabilities to an existing ITS, that tutors students in diagnosing malfunctions in a heating system. We developed two different feedback generation engines, that we systematically evaluated in a three way comparison that included the original system as well. We focused on aggregation, i.e., on how lengthy information can be grouped and presented as more manageable chunks. We found that syntactic aggregation does not improve learning, but that functional aggregation, i.e. abstraction, does.
Expert or novice tutors? (2002-2006) In the literature there are no direct comparisons between expert and novice tutors, in particular, there is scant if any evidence that expert tutors are actually more effective. From the point of view of computationally modeling a dialogue, the simpler the language phenomena, the simpler the computational task. Expert tutors are likely to use more complex dialogue strategies than novice tutors. If expert tutors and novice tutors were equally effective, then modeling novice tutors would make the computational task easier. To answer these questions, we are collecting dialogues and building NL interfaces in two domains.
- In the first, students learn to extrapolate complex letter patterns. Subjects were individually tutored by three different tutors: the first had years of experience as a professional tutor, the second had years of experience as a lecturer but scarce experience in one on one tutoring, the third was an undergraduate with no teaching or tutoring experience. A control group was not tutored at all. Every group of tutored subjects did significantly better than the control group on both problems in the post-test. Additionally, subjects who were tutored by the expert tutor did significantly better than the other tutored subjects on the more complex second problem.
  In this domain, we also experimented with the effect of automatic generation of simple feedback. We built three different versions of an interface to an ITS that tutors students in this domain. In the neutral version of the ITS, the only feedback students receive is via color coding, green for correct, red for incorrect; in the positive version, they receive feedback via the same color coding, and in addition, verbal feedback on correct responses only; in the negative version, they receive feedback via the same color coding, and in addition, verbal feedback on incorrect responses only. We ran a between-subject experiment, in which each group of subjects interacted with one of the systems. We found that, even if students in the verbal conditions do perform slightly better and make fewer mistakes, these differences are not significant.
  Currently, we are annotating the dialogues for tutor and student moves to extract dialogue patterns. We will then build a more sophisticated interface in which feedback is given according to these dialogue patterns.
We are also collecting tutoring dialogues in a Computer Science domain, specifically, basic data structures and algorithms such as linked lists, stacks, and binary search trees. Students take a pretest, are tutored, and take the same test as posttest. There are two tutoring settings, in one students are tutored by an expert tutor (a retired CS professor with 25 years of experience in tutoring math and CS), in the other, by a novice tutor (an undergraduate in CS with very limited prior experience in tutoring). Preliminary results show that students learn in both conditions, and that the learning gain with the expert tutor is about 30% greater than with the novice tutor. If the expert tutor is really more effective, following the same methodology discussed above, we will annotate his dialogues with correlates of successful learning episodes, mine for dialogue patterns that correlate with those episodes, and include them in our dialogue architecture. We have already built the basic CS ITS. If the expert tutor is not more effective, we will apply the same methodology the the dialogues with the novice tutor.