Add conclusion

3 files changed, 20 insertions, 4 deletions

diff --git a/aied2017/conclusion.tex b/aied2017/conclusion.tex
index 47c04f1..e2cd71b 100644
--- a/aied2017/conclusion.tex
+++ b/aied2017/conclusion.tex
@@ -1 +1,19 @@
-\section{Conclusion}
\ No newline at end of file
+\section{Conclusion}
+
+AST patterns can be used to describe structural information in programs’ ASTs. By encoding only relations between selected nodes, each pattern can match many programs. AST patterns thus function as “regular expressions for trees”. We have used patterns as features for machine learning.
+
+We presented a method for automatically extracting AST patterns from student programs. Our patterns encode relations between data objects in a program, with each pattern connecting either two instances of the same variable, a variable and a value, or two values. We consider such patterns as the atomic syntactic relations in a program, and use them as machine-learning features.
+
+Evaluation shows that our patterns are useful for classifying Prolog programs. It is likely, however, that other programming languages will require different kinds of patterns. In Python, for example, variables can take on different values and appear in many places. This will likely require patterns relating more than two instances of a variable, or multiple variables.
+
+We explained how patterns can be used as features to induced rules for classifying correct and incorrect programs. Since the goal of our research is to generate hints, we adapted the CN2 algorithm to produce rules useful for that purpose. We induce rules in two passes: we first learn the I-rules for incorrect programs, and then use the programs not covered by any I-rule to learn the C-rules for correct programs.
+
+We showed how to generate hints based on I- and C-rules by highlighting buggy patterns or pointing out what patterns are missing. Evaluation on past student data shows that useful hints can be provided for many incorrect submissions this way. The quality of feedback could be improved by annotating rules with explanations in natural language. Furthermore, since patterns and rules are easily interpretable, they can also help when manually authoring tutoring systems, with I-rules indicating the common errors and C-rules the typical solution strategies for each problem.
+
+In the future we will attempt to improve rule accuracy for certain problems, such as \code{union}. This will likely necessitate new kinds of patterns, for example to handle the cut operator. Adapting our methods to handle Python programs will give us some insight into what kinds of patterns could be useful in different situations. Finally, we will implement hint generation in an online programming tutor CodeQ, and evaluate the effect of automatic feedback on students’ problem-solving.
+
+
+%%% Local Variables:
+%%% mode: latex
+%%% TeX-master: "aied2017"
+%%% End:
\ No newline at end of file
diff --git a/aied2017/method.tex b/aied2017/method.tex
index 5b7614b..f3606c0 100644
--- a/aied2017/method.tex
+++ b/aied2017/method.tex
@@ -64,7 +64,7 @@ Even though our main interest is discovery of patterns, we can still use induced
 
 \subsection{Generating hints}
 
-Once we have induced the rules for a given problem, we can use them to provide hints based on buggy or missing patterns. To generate a hint for an incorrect program, each rule is considered in turn.
+Once we have induced the rules for a given problem, we can use them to provide hints based on buggy or missing patterns. To generate a hint for an incorrect program, each rule is considered in turn. We consider two types of feedback: \emph{buggy} hints based on I-rules, and \emph{intent} hints based on C-rules.
 
 First, all I-rules are checked to find any known incorrect patterns in the program. To find the most likely incorrect patterns, the rules are considered in the order of decreasing quality. If all patterns in the rule “$p_1 ∧ ⋯ ∧ p_k ⇒ \mathsf{incorrect}$” match, we highlight the relevant leaf nodes. As an aside, we found that most I-rules are based on a single pattern. For the incorrect \code{sum} program from the previous section, our method produces the following highlight:
 
diff --git a/aied2017/patterns.tex b/aied2017/patterns.tex
index fe2ae6c..47fc9e2 100644
--- a/aied2017/patterns.tex
+++ b/aied2017/patterns.tex
@@ -175,8 +175,6 @@ The leftmost pattern in Fig.~\ref{fig:sum}, drawn with dotted blue arrows, descr
 \noindent
 We include such patterns in our feature set to cover the base-case clauses in recursive programs, which often include no variables.
 
-% TODO move to conclusion
-%While the patterns used in this paper appear to be useful for analyzing Prolog programs, it is likely that other kinds of patterns will be needed for other programming languages. In Python, for example, variables can take on different values and be accessed from many places. This will likely require patterns relating more than two instances of a variable, or multiple variables.
 
 %%% Local Variables:
 %%% mode: latex