diff options
Diffstat (limited to 'paper/method.tex')
| -rw-r--r-- | paper/method.tex | 13 |
1 files changed, 11 insertions, 2 deletions
diff --git a/paper/method.tex b/paper/method.tex index 43362da..0c3fdff 100644 --- a/paper/method.tex +++ b/paper/method.tex @@ -22,7 +22,7 @@ For each selected pair of leaf nodes $(a,b)$ we construct a pattern by walking t Patterns are extracted automatically given above constraints (each pattern connecting a pair of variables or values). We find that such patterns work well for Prolog. Other languages, however, will likely require different kinds of patterns to achieve good performance. -Finally, to avoid inducing rules specific to a particular program (covering typos and other idiosyncratic mistakes), we ignore rare patterns. In this study we used patterns that occurred in at least five submissions. These patterns form the feature space for rule learning. +In order to avoid inducing rules specific to a particular program (covering typos and other idiosyncratic mistakes), we ignore rare patterns. In this study we used patterns that occurred in at least five submissions. These patterns form the feature space for rule learning. \subsection{Learning rules} @@ -49,6 +49,15 @@ A possible way to solve this problem is to remove programs that are covered by s While our main interest is discovering important patterns, induced rules can still be used to classify new programs, for example to evaluate rule quality. Classification proceeds in three steps: 1) if a negative rule covers the program, classify it as incorrect; 2) else if a positive rule covers the program, classify it as correct; 3) otherwise, if no rule covers the program, classify it as incorrect. +We note that Prolog clauses can often be written in various ways. For example, the clause “\code{sum([],0).}” can also be written as + +\begin{Verbatim} +sum(List,Sum):- List = [], Sum = 0. +\end{Verbatim} + +\noindent +Our method covers such variations by including additional patterns and rules. Another option would be to use rules in conjunction with program canonicalization, by transforming each submission into a semantically equivalent normalized form before extracting patterns~\cite{rivers2015data-driven}. + \subsection{Generating hints} Once we have induced the rules for a given problem, we can use them to provide hints based on buggy or missing patterns. To generate a hint for an incorrect program, each rule is considered in turn. We consider two types of feedback: \emph{buggy hints} based on negative rules, and \emph{intent hints} based on positive rules. @@ -78,7 +87,7 @@ For example, if we find the following missing pattern for an incorrect program i \noindent we could display a message to the student saying “comparison between \code{X} and some other value is missing”, or “your program is missing the goal \code{X} \code{\textbackslash{}=} \code{?}”. -This method can find more than one missing pattern for a given partial program. In such cases we return the most commonly occurring pattern as the main hint, and other candidate patterns as alternative hints. We use main and alternative intent hints to establish the upper and lower bounds when evaluating automatic hints. +This method can find several missing patterns for a given partial program. In such cases we return the most commonly occurring pattern as the main hint, and other candidate patterns as alternative hints. We use main and alternative intent hints to establish the upper and lower bounds when evaluating hints. %%% Local Variables: |