diff options
| -rw-r--r-- | aied2017/evaluation.tex | 82 |
1 files changed, 81 insertions, 1 deletions
diff --git a/aied2017/evaluation.tex b/aied2017/evaluation.tex index 22e4253..50e9eb4 100644 --- a/aied2017/evaluation.tex +++ b/aied2017/evaluation.tex @@ -1 +1,81 @@ -\section{Evaluation}
\ No newline at end of file +\section{Evaluation and discussion} +We evaluated our approach on 44 programming assignments. We preselected 70\% of students +whose submissions were used as learning data for rule learning. The submissions from +the remaining 30\% of students were used as testing data to evaluate classification accuracy of learned rules +and to retrospectively evaluate quality of given hints. + +\begin{table}[t] +\centering + \begin{tabular}{|l|rrr|rr|rrr|r|} + \hline + Problem& \multicolumn{3}{c|}{CA} & \multicolumn{2}{c|}{Buggy hints} & \multicolumn{3}{c|}{Intent hints} & No hint \\ + \hline + & Rules & Maj & RF & All & Imp & All & Imp & Alt & \\ + \hline + sister & 0.988 & 0.663 & 0.983 & 128 & 128 & 127 & 84 & 26 & 34 \\ + del & 0.948 & 0.669 & 0.974 & 136 & 136 & 39 & 25 & 10 & 15 \\ + sum & 0.945 & 0.510 & 0.956 & 59 & 53 & 24 & 22 & 1 & 6 \\ + is\_sorted & 0.765 & 0.765 & 0.831 & 119 & 119 & 0 & 0 & 0 & 93 \\ + union & 0.785 & 0.766 & 0.813 & 106 & 106 & 182 & 66 & 7 & 6 \\ + ...& & & & & & & & & \\ + \hline + Average & 0.857 & 0.666 & 0.908 & 79.73 & 77.34 & 46.75 & 26.36 & 5.55 & 23.75 \\ + \hline + \end{tabular} +\caption{Results on 5 selected domains and averaged results over 44 domains. +Columns 2, 3, and 4 contain classification accuracies of our rule learning method, majority classifier, +and random forest, respectively. Columns 5 and 6 report the number of all generated buggy hints +and the number of hints that were actually implemented by students. The following three columns +contain the number of all generated intent hints (All), +the number of implemented hints (Imp) and the number of implemented alternative hints (Alt). +The numbers in the last column are student submission where hints could not be generated. +The last row averages results over all 44 domains.} +\label{table:eval} +\end{table} + +Table~\ref{table:eval} contains results on five selected problems (each problem represents a +different group of problems) and averaged results over all problems.\footnote{We report only a subset of results due to space +restrictions. The table with results for all 44 problems can be found at +\url{www.ailab.si/aied2017}. } The second, third, and fourth columns provide classification accuracies (CA) of rule-based classifier, +majority classifier, and random forest on testing data. The majority classifier and the random forests method, +which was the best performing machine learning algorithm over all problems, serve as lower and upper bound +for CA. For example, in the case of ``sister'' problem, +our rules correctly classified 99\% testing submissions, the majority was 66\%, and the random +forests achieved 98\% - slightly worse than the rules on this particular problem. The CA +is also good for problems ``del'' and ``sum'', however it is lower in cases of ``is\_sorted'' and ``union'', +suggesting that the proposed set of AST patterns is insufficient in certain problems. Indeed, after analysing the problem ``is\_sorted'' +we observed that our pattern set does not enclose patterns containing empty list ``[]'', an important pattern +for a predicate that tests whether a list is sorted. For this reason, the rule learning +algorithm failed to learn any C-rules and therefore all programs were classified as incorrect. Furthermore, many +solutions of the problem ``union'' include the use of the cut operator (exclamation mark), which +is, again, ignored by out pattern generation algorithm. + +To evaluate the quality of hints we selected all incorrect submission from student traces +that resulted in a correct program (solution). For example, from 403 submissions in the ``sister'' testing set, +there were 289 relevant incorrect solutions. + +The columns under ``Buggy hints'' contain evaluation of hints generated from rules +for incorrect class (I-rules). For each generated buggy hint we check whether +it was implemented by the student in the final submission (the solution). The column ``All'' is +the number of all generated buggy hints and the column ``Imp'' is the number of +implemented hints. The results suggest that almost all buggy hints are implemented in the final solution. + +The intent hints are generated when algorithm fails to find any buggy hints. The column ``All'' contains the number of +generated intent hints, ``Imp'' the number of implemented main intent hints, while ``Alt'' is the number +of implemented alternative hints. Notice that the percentage of implemented intent hints is significantly lower +when compared to buggy hints: in the case +of problem ``sister'' 84 out of 127 (66\%) hints were implemented, while in case of problem ``union'' only 66 out of 182 (36\%) hints were implemented. + +The last column shows the number of submissions where no hints could be generated. This value is relatively high +for the ``is\_sorted'' problem, because the algorithm could not learn any C-rules and consequently no intent hints where generated. + +To sum up, buggy hints seem to be good and reliable, since they are always implemented when presented, even when we tested +them in retrospective (the students did not see these hints). The percentage of implemented intent hints is, +on average, lower (56\%; 26.36 of 46.75), which is still not a bad result, providing that it is difficult to determine intent +of a programer. Last but not least, high classification accuracies in many problems imply that it is possible +to correctly determine the correctness of a program by simply verifying the presence of a small number of patterns. +Our hypothesis is that these patterns represent the parts of the problem solution that the students have +difficulties with. + + + |