diff options
| -rw-r--r-- | aied2017/evaluation.tex | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/aied2017/evaluation.tex b/aied2017/evaluation.tex index f81c2fd..bb3f4c6 100644 --- a/aied2017/evaluation.tex +++ b/aied2017/evaluation.tex @@ -58,7 +58,7 @@ of implemented alternative hints. Notice that the percentage of implemented inte when compared to buggy hints: in the case of problem \code{sister} 84 out of 127 (66\%) hints were implemented, whereas in the case of problem \code{union} only 66 out of 182 (36\%) hints were implemented. On average, 56\% of main intent hints were implemented. The last column shows the number of submissions where no hints could be generated. This value is relatively high -for the \code{is\_sorted} problem, because the algorithm could not learn any C-rules and consequently no intent hints were generated. +for the \code{is\_sorted} problem, because the algorithm could not learn any C-rules and thus no intent hints were generated. To sum up, buggy hints seem to be good and reliable, since they are always implemented when presented, even when we tested them on past data -- the decisions of students were not influenced by these hints. The percentage of implemented intent hints is, on average, lower (56\%), which is still not a bad result, providing that it is difficult to determine the programmer’s intent. In 12\% (244 out 2057) where main intent hints were not implemented, students implemented an alternative hint that was identified by our algorithm. Overall we were able to generate a hint in 84.5\% of cases. In 73\% of all cases, the hints were also implemented. Last but not least, high classification accuracies in many problems imply that it is possible to correctly determine the correctness of a program by simply verifying the presence of a small number of patterns. Our hypothesis is that there exist some crucial patterns in programs that students need to resolve. When they figure out these patterns, the implementation of the remaining of the program is often straightforward. |