diff options
Diffstat (limited to 'aied2017')
| -rw-r--r-- | aied2017/aied2017.tex | 23 | ||||
| -rw-r--r-- | aied2017/background.tex | 1 | ||||
| -rw-r--r-- | aied2017/conclusion.tex | 1 | ||||
| -rw-r--r-- | aied2017/dataset.tex | 1 | ||||
| -rw-r--r-- | aied2017/evaluation.tex | 1 | ||||
| -rw-r--r-- | aied2017/introduction.tex | 1 | ||||
| -rw-r--r-- | aied2017/method.tex | 1 | ||||
| -rw-r--r-- | aied2017/outline.org | 81 |
8 files changed, 110 insertions, 0 deletions
diff --git a/aied2017/aied2017.tex b/aied2017/aied2017.tex new file mode 100644 index 0000000..d342a9c --- /dev/null +++ b/aied2017/aied2017.tex @@ -0,0 +1,23 @@ +\documentclass{llncs} + +\usepackage[utf8]{inputenc} + +\begin{document} + +\title{TODO} +\author{TODO} +\institute{University of Ljubljana, Faculty of Computer and Information Science, Slovenia} +\maketitle + +\begin{abstract} +TODO +\end{abstract} + +\input{introduction} +\input{background} +\input{dataset} +\input{method} +\input{evaluation} +\input{conclusion} + +\end{document} diff --git a/aied2017/background.tex b/aied2017/background.tex new file mode 100644 index 0000000..563e5ea --- /dev/null +++ b/aied2017/background.tex @@ -0,0 +1 @@ +\section{Background}
\ No newline at end of file diff --git a/aied2017/conclusion.tex b/aied2017/conclusion.tex new file mode 100644 index 0000000..47c04f1 --- /dev/null +++ b/aied2017/conclusion.tex @@ -0,0 +1 @@ +\section{Conclusion}
\ No newline at end of file diff --git a/aied2017/dataset.tex b/aied2017/dataset.tex new file mode 100644 index 0000000..e8d168d --- /dev/null +++ b/aied2017/dataset.tex @@ -0,0 +1 @@ +\section{Dataset}
\ No newline at end of file diff --git a/aied2017/evaluation.tex b/aied2017/evaluation.tex new file mode 100644 index 0000000..22e4253 --- /dev/null +++ b/aied2017/evaluation.tex @@ -0,0 +1 @@ +\section{Evaluation}
\ No newline at end of file diff --git a/aied2017/introduction.tex b/aied2017/introduction.tex new file mode 100644 index 0000000..ac26833 --- /dev/null +++ b/aied2017/introduction.tex @@ -0,0 +1 @@ +\section{Introduction}
\ No newline at end of file diff --git a/aied2017/method.tex b/aied2017/method.tex new file mode 100644 index 0000000..3a14d53 --- /dev/null +++ b/aied2017/method.tex @@ -0,0 +1 @@ +\section{Method}
\ No newline at end of file diff --git a/aied2017/outline.org b/aied2017/outline.org new file mode 100644 index 0000000..a70c1a3 --- /dev/null +++ b/aied2017/outline.org @@ -0,0 +1,81 @@ +* Introduction (1) :Timotej: +** Generics +ITSs are useful, but expensive to create. Internet increases the reach of ITSs (think MOOC) and allows us to gather a lot of data. The challenge is to extract useful information from this data to improve feedback to students. + - koedinger2013new + +** Programming ITS +A particularly challenging domain: + - no meaningful “solutions steps” + - no easily discernible and independent “program features” + +** Problem +Find good features to describe student programs that: + - are automatically learnable + - can serve as a basis for generating hints + +** Contributions +A pattern language for describing program features. +Rules for classifying correct/incorrect Prolog programs. + +* Background (2) :Timotej: +Model-tracing is not well-suited for programming tutors. + - Hint Factory (Barnes & Stamper) for programming (Jin, Rivers) + - student traces are basically insane anyway, so learning from them is not a good idea (Piech 2015) +Instead we focus on analyzing individual submissions - CBM (maybe just /like/ CBM?). + - why: human tutors are usually able to comment without referring to previous program versions + +Analyzing program based on formal descriptions (usually defined manually): + - plans (Johnson) + - techniques (Brna/Huang/Le) + - strategies (Gerdes) +Comparing programs directly / after canonicalization (Rivers 2015) + - not easy to determine which differences are crucial + +Defining “programming attributes/features”: + - AST subtrees (Nguyen 2014) + - linkage graphs (Jin 2012) + - n-grams or other purely textual attributes + - mostly used for plagiarism (MOSS) / code clone detection + +Tgrep / trx for structural patterns in NLP - use the same for describing code! + +* Dataset (3) :Timotej: +Solution traces +** Motivating example: ancestor/2 + - show and explain correct / buggy program + - examples of patterns & rules + - buggy rules ≈ error classes + - correct rules ≈ program strategies or “plans” → determine intent + - how correct / buggy rules can be used to automatically determine intent and generate feedback + - using rules to assist authoring tutors + +** Patterns +Programming is all about manipulating data (variables/literals). +Patterns encode (only) relations between data nodes in an AST, and are more granular than subtrees. +Patterns abstract away some information +Algorithm for extracting patterns. + - which patterns are considered + +* Method (2) :Martin: +Rule learning. + - first learn correct, then buggy rules +Automatic hint generation. + +* Evaluation (3) :Martin:Timotej: +Example: describe patterns/rules for ancestor/2 (and/or another domain). + - example trace +CA/AUC for rules/RF/…. + Using e.g. RF with patterns for attributes achieves a high classification accuracy for correct/incorrect programs. + Rules are less accurate, but can be used for hints (how can we improve this)? +Simulated hints: for how many submissions would we suggest a hint that was ultimately implemented in the final solution? + Hints from buggy rules are almost always implemented; correct rules are less certain. + +* Conclusions / future work (1) + - syntactic variations (e.g. using = instead of implicit matching) + - combine with canonicalization + - ABML for rule learning + - integrate into a tutor (CodeQ / CloudCoder) to test in situ + +* Misc +Problems: + - predicates with similar clauses (e.g. union/3) |