From e4d5f477ca121532a7ca568543da11166065c5dd Mon Sep 17 00:00:00 2001 From: Martin Mozina Date: Mon, 5 Feb 2018 17:07:26 +0100 Subject: Added greatest absolutist --- aied2018/aied2018.tex | 1 + aied2018/rules.tex | 120 +++++++++++++++++++++++++++++++++++++++----------- 2 files changed, 96 insertions(+), 25 deletions(-) diff --git a/aied2018/aied2018.tex b/aied2018/aied2018.tex index c26d1a2..514b7e7 100644 --- a/aied2018/aied2018.tex +++ b/aied2018/aied2018.tex @@ -14,6 +14,7 @@ \usepackage{color} \newcommand\red[1]{{\begingroup\color[rgb]{0.8,0.15,0.15}#1\endgroup}} \newcommand\blue[1]{{\begingroup\color[rgb]{0.15,0.15,0.8}#1\endgroup}} +\newcommand\green[1]{{\begingroup\color[rgb]{0.15,0.8,0.15}#1\endgroup}} \usepackage{fancyvrb} \fvset{commandchars=\\\{\},baselinestretch=0.98,samepage=true,xleftmargin=2.5mm} diff --git a/aied2018/rules.tex b/aied2018/rules.tex index c60b989..55a9a10 100644 --- a/aied2018/rules.tex +++ b/aied2018/rules.tex @@ -2,7 +2,7 @@ \label{sec:rules} \subsection{The learning algorithm} -The goal of learning rules in this paper is to extract and explain common approaches and mistakes in student programs. We use a rule learner called ABCN2e implemented within the Orange data mining library~\cite{demsar2013orange}. ABCN2e is an improvement of the original CN2 algorithm~\cite{clarkECML1991} for learning unordered rules. The differences between CN2 and ABCN2e are described in a technical report found at \url{https://ailab.si/abml.} +The goal of learning rules in this paper is to extract and explain common approaches and mistakes in student programs. We use a rule learner called ABCN2e implemented within the Orange data mining library~\cite{demsar2013orange}. ABCN2e is a variant of the classical CN2 algorithm~\cite{clarkECML1991} for learning unordered rules. The differences between CN2 and ABCN2e are described in a technical report found at \url{https://ailab.si/abml.} General rule-learning algorithms, such as CN2, tend to generate large amounts of specific rules, which leads to more accurate results, however this makes them less appropriate for explaining. We will now describe a problem specific configuration of the rule-learning algorithm that extracts relevant and explainable patterns from student programs. @@ -23,11 +23,11 @@ Different approaches can be represented with rules explaining correct programs. We use the same constraints as in the case of n-rules and learn rules for correct programs called \emph{p-rules}. In this case, we always require that conditions mention the presence of patterns, since it is easier to explain different approaches of students with something they have written and not with something they have not. To account for possible buggy patterns, the requirement to achieve 90\% classification accuracy was not evaluated on full data, but only on data not covered by n-rules. Hence, a rule can cover an example with a specific approach even though if it contains a buggy pattern. \section{Interpretation of rules} -Learned rules can be used to analyse student programing. For demonstration we will describe and interpret several learned rules from three Python exercises. The first exercise is \textit{Fahrenheit to Celsius}, which requires a simple user-computer interaction and a single expression. The second is \textit{Buy Five}, where students use loops for the first time. The third exercise is \textit{Greatest Absolutist}, which is one of the introductory Python exercises for functions. +Learned rules can be used to analyze student programing. We will describe and interpret several learned rules from two Python exercises. The first exercise is \textit{Fahrenheit to Celsius}, which requires a simple user-computer interaction and a single expression. The second exercise is \textit{Greatest Absolutist}, one of the introductory Python exercises for functions. \subsection{Fahrenheit to Celsius} The first problem in CodeQ Python class is to implement a program that converts from degrees Fahrenheit to degrees Celsius. The program should ask the user to input temperature in Fahrenheit degrees, and then output the temperature in Celsius. A sample correct program is: -\begin{Verbatim}[fontfamily=tt] +\begin{Verbatim} F = float(input("Fahrenheit: ")) C = 5 / 9 * (F - 32) print("Celsius: ", C) @@ -35,29 +35,29 @@ print("Celsius: ", C) Students have so far submitted 1177 programs for this problem, 495 of them were correct and 682 incorrect. Our systems extracted 891 relevant AST patterns, which were used as attributes in rule learning. The rule learner induced 24 n-rules, 14 of those mention only presence of patterns, and 16 p-rules. -We shall first examine n-rules that mention only presence of patterns in their conditions. The most accurate rule according to the rule learner was: -\begin{Verbatim}[fontfamily=tt] +We will first take a look at n-rules that mention only presence of patterns in their conditions. The most accurate rule according to the rule learner was: +\begin{Verbatim} IF regex-20==T THEN correct=F [208, 1] \end{Verbatim} This rule covers programs where \texttt{regex-20} is present, implies an incorrect program (\texttt{correct=F}) and covers 208 incorrect programs and 1 correct. Regex-20 is the AST pattern describing a call to the \texttt{int} function: -\begin{Verbatim}[fontfamily=tt] +\begin{Verbatim} (Module (body (Assign (value (Call (func (Name (id int) (ctx Load)))))))) \end{Verbatim} The second best n-rule that covers 72 incorrect and 0 correct programs was: -\begin{Verbatim}[fontfamily=tt] +\begin{Verbatim} IF regex-5==T AND regex-35==T THEN correct=F [72 0]. \end{Verbatim} -Regex-5 describes a pattern in programs, where the result of function \texttt{input} is not casted to \texttt{float}; in fact, it is not casted to at all, students merely store the returned string. Regex-32 pattern describes the substraction of value 32 from a variable in an expression. Two sample program that match the first rule (left) and the second rule(right) are (matching parts of code are underlined): -\begin{Verbatim}[fontfamily=tt] -g2 = input() g2 \underline{= input(}'Temperature [F]? ') -g1 \underline{= int(}g2) g1 \underline{= ((g2 - 32) *} (5 / 9)) +Regex-5 describes the pattern in programs, where the result of the \texttt{input} function is not casted to \texttt{float}; in fact, it is not casted to at all, students merely store the returned string. Regex-35 pattern describes a substraction of value 32 from a variable in an expression. Two sample program that match the first rule (left) and the second rule(right) are (code that matches patterns is highlighted): +\begin{Verbatim} +g2 = input() g2 = \blue{input}('Temperature [F]? ') +g1 = \blue{int}(g2) g1 = (\red{(g2 - 32) *} (5 / 9)) print(((g1-32)*(5/9))) print(g2, 'F equals', g1, 'C') \end{Verbatim} -These two rules describe two (out of many) types of errors made by students. The left program is incorrect, since it doesnt work when a user inputs a floating value. The right program is incorrect, because the input string must be casted to float. Not casting it (regex-5) and then using it in an expression (regex-35) will result in a Python exception stating that it can not subtract a number from a string. +These two rules describe two (out of many) errors made by students. The left program is incorrect, since it doesnt work when a user inputs a floating value. The right program is incorrect, because the input string must be casted to float. Not casting it (regex-5) and then using it in an expression (regex-35) will result in a Python exception stating that it can not subtract a number from a string. The next two rules are the most accurate two n-rules that also contain missing patterns in their conditions: -\begin{Verbatim}[fontfamily=tt] +\begin{Verbatim} IF regex-0==F THEN correct=F [106 0] IF regex-1==F AND regex-16==T THEN correct=F [100 0] \end{Verbatim} @@ -65,8 +65,8 @@ The pattern regex-0 matches programs with a call to function \texttt{print}. A p The second rule covers programs with missing regex-1 and present regex-16. Regex-16 matches programs with a call to the \texttt{print} function, where the argument contains a formula which subtracts 32 from a variable and then further multiplies the result. Regex-1 describes a call to the function \texttt{float} as the first item in an expression, i.e. \texttt{= float(...)}. This rule therefore represents programs that have the formula in the \texttt{print} function (regex-16 is present), however fail to cast input from string to float (regex-1 is missing). -Finally, let us examine best four p-rules: -\begin{Verbatim}[fontfamily=tt] +Let us now examine the other type of rules. The best four p-rules are: +\begin{Verbatim} IF regex-2!=F AND regex-8!=F THEN correct=T [ 1 200] IF regex-1!=F AND regex-42!=F THEN correct=T [ 0 68] IF regex-1!=F AND regex-8!=F THEN correct=T [ 3 217] @@ -74,24 +74,94 @@ IF regex-80!=F THEN correct=T [ 0 38] \end{Verbatim} The patterns in the condition of the first rule, regex-2 and regex-8, correspond to a call to the function \texttt{input} within the function \texttt{float}, i.e.\texttt{float(input())}, and a call to the function \texttt{print}, which contains the pattern \texttt{-32)*} in the first argument, respectively. Programs matching both patterns wrap the function \texttt{float} around \texttt{input}, and have an expression that subtracts 32 and then uses multiplication within the \texttt{print}. -This first rule demonstrates an important property of p-rules: although the patterns regex-2 and regex-8 are not sufficient for a program to be correct, as it is trivial to implement an incorrect program containing both patterns, only one out of 201 submissions macthing these two patterns was incorrect. This suggests that the conditions of p-rules represent the critical elements of the solution. When a students figures these out, implementing the rest of the program shold be straightforward. A sample program matching the first rule is: -\begin{Verbatim}[fontfamily=tt] -g1 = \underline{float(input(}'Temperature [F]: ')) -print(((g1 \underline{- 32) *} (5 / 9))) +This first rule demonstrates an important property of p-rules: although the patterns regex-2 and regex-8 are not sufficient for a program to be correct (it is trivial to implement an incorrect program containing both patterns), only one out of 201 submissions macthing these two patterns was incorrect. This suggests that the conditions of p-rules represent the critical elements of the solution. When a students figures these two out, implementing the rest of the program should be straightforward. A sample program matching the first rule is: +\begin{Verbatim} +g1 = \blue{float(input(}'Temperature [F]: ')) +print(((g1 \red{- 32) *} (5 / 9))) \end{Verbatim} -The second and the third p-rules are variations of the first. For example, the second rule describes programs that have the formula in the second argument of the \texttt{print} function. The fourth rule, however, is different. The pattern regex-80 describes programs that subtract 32 from a variable, which is casted to float. The following program matches regex-80:: -\begin{Verbatim}[fontfamily=tt] +The second and the third p-rules are variations of the first. For example, the second rule describes programs that have the formula in the second argument of the \texttt{print} function. The fourth rule, however, is different. The pattern regex-80 describes programs that subtract 32 from a variable, which is casted to float. The following program matches regex-80: +\begin{Verbatim} g1 = input('Fahrenheit?') -g0 = (\underline{(float(g1) - 32)} * (5 / 9)) +g0 = (\blue{(float(g1) - 32)} * (5 / 9)) print(g0) \end{Verbatim} -\subsection{Buy Five} - \subsection{Greatest Absolutist} +In this more complex exercise a student has to implement a function named \texttt{max\_abs} that accepts a list of numbers as an argument and returns the number with the largest absolute value. A sample correct solution is: +\begin{Verbatim}[fontfamily=tt] +def max_abs(l): + vmax = l[0] + for v in l: + if (abs(v) > abs(vmax)): + vmax = v + return vmax +\end{Verbatim} + +We have received 155 submissions (57 correct, 98 incorrect) for this exercise. Due to its higher complexity and since the solutions are much more diverse, we obtained 8298 patterns to be used as attributes in learning. High number of patterns together with a low number of learning examples could present a problem for rule learning: since the space of possible rules is large, some of the learned rules might be a result of statistical anomalies. One needs to apply a certain amount of caution when interpreting these rules. + +The rule-learning algorithm learned 15 n-rules (7 mentioning only presence of patterns) and 6 p-rules. Bellow we can see the two best n-rules referring to the presence of patterns and two programs; the left one is covered by the first rule, and the right one by the second rule: +\begin{Verbatim} +IF regex-64==T THEN correct=F [22 0] +IF regex-2==T AND regex-70==T THEN correct=F [17 0] +\end{Verbatim} + +\begin{Verbatim} +def max_abs(l): def max_abs(l): + vmax = 0 vmax = None + for i in range(len(l)): for v in l: + if \blue{vmax} < abs(l[i]): if vmax==None or vmax