diff options
Diffstat (limited to 'README.txt')
| -rw-r--r-- | README.txt | 122 |
1 files changed, 122 insertions, 0 deletions
diff --git a/README.txt b/README.txt new file mode 100644 index 0000000..1a9e772 --- /dev/null +++ b/README.txt @@ -0,0 +1,122 @@ +What is kpov_judge + + kpov_judge is a system for the automated grading of assignments for + networking and system administration courses. + + Each assignment is represented by a task which a student should perform. + Tasks are customised for each student by setting a number of parameters + for each task. + + Typically, each task involves a number of interconnected computers, one + of which contains the kpov_judge testing program (test_task.py). After + solving the task, a student runs the test_task.py to check their solution. + The testing results are uploaded to a server for grading and feedback can + be provided immediately. + +Using kpov_judge as a student + + For each task, the student is supposed to set up some services or resources + on one or multiple computers. The services or resources are then checked to + see if the task has been performed successfully. The result of the test is + reported back to the student. + + The instructions for each task are included in the task itself. The + parameters for each task are either prepared for each student in advance or + generated the moment a student tries to check whether they have performed + the task successfully. + + The computers for each task are typically set up by downloading a set of + virtual disk images, creating a number of virtual machines using these + disks and connecting these virtual machines using virtual or physical + networks. After the computers are set up, a student can set up the services + required by the instructions. + + The checking is done by the program test_task.py. This program can run in + both an on-line mode where some of the testing is done by a remote server + or an off-line mode where all the testing is done locally. Since kpov_judge + is expected to be mostly used by schools, usually the on-line mode will be + used by students for handing in graded assignments and for immediate + feedback during practice. + +Using kpov_judge as a teacher + + TBD + +Preparing tasks + + Within the kpov_judge repository, tasks are stored under kpov_judge/tasks. + New tasks should be added as sub-directories of tasks. + + Each task is defined by a python file named task.py . The file task.py + contains: + - instructions - the task instructions in multiple languages, + - computers - a dictionary describing the computers used for this task, + - networks - a dictionary describing the network links for this task, + - params_meta - a dictionary with a description and other metadata for + each parameter for this task + - task(...) - a function for testing the student's solution of this task. + All the arguments for this function should be described in + params_meta and should be public. This function is run by + the student to test their task solution. The output of + task(...) should be relatively hard to fake - for example, + if the student had to set up the default route on one of + the computers, this function could return a string + containing the output of the 'netstat -r' command. The + idea is to make it harder for the student to fake the + correct output of task(...) than it would be to just + solve the task. + - task_check(results, params) - check the output of task(...). + The parameter results contains the return value of task(...). + The parameter params contains a dictionary of values for each + parameter used for this task + This function is used for grading the task and should return + a value between 0 and 10 with 0 representing the lowest and 10 + representing the highest number of points attainable for solving + the task. + - gen_params(user_id, params_meta) - prepare all the parameters for + this task. For most tasks, most of the parameters will be randomly + generated according to the metadata in params_meta. A number of + helper functions for generating the parameters are available in + kpov_util.py + - prepare_disks(templates, params) - prepare the disk images for this + task. For some tasks it might be neccessarry to create or edit + some files on the virtual disk images used by each student to set + up the computers needed for this task. The parameter templates + contains a dictionary of guestfs objects. Refer to the libguestfs + documentation for more information. + + Typically, a new task is created by the following steps: + - prepare a (virtual) testing computer + - checkout the kpov_judge repository on the testing computer + - create a copy of an existing task to use as reference + - change the instructions in task.py + - create the other computers and networks involved in this task + - change the dictionaries computers and networks in task.py + - change the params_meta to correspond to the new instructions. + - write a task(...) function which returns a simple string + - write a task_check function which simply prints out the parameter + results + - write a gen_params(user_id, params_meta) function + - run test_task -g to test gen_params and set the initial task parameters + - debug everything you have created so far by performing the task + and running test_task.py as many times as needed + - write the task_check and gen_params functions + - write the prepare_disks function + - commit the new task to the repository + - checkout the task on the web-based evaluation system + - upload the clean virtual disk images, add the task to the web-based + system + - log in as a student, download the images, solve task, run task_check + - debug the task if needed + + For each course taught using kpov_judge, a directory should be created + under kpov_judge/courses. Each course's directory can be arranged in any + way the lecturer sees fit. For fri_kpov, the directory contains one + sub-directory for each lesson. Each lesson consists of a task the student + should perform before attending class (preparation), the instructions + for that week's class (lecture) and a task the student will be graded on + after the class (evaluation). + + + + |