diff options
Diffstat (limited to 'kpov_judge/README.txt')
| -rw-r--r-- | kpov_judge/README.txt | 122 |
1 files changed, 0 insertions, 122 deletions
diff --git a/kpov_judge/README.txt b/kpov_judge/README.txt deleted file mode 100644 index 1a9e772..0000000 --- a/kpov_judge/README.txt +++ /dev/null @@ -1,122 +0,0 @@ -What is kpov_judge - - kpov_judge is a system for the automated grading of assignments for - networking and system administration courses. - - Each assignment is represented by a task which a student should perform. - Tasks are customised for each student by setting a number of parameters - for each task. - - Typically, each task involves a number of interconnected computers, one - of which contains the kpov_judge testing program (test_task.py). After - solving the task, a student runs the test_task.py to check their solution. - The testing results are uploaded to a server for grading and feedback can - be provided immediately. - -Using kpov_judge as a student - - For each task, the student is supposed to set up some services or resources - on one or multiple computers. The services or resources are then checked to - see if the task has been performed successfully. The result of the test is - reported back to the student. - - The instructions for each task are included in the task itself. The - parameters for each task are either prepared for each student in advance or - generated the moment a student tries to check whether they have performed - the task successfully. - - The computers for each task are typically set up by downloading a set of - virtual disk images, creating a number of virtual machines using these - disks and connecting these virtual machines using virtual or physical - networks. After the computers are set up, a student can set up the services - required by the instructions. - - The checking is done by the program test_task.py. This program can run in - both an on-line mode where some of the testing is done by a remote server - or an off-line mode where all the testing is done locally. Since kpov_judge - is expected to be mostly used by schools, usually the on-line mode will be - used by students for handing in graded assignments and for immediate - feedback during practice. - -Using kpov_judge as a teacher - - TBD - -Preparing tasks - - Within the kpov_judge repository, tasks are stored under kpov_judge/tasks. - New tasks should be added as sub-directories of tasks. - - Each task is defined by a python file named task.py . The file task.py - contains: - - instructions - the task instructions in multiple languages, - - computers - a dictionary describing the computers used for this task, - - networks - a dictionary describing the network links for this task, - - params_meta - a dictionary with a description and other metadata for - each parameter for this task - - task(...) - a function for testing the student's solution of this task. - All the arguments for this function should be described in - params_meta and should be public. This function is run by - the student to test their task solution. The output of - task(...) should be relatively hard to fake - for example, - if the student had to set up the default route on one of - the computers, this function could return a string - containing the output of the 'netstat -r' command. The - idea is to make it harder for the student to fake the - correct output of task(...) than it would be to just - solve the task. - - task_check(results, params) - check the output of task(...). - The parameter results contains the return value of task(...). - The parameter params contains a dictionary of values for each - parameter used for this task - This function is used for grading the task and should return - a value between 0 and 10 with 0 representing the lowest and 10 - representing the highest number of points attainable for solving - the task. - - gen_params(user_id, params_meta) - prepare all the parameters for - this task. For most tasks, most of the parameters will be randomly - generated according to the metadata in params_meta. A number of - helper functions for generating the parameters are available in - kpov_util.py - - prepare_disks(templates, params) - prepare the disk images for this - task. For some tasks it might be neccessarry to create or edit - some files on the virtual disk images used by each student to set - up the computers needed for this task. The parameter templates - contains a dictionary of guestfs objects. Refer to the libguestfs - documentation for more information. - - Typically, a new task is created by the following steps: - - prepare a (virtual) testing computer - - checkout the kpov_judge repository on the testing computer - - create a copy of an existing task to use as reference - - change the instructions in task.py - - create the other computers and networks involved in this task - - change the dictionaries computers and networks in task.py - - change the params_meta to correspond to the new instructions. - - write a task(...) function which returns a simple string - - write a task_check function which simply prints out the parameter - results - - write a gen_params(user_id, params_meta) function - - run test_task -g to test gen_params and set the initial task parameters - - debug everything you have created so far by performing the task - and running test_task.py as many times as needed - - write the task_check and gen_params functions - - write the prepare_disks function - - commit the new task to the repository - - checkout the task on the web-based evaluation system - - upload the clean virtual disk images, add the task to the web-based - system - - log in as a student, download the images, solve task, run task_check - - debug the task if needed - - For each course taught using kpov_judge, a directory should be created - under kpov_judge/courses. Each course's directory can be arranged in any - way the lecturer sees fit. For fri_kpov, the directory contains one - sub-directory for each lesson. Each lesson consists of a task the student - should perform before attending class (preparation), the instructions - for that week's class (lecture) and a task the student will be graded on - after the class (evaluation). - - - - |