1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
|
What is kpov_judge
kpov_judge is a system for the automated grading of assignments for
networking and system administration courses.
Each assignment is represented by a task which a student should perform.
Tasks are customised for each student by setting a number of parameters
for each task.
Typically, each task involves a number of interconnected computers, one
of which contains the kpov_judge testing program (test_task.py). After
solving the task, a student runs the test_task.py to check their solution.
The testing results are uploaded to a server for grading and feedback can
be provided immediately.
Using kpov_judge as a student
For each task, the student is supposed to set up some services or resources
on one or multiple computers. The services or resources are then checked to
see if the task has been performed successfully. The result of the test is
reported back to the student.
The instructions for each task are included in the task itself. The
parameters for each task are either prepared for each student in advance or
generated the moment a student tries to check whether they have performed
the task successfully.
The computers for each task are typically set up by downloading a set of
virtual disk images, creating a number of virtual machines using these
disks and connecting these virtual machines using virtual or physical
networks. After the computers are set up, a student can set up the services
required by the instructions.
The checking is done by the program test_task.py. This program can run in
both an on-line mode where some of the testing is done by a remote server
or an off-line mode where all the testing is done locally. Since kpov_judge
is expected to be mostly used by schools, usually the on-line mode will be
used by students for handing in graded assignments and for immediate
feedback during practice.
Using kpov_judge as a teacher
TBD
Preparing tasks
Within the kpov_judge repository, tasks are stored under kpov_judge/tasks.
New tasks should be added as sub-directories of tasks.
Each task is defined by a python file named task.py . The file task.py
contains:
- instructions - the task instructions in multiple languages,
- computers - a dictionary describing the computers used for this task,
- networks - a dictionary describing the network links for this task,
- params_meta - a dictionary with a description and other metadata for
each parameter for this task
- task(...) - a function for testing the student's solution of this task.
All the arguments for this function should be described in
params_meta and should be public. This function is run by
the student to test their task solution. The output of
task(...) should be relatively hard to fake - for example,
if the student had to set up the default route on one of
the computers, this function could return a string
containing the output of the 'netstat -r' command. The
idea is to make it harder for the student to fake the
correct output of task(...) than it would be to just
solve the task.
- task_check(results, params) - check the output of task(...).
The parameter results contains the return value of task(...).
The parameter params contains a dictionary of values for each
parameter used for this task
This function is used for grading the task and should return
a value between 0 and 10 with 0 representing the lowest and 10
representing the highest number of points attainable for solving
the task.
- gen_params(user_id, params_meta) - prepare all the parameters for
this task. For most tasks, most of the parameters will be randomly
generated according to the metadata in params_meta. A number of
helper functions for generating the parameters are available in
kpov_random_helpers.py
- prepare_disks(templates, params) - prepare the disk images for this
task. For some tasks it might be neccessarry to create or edit
some files on the virtual disk images used by each student to set
up the computers needed for this task. The parameter templates
contains a dictionary of guestfs objects. Refer to the libguestfs
documentation for more information.
Typically, a new task is created by the following steps:
- prepare a (virtual) testing computer
- checkout the kpov_judge repository on the testing computer
- create a copy of an existing task to use as reference
- change the instructions in task.py
- create the other computers and networks involved in this task
- change the dictionaries computers and networks in task.py
- change the params_meta to correspond to the new instructions.
- write a task(...) function which returns a simple string
- write a task_check function which simply prints out the parameter
results
- write a gen_params(user_id, params_meta) function
- run test_task -g to test gen_params and set the initial task parameters
- debug everything you have created so far by performing the task
and running test_task.py as many times as needed
- write the task_check and gen_params functions
- write the prepare_disks function
- commit the new task to the repository
- checkout the task on the web-based evaluation system
- upload the clean virtual disk images, add the task to the web-based
system
- log in as a student, download the images, solve task, run task_check
- debug the task if needed
For each course taught using kpov_judge, a directory should be created
under kpov_judge/courses. Each course's directory can be arranged in any
way the lecturer sees fit. For fri_kpov, the directory contains one
sub-directory for each lesson. Each lesson consists of a task the student
should perform before attending class (preparation), the instructions
for that week's class (lecture) and a task the student will be graded on
after the class (evaluation).
|
