summaryrefslogtreecommitdiff
path: root/kpov_judge/README.txt
blob: b0752310159b53407982e7cc58f5c336788ced40 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
What is kpov_judge
    
    kpov_judge is a system for the automated grading of assignments for
    networking and system administration courses.

    Each assignment is represented by a task which a student should perform.
    Tasks are customised for each student by setting a number of parameters
    for each task.

    Typically, each task involves a number of interconnected computers, one 
    of which contains the kpov_judge testing program (test_task.py). After
    solving the task, a student runs the test_task.py to check their solution.
    The testing results are uploaded to a server for grading and feedback can
    be provided immediately.

Using kpov_judge as a student
    
    For each task, the student is supposed to set up some services or resources
    on one or multiple computers. The services or resources are then checked to
    see if the task has been performed successfully. The result of the test is 
    reported back to the student.
    
    The instructions for each task are included in the task itself. The
    parameters for each task are either prepared for each student in advance or
    generated the moment a student tries to check whether they have performed
    the task successfully.

    The computers for each task are typically set up by downloading a set of 
    virtual disk images, creating a number of virtual machines using these 
    disks and connecting these virtual machines using virtual or physical
    networks. After the computers are set up, a student can set up the services
    required by the instructions.

    The checking is done by the program test_task.py. This program can run in 
    both an on-line mode where some of the testing is done by a remote server
    or an off-line mode where all the testing is done locally. Since kpov_judge
    is expected to be mostly used by schools, usually the on-line mode will be
    used by students for handing in graded assignments and for immediate 
    feedback during practice.
    
Using kpov_judge as a teacher
   
    TBD

Preparing tasks
    
    Within the kpov_judge repository, tasks are stored under kpov_judge/tasks.
    New tasks should be added as sub-directories of tasks.

    Each task is defined by a python file named task.py . The file task.py 
    contains:
        - instructions - the task instructions in multiple languages,
        - computers - a dictionary describing the computers used for this task, 
        - networks - a dictionary describing the network links for this task, 
        - params_meta - a dictionary with a description and other metadata for
            each parameter for this task
        - task(...) - a function for testing the student's solution of this task.
            All the arguments for this function should be described in
            params_meta and should be public. This function is run by
            the student to test their task solution. The output of
            task(...) should be relatively hard to fake - for example,
            if the student had to set up the default route on one of
            the computers, this function could return a string
            containing the output of the 'netstat -r' command. The
            idea is to make it harder for the student to fake the
            correct output of task(...) than it would be to just
            solve the task.
        - task_check(results, params) - check the output of task(...).
            The parameter results contains the return value of task(...).
            The parameter params contains a dictionary of values for each
            parameter used for this task
            This function is used for grading the task and should return
            a value between 0 and 10 with 0 representing the lowest and 10
            representing the highest number of points attainable for solving
            the task.
        - gen_params(user_id, params_meta) - prepare all the parameters for 
            this task. For most tasks, most of the parameters will be randomly
            generated according to the metadata in params_meta. A number of
            helper functions for generating the parameters are available in
            kpov_random_helpers.py
        - prepare_disks(templates, params) - prepare the disk images for this
            task. For some tasks it might be neccessarry to create or edit
            some files on the virtual disk images used by each student to set
            up the computers needed for this task. The parameter templates
            contains a dictionary of guestfs objects. Refer to the libguestfs
            documentation for more information.

    Typically, a new task is created by the following steps:
        - prepare a (virtual) testing computer
        - checkout the kpov_judge repository on the testing computer
        - create a copy of an existing task to use as reference
        - change the instructions in task.py
        - create the other computers and networks involved in this task
        - change the dictionaries computers and networks in task.py
        - change the params_meta to correspond to the new instructions.
        - write a task(...) function which returns a simple string
        - write a task_check function which simply prints out the parameter
            results
        - write a gen_params(user_id, params_meta) function
        - run test_task -g to test gen_params and set the initial task parameters
        - debug everything you have created so far by performing the task
            and running test_task.py as many times as needed
        - write the task_check and gen_params functions
        - write the prepare_disks function
        - commit the new task to the repository
        - checkout the task on the web-based evaluation system
        - upload the clean virtual disk images, add the task to the web-based
            system
        - log in as a student, download the images, solve task, run task_check
        - debug the task if needed

    For each course taught using kpov_judge, a directory should be created
    under kpov_judge/courses. Each course's directory can be arranged in any
    way the lecturer sees fit. For fri_kpov, the directory contains one 
    sub-directory for each lesson. Each lesson consists of a task the student
    should perform before attending class (preparation), the instructions
    for that week's class (lecture) and a task the student will be graded on
    after the class (evaluation).