README.md 10.2 KB
Newer Older
Alan Marchiori's avatar
Alan Marchiori committed
1
2
# labtool

Alan Marchiori's avatar
push    
Alan Marchiori committed
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
Labtool, **lt**, is a tool to help automated CS lab courses developed at Bucknell University. It assumes student will use the CLI and will submit work via git using a gitlab server. Course rubrics are stored on the same file system that holds the **lt** executable. These must be globally readable files.

## Student guide

As a student, you first have to update your path to find lt.

```
$ echo "PATH=\$PATH:~cs206/bin:~cs206/bin/lt" >> ~/.bashrc
$ source ~/.bashrc
```

Then you can verify that lt runs.

`$ lt test`

And read the online help.

`$ lt help`

Then you need to initialize a course.

`$ lt init <COURSE> <SECTION> <SEMESTER>`

This will perform the following actions:
1. store your gitlab personal access token in your `local_conf` file.
2. ensure you have an ssh key (id_rsa.pub) and add this to your gitlab account.
3. create the correctly named course on gitlab, adding all instructors and TAs as maintainers.
4. clone the gitlab repo locally
5. copy the .gitignore from the course definition to the local repo, push the .gitignore to gitlab.


Then you can use the `lt check` command to check labs.

## TA guide

If you haven't already, add the path to labtool:
```
$ echo "PATH=\$PATH:~cs206/bin:~cs206/bin/lt" >> ~/.bashrc
$ source ~/.bashrc
```

Then you can verify that lt runs.

`$ lt test`

And read the online help.

Alan Marchiori's avatar
Alan Marchiori committed
50
`$ lt --help`
Alan Marchiori's avatar
push    
Alan Marchiori committed
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235

Then you need to initialize a course with the TA flag.

`$ lt init --ta <COURSE> <SECTION> <SEMESTER>`

You will now have a folder like **CSCI206-SEMESTER-SECTION-TA**. If you want this to not be in the root of your home directory, use the --prefix option.

Change into the newly created folder. You will see it is currently empty. The grade command will search and clone all student repos for this course/section/semester.

The main command you will use


## Global configuration

The tool is meant to be flexible with most configuration stored at the COURSE level. However, the file `config/__init__.py` contains all globally defined values.

Courses are found by searching the given list of `course_paths`. A course is defined by creating a file named `courses.json` in one of these paths. If multiple `courses.json` files are found, they are all loaded. Each file can define multiple SECTIONS of a course. See the [Course rubrics section](#course-rubrics) for more information.

## Deployment at Bucknell
In the `Makefile` there is a `dist` target that compiles the python code with pyinstaller. This is to provide some level of obfuscation to discourage students from trying to break/exploit the tool. There is a symbolic link in ~cs206/bin to the labtool dist path where the **lt** executable is generated.

Upon deploying a new executable it is important to verify that it runs correctly `$ lt test`. Sometimes you will see errors like:

    [24843] Error loading Python lib '/nfs/unixspace/linux/accounts/COURSES/cs206/labtool/dist/lt/libpython3.6m.so': dlopen:
    /nfs/unixspace/linux/accounts/COURSES/cs206/labtool/dist/lt/libpython3.6m.so: cannot open shared object file: No such file or directory


If you see these, run a `make clean` and then `make dist`. From what I can tell something is cached incorrectly in the `lt.spec` file. A clean build solves this problem.

## Course rubrics

A course is defined by creating a file `courses.json` in the root of any of the `course_paths`. This is a very simple file containing one object where the keys are course names and the value is a filesystem path the rubric (relative to the location fo the `courses.json` file).

For example, in `~cs206/` we have a `courses.json` file with the contents:
```
{
  "CSCI206-S20-60": "2020-spring/csci206_s20.json",
  "CSCI206-S20-61": "2020-spring/csci206_s20.json",
  "CSCI206-S20-62": "2020-spring/csci206_s20.json"
}
```

This defines three sections of CSCI206 for the spring 2020 semester. The `course_name_fmt` we use is NAME-SEMESTER-SECTION. This is configurable in the global config file. Students will enter each part (name, semester, section) separately and we need a common format to find the course rubric.

In this example, each section of the course uses the same rubric (`2020-spring/csci206_s20.json`).

The labs for each course are defined in their own json files in the path given in the `courses.json` file.

### Defining a course/section

The main `courses.json` file points to a course/section json file which defines all course-specific information.

The file `csci206_s20.json` contains:
```
{
    "name": "CSCI206",
    "instructors": ["amm042","xmeng","jvs008"],
    "tas": ["gic004", "mat029", "jcd023"],
    "rubric_paths":[
        "/home/accounts/COURSES/cs206/2020-spring/rubrics",
        "/nfs/unixspace/linux/accounts/COURSES/cs206/2020-spring/rubrics",
        "/unixspace/csci206/2020-spring/rubrics"]
}
```

All of these fields are required and should be fairly self-explanatory. The instructors and tas must be consistent with usernames on the *gitlab server*! The `rubric_paths` is yet another set of search paths to find rubrics for this particular course. We have multiple mountpoints to handle different machine configurations, ensuring the rubrics can always be found. However, when searching for a rubric, the *first* usable rubric is selected.

In the `rubric_path` you will create a rubric for each lab using the file name `lab##.json` where ## is a two-digit number that corresponds to the lab number. For example, lab 1 would be defined in the file `lab01.json`. This is not configurable.

### Defining a lab rubric

As an example, here is the `lab01.json` file.
```
{
	"course": "csci206",
	"version": 1,
	"name": "lab01",
	"path": "Labs/Lab01",
	"parts":
			{"setup":
				{
					"index": 3,
					"name": "Course organization",
					"prompt": "Create the Lab01 folder and create the 'empty' file in the right location.",
					"points": 10,
					"check": [{
						"is_empty": ["empty"]
					}],
					"grade": [
						{"execute": ["ls -lah empty"]}
					],
					"on_error": "Failed: Be sure you created the file 'empty' and it is zero bytes. Hint: use the touch command."
				},
			"create_text":
				{
					"index": 4,
					"name": "Creating and viewing a text file",
					"points": 10,

					"prompt": "Create the lab01.txt file as instructed in the lab.",
					"check": [
						{"wc":[{"lab01.txt":{"min_lines": 4}}],
						"contains":[{"lab01.txt": ["CSCI206", "Lab01"]}]
						}],
					"grade": [
						{"execute": ["head -n 4 lab01.txt"]}
					]
				},
			"answers":
				{
					"index": 5,
					"name": "The editor wars",
					"points": 80,

					"prompt": "Answer the questions on the grading rubric.",

					"check": [
					 	{	"wc":[{"lab01.txt":{"min_lines": 10}}],
							"contains":[{"lab01.txt": ["vim", "emacs", "After careful consideration", "Lab01", "ls", "pwd", "cp", "mkdir", "rmdir", "touch", "cat", "more", "less", "head", "tail"]}]
							}],
					"grade": [
						{"execute": ["cat lab01.txt"]}]
				}
			}
}
```

The first few fields define what course this rubric belongs to. It is important to note that **path** is used to construct the path to student work from the root of the student's git repository.
* "course": "csci206"
* "version": 1
* "name": "lab01"
* "path": "Labs/Lab01"

Then you will see an object called **parts**. This corresponds to each logical part of the lab. Each part is identical. Below is the first part:

```
{"setup":
  {
    "index": 3,
    "name": "Course organization",
    "prompt": "Create the Lab01 folder and create the 'empty' file in the right location.",
    "points": 10,
    "check": [{
      "is_empty": ["empty"]
    }],
    "grade": [
      {"execute": ["ls -lah empty"]}
    ],
    "on_error": "Failed: Be sure you created the file 'empty' and it is zero bytes. Hint: use the touch command."
  }
```

This part is called **setup** and the numerical part is defined by the **index** (3 in this case). This defines ordering of the lab parts (not the position in the json file). Parts are sorted by index.
* name: is a short name for this part of the lab.
* prompt: is a longer prompt used to grade this part (students and TAs see this).
* points: is the number of points this part of the lab is worth.
* check: is a list of checks that the *check* command executes.
* grade: is a list of checks that the *grade* command executes (TA only).

Each check object is a key: value, where key is the type of check and the value is passed as the arguments. These are defined in the file `comamnds/checker.py` and that is the best place to go for details. The checks are executed in the order defined in the json file. For the check command, we stop as soon as a check fails. The grade command continues executing all checks even if there is a failure.

The most common check is **execute**, this executes a command or list of commands and performs checks on the output.

A more complex check is below.
```
"check": [
						{"exists": ["nogood.o", "nogood"]},

						{"execute": [
							{"file nogood.o":{
								"message": "Checking if nogood.o is a compiled object.",
								"diff": false,
								"stdout": "nogood.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped\n"
							}},
							"rm -f nogood",
							{"ld -o nogood nogood.o /usr/lib64/crt1.o /usr/lib64/crti.o /usr/lib64/crtn.o -dynamic-linker /lib64/ld-linux-x86-64.so.2 -lc": {"returncode": 0,
								"message": "Linking nogood.o and checking output"}},
							{"./nogood":{
									"stdout": "n = 5, n squared = 25, n cubed = 125\n"
								}}
						]}
					]
```

This first checks if the files "nogood.o" and "nogood" exist. Then the command "file nogood.o" is executed. The message is shown to the user. diff: false turns off diff output if the stdout check fails. The stdout parameter is compared to the output of the file command. If it doesn't match, the check fails. By default it would display a diff, but since we explicitly disabled it, it just fails.  Next, we execute "rm -f nogood", this file was generated by a previous check and we want to regenerate it. Note, this is just a string not an object, so no checks are performed. It is just executed. Then a long link command is executed and we check that the return code is zero. Then we run the newly generated nogood file and verify the stdout matches the given string. Note for stdin and stdout, you can define a single string or list of strings. Lists will be joined with a newline character to form a single string.