Commit bd42c2cc authored by Alan Marchiori's avatar Alan Marchiori
Browse files

push

parent 4df328f4
# labtool
Labtool, **lt**, is a tool to help automated CS lab courses developed at Bucknell University. It assumes student will use the CLI and will submit work via git using a gitlab server. Course rubrics are stored on the same file system that holds the **lt** executable. These must be globally readable files.
## Student guide
As a student, you first have to update your path to find lt.
```
$ echo "PATH=\$PATH:~cs206/bin:~cs206/bin/lt" >> ~/.bashrc
$ source ~/.bashrc
```
Then you can verify that lt runs.
`$ lt test`
And read the online help.
`$ lt help`
Then you need to initialize a course.
`$ lt init <COURSE> <SECTION> <SEMESTER>`
This will perform the following actions:
1. store your gitlab personal access token in your `local_conf` file.
2. ensure you have an ssh key (id_rsa.pub) and add this to your gitlab account.
3. create the correctly named course on gitlab, adding all instructors and TAs as maintainers.
4. clone the gitlab repo locally
5. copy the .gitignore from the course definition to the local repo, push the .gitignore to gitlab.
Then you can use the `lt check` command to check labs.
## TA guide
If you haven't already, add the path to labtool:
```
$ echo "PATH=\$PATH:~cs206/bin:~cs206/bin/lt" >> ~/.bashrc
$ source ~/.bashrc
```
Then you can verify that lt runs.
`$ lt test`
And read the online help.
`$ lt help`
Then you need to initialize a course with the TA flag.
`$ lt init --ta <COURSE> <SECTION> <SEMESTER>`
You will now have a folder like **CSCI206-SEMESTER-SECTION-TA**. If you want this to not be in the root of your home directory, use the --prefix option.
Change into the newly created folder. You will see it is currently empty. The grade command will search and clone all student repos for this course/section/semester.
The main command you will use
## Global configuration
The tool is meant to be flexible with most configuration stored at the COURSE level. However, the file `config/__init__.py` contains all globally defined values.
Courses are found by searching the given list of `course_paths`. A course is defined by creating a file named `courses.json` in one of these paths. If multiple `courses.json` files are found, they are all loaded. Each file can define multiple SECTIONS of a course. See the [Course rubrics section](#course-rubrics) for more information.
## Deployment at Bucknell
In the `Makefile` there is a `dist` target that compiles the python code with pyinstaller. This is to provide some level of obfuscation to discourage students from trying to break/exploit the tool. There is a symbolic link in ~cs206/bin to the labtool dist path where the **lt** executable is generated.
Upon deploying a new executable it is important to verify that it runs correctly `$ lt test`. Sometimes you will see errors like:
[24843] Error loading Python lib '/nfs/unixspace/linux/accounts/COURSES/cs206/labtool/dist/lt/libpython3.6m.so': dlopen:
/nfs/unixspace/linux/accounts/COURSES/cs206/labtool/dist/lt/libpython3.6m.so: cannot open shared object file: No such file or directory
If you see these, run a `make clean` and then `make dist`. From what I can tell something is cached incorrectly in the `lt.spec` file. A clean build solves this problem.
## Course rubrics
A course is defined by creating a file `courses.json` in the root of any of the `course_paths`. This is a very simple file containing one object where the keys are course names and the value is a filesystem path the rubric (relative to the location fo the `courses.json` file).
For example, in `~cs206/` we have a `courses.json` file with the contents:
```
{
"CSCI206-S20-60": "2020-spring/csci206_s20.json",
"CSCI206-S20-61": "2020-spring/csci206_s20.json",
"CSCI206-S20-62": "2020-spring/csci206_s20.json"
}
```
This defines three sections of CSCI206 for the spring 2020 semester. The `course_name_fmt` we use is NAME-SEMESTER-SECTION. This is configurable in the global config file. Students will enter each part (name, semester, section) separately and we need a common format to find the course rubric.
In this example, each section of the course uses the same rubric (`2020-spring/csci206_s20.json`).
The labs for each course are defined in their own json files in the path given in the `courses.json` file.
### Defining a course/section
The main `courses.json` file points to a course/section json file which defines all course-specific information.
The file `csci206_s20.json` contains:
```
{
"name": "CSCI206",
"instructors": ["amm042","xmeng","jvs008"],
"tas": ["gic004", "mat029", "jcd023"],
"rubric_paths":[
"/home/accounts/COURSES/cs206/2020-spring/rubrics",
"/nfs/unixspace/linux/accounts/COURSES/cs206/2020-spring/rubrics",
"/unixspace/csci206/2020-spring/rubrics"]
}
```
All of these fields are required and should be fairly self-explanatory. The instructors and tas must be consistent with usernames on the *gitlab server*! The `rubric_paths` is yet another set of search paths to find rubrics for this particular course. We have multiple mountpoints to handle different machine configurations, ensuring the rubrics can always be found. However, when searching for a rubric, the *first* usable rubric is selected.
In the `rubric_path` you will create a rubric for each lab using the file name `lab##.json` where ## is a two-digit number that corresponds to the lab number. For example, lab 1 would be defined in the file `lab01.json`. This is not configurable.
### Defining a lab rubric
As an example, here is the `lab01.json` file.
```
{
"course": "csci206",
"version": 1,
"name": "lab01",
"path": "Labs/Lab01",
"parts":
{"setup":
{
"index": 3,
"name": "Course organization",
"prompt": "Create the Lab01 folder and create the 'empty' file in the right location.",
"points": 10,
"check": [{
"is_empty": ["empty"]
}],
"grade": [
{"execute": ["ls -lah empty"]}
],
"on_error": "Failed: Be sure you created the file 'empty' and it is zero bytes. Hint: use the touch command."
},
"create_text":
{
"index": 4,
"name": "Creating and viewing a text file",
"points": 10,
"prompt": "Create the lab01.txt file as instructed in the lab.",
"check": [
{"wc":[{"lab01.txt":{"min_lines": 4}}],
"contains":[{"lab01.txt": ["CSCI206", "Lab01"]}]
}],
"grade": [
{"execute": ["head -n 4 lab01.txt"]}
]
},
"answers":
{
"index": 5,
"name": "The editor wars",
"points": 80,
"prompt": "Answer the questions on the grading rubric.",
"check": [
{ "wc":[{"lab01.txt":{"min_lines": 10}}],
"contains":[{"lab01.txt": ["vim", "emacs", "After careful consideration", "Lab01", "ls", "pwd", "cp", "mkdir", "rmdir", "touch", "cat", "more", "less", "head", "tail"]}]
}],
"grade": [
{"execute": ["cat lab01.txt"]}]
}
}
}
```
The first few fields define what course this rubric belongs to. It is important to note that **path** is used to construct the path to student work from the root of the student's git repository.
* "course": "csci206"
* "version": 1
* "name": "lab01"
* "path": "Labs/Lab01"
Then you will see an object called **parts**. This corresponds to each logical part of the lab. Each part is identical. Below is the first part:
```
{"setup":
{
"index": 3,
"name": "Course organization",
"prompt": "Create the Lab01 folder and create the 'empty' file in the right location.",
"points": 10,
"check": [{
"is_empty": ["empty"]
}],
"grade": [
{"execute": ["ls -lah empty"]}
],
"on_error": "Failed: Be sure you created the file 'empty' and it is zero bytes. Hint: use the touch command."
}
```
This part is called **setup** and the numerical part is defined by the **index** (3 in this case). This defines ordering of the lab parts (not the position in the json file). Parts are sorted by index.
* name: is a short name for this part of the lab.
* prompt: is a longer prompt used to grade this part (students and TAs see this).
* points: is the number of points this part of the lab is worth.
* check: is a list of checks that the *check* command executes.
* grade: is a list of checks that the *grade* command executes (TA only).
Each check object is a key: value, where key is the type of check and the value is passed as the arguments. These are defined in the file `comamnds/checker.py` and that is the best place to go for details. The checks are executed in the order defined in the json file. For the check command, we stop as soon as a check fails. The grade command continues executing all checks even if there is a failure.
The most common check is **execute**, this executes a command or list of commands and performs checks on the output.
A more complex check is below.
```
"check": [
{"exists": ["nogood.o", "nogood"]},
{"execute": [
{"file nogood.o":{
"message": "Checking if nogood.o is a compiled object.",
"diff": false,
"stdout": "nogood.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped\n"
}},
"rm -f nogood",
{"ld -o nogood nogood.o /usr/lib64/crt1.o /usr/lib64/crti.o /usr/lib64/crtn.o -dynamic-linker /lib64/ld-linux-x86-64.so.2 -lc": {"returncode": 0,
"message": "Linking nogood.o and checking output"}},
{"./nogood":{
"stdout": "n = 5, n squared = 25, n cubed = 125\n"
}}
]}
]
```
This first checks if the files "nogood.o" and "nogood" exist. Then the command "file nogood.o" is executed. The message is shown to the user. diff: false turns off diff output if the stdout check fails. The stdout parameter is compared to the output of the file command. If it doesn't match, the check fails. By default it would display a diff, but since we explicitly disabled it, it just fails. Next, we execute "rm -f nogood", this file was generated by a previous check and we want to regenerate it. Note, this is just a string not an object, so no checks are performed. It is just executed. Then a long link command is executed and we check that the return code is zero. Then we run the newly generated nogood file and verify the stdout matches the given string. Note for stdin and stdout, you can define a single string or list of strings. Lists will be joined with a newline character to form a single string.
......@@ -189,18 +189,23 @@ def init(course, semester, section, prefix, ta):
if git.do_clone(repo['ssh_url_to_repo'], localpath=lp):
# copy cs206 gitignore and add to repo
igf = os.path.join(lp, '.gitignore')
src = "curl {} -o {}".format(
"http://eg.bucknell.edu/~cs206/.gitignore", igf)
c,t = run(src)
if c == 0:
success(src)
git.do_push(addfiles=['.gitignore'],
cwd=lp, message='Added by labtool')
else:
warn("Failed to initialize gitignore file.")
warn(src)
warn(t)
# src = "curl {} -o {}".format(
# "http://eg.bucknell.edu/~cs206/.gitignore", igf)
# c,t = run(src)
# if c == 0:
# success(src)
if '.gitignore' in coursedef:
if os.path.exists(os.expanduser(coursedef['.gitignore'])):
shutil.copyfile(os.expanduser(coursedef['.gitignore']),
lp)
git.do_push(addfiles=['.gitignore'],
cwd=lp, message='Added by labtool')
else:
warn("Failed to initialize gitignore file.")
warn(coursedef['.gitignore'])
success("Initialization complete!")
else:
error("Sorry, I was unable to clone the repo (did something go wrong earlier?)!")
......@@ -9,10 +9,17 @@ local_conf = "~/.labtool/config.json"
# course name for user's path and git repo
course_name_fmt = "{course}-{semester}-{section}"
# places to search for coures.json, all courses are loaded
course_paths = ['/home/accounts/COURSES/cs206',
'/nfs/unixspace/linux/accounts/COURSES/cs206/',
'/unixspace/csci206/']
# the default local_home is $HOME, it can be overridden with --prefix
local_home = "~/" + course_name_fmt
# shown to the user when asking for a Gitlab API token
gitlab_api_token_url = "https://gitlab.bucknell.edu/profile/personal_access_tokens"
# these fields are used to construct the gitlab server API endpoint
gitlab_url = "https://gitlab.bucknell.edu/"
gitlab_api = "api/v4/"
......@@ -4,6 +4,7 @@ import os.path
import json
import collections
import config
from config.echo import *
from config.user import UserConfig
......@@ -126,14 +127,9 @@ def load_courses(searchpath):
raise Exception("No courses file found in the course search path!")
return c
# places to search for coures.json, all courses are loaded
course_paths = ['/home/accounts/COURSES/cs206',
'/nfs/unixspace/linux/accounts/COURSES/cs206/',
'/unixspace/csci206/']
# dictionary of all course infos
# TODO this slows startup and may not be needed, move to lazy loading.
all = load_courses(course_paths)
all = load_courses(config.course_paths)
# define valid course strings and course definition search paths
#courses = ['CSCI206']
......
......@@ -35,9 +35,9 @@ smtp_handler = logging.handlers.SMTPHandler(
subject=u"labtool crash report!")
__version__ = '1.0.3'
__date__ = '2020-01-14T09:52:51.557670'
__date__ = '2020-01-20T12:25:27.481510'
__user__ = 'cs206'
__host__ = 'brki164-lnx-1.bucknell.edu'
__host__ = 'linuxremote3.bucknell.edu'
@click.group()
def main():
"""Bucknell University Computer Science lab tool [lt].
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment