Autograding issues - compromised source code - getting worse.

[cmh42]

The problem with compromised (source code of) cells in assignment notebooks (via Formgrader/Autograder) is getting worse. In a recent Assignment 5% of the notebooks (15 out of 300) either broke the batch autograding process (most of them in fact) or simply failed to auto grade. A couple of years ago this sort of problem could usually resolved by tweaking the compromised part of the source code of the notebook. But no longer - the problem seems to cascade through the source code. So the only viable option is to copy across all of the students work from their compromised notebook to a copy of the original release version. This is very time consuming. I have asked students for details of what they did. However there is clearly a bug in the formgrader setup which sometimes allows students to change the cell type (usually from code to markdown). Another problem seems to be when students insert or delete cells. For insertion Formgrader needs to be set up so that either (1) the insertion of new cell is impossible or (2) the insertion of new cells is possible but this does not affect autograding. For deletion - students should only be able to delete new cells that they have inserted (if this is a possibility - as per my last point).

Here is a typical error log (they are all the same as this in fact). The Homework of student 26-lz23126 is one of those that broke the batch autograding process. The error message is totally inappropriate.

[AutogradeApp | WARNING] Removing failed assignment: /home/jovyan/instructor-courses/MATH20014_2024_TB-1/autograded/Homework3_Handgraded/26-lz23126
[AutogradeApp | WARNING] Removing failed assignment: /home/jovyan/instructor-courses/MATH20014_2024_TB-1/autograded/Homework3_Handgraded/26-lz23126
[AutogradeApp | ERROR] One or more notebooks in the assignment use an old version
of the nbgrader metadata format. Please back up your class files
directory and then update the metadata using:

nbgrader update .

[AutogradeApp | ERROR] One or more notebooks in the assignment use an old version
of the nbgrader metadata format. Please back up your class files
directory and then update the metadata using:

nbgrader update .

[perllaghu]

I think a chunk of this is actually a jupyterlab/notebook problem, rather than an nbgrader issue.

By that, I mean that adding/removing/duplicating cells is a function of the notebook, and nbgrader has not real control there.

Jupyterlab/notebook has a way of marking a notebook "uneditable", and nbgrader can mark a cell "read-only".... but neither will stop duplication.