Grading scanned answer sheets (Background)

At the University of Twente, paper multiple-choice exams are graded automatically by ConTest. After the exam, the Examinations Office scans the stack of answer forms, and the software grades every student and delivers the results as an Excel workbook. Figure 1 shows the two stages of that software. The questions that follow design one stage each. The stages share only two files, so each question can be answered without the other.

graph LR
    accTitle: Two stages of the grading software
    accDescr: The stack of scanned sheets goes into the stage Process sheets, which writes a results file and a key file. The stage Build workbook reads the two files and produces the Excel workbook.
    S[/"Stack of<br>scanned sheets"/] --> P[["Process sheets"]]
    P --> R[("Results file")]
    P --> K[("Key file")]
    R --> B[["Build workbook"]]
    K --> B
    B --> W[("Excel workbook")]
Figure 1: The two stages of the grading software. The first processes the scanned sheets into two files; the second builds the workbook from those files only.

The answer forms. ConTest offers two answer forms, which can be downloaded from the page above: form 38431 with up to 100 questions and options A to D, and form 25223 with up to 50 questions and options A to G. The positions of the bubbles differ between the two forms. The software reads three parts of a sheet:

  • the form code, printed in the top-left corner;
  • the student-number grid, with 7 columns, one per digit, and one bubble for each digit 1 through 9 and 0 in every column; and
  • one question row per question.

Name, course, version, and exam date are handwritten and are not read, so all students are graded with the same key. The teacher block and the true/false block of form 25223 are not used.

The examiner fills in the correct answers on an ordinary answer sheet and marks the student number 0000000. Such a sheet is called the key sheet. Together with the stack, the examiner supplies N, the number of questions, and the names of the files to write.

The two files. Questions and student-number columns are numbered from 1, as printed on the form. An answer is a letter, Blank when no bubble was marked, or Multiple when several were.

  • The results file has one row per student sheet in stack order: position in the stack, student number, the answer to every question, points, score (%), grade, blank rows, multiple rows, and reason. A sheet that is not graded is flagged: its row holds only the position and the reason.
  • The key file has the correct letter for every question.

How to answer. Answer each question with modular pseudocode or flowcharts. Give every procedure inputs, outputs, and one responsibility, and let a main procedure call the others. Each question lists operations that already exist and that you may use without designing them. Naming the procedures, including those operations, is part of the task: a good name says what the procedure does.

Process the scanned answer sheets

Design the stage Process sheets. You may use the operations in Table 1.

Table 1: Operations that may be used without designing them.
Operation Needs Gives
Scan a sheet a sheet an image of the sheet
Read the printed form code an image the form code, or Unreadable
Recognize the marked bubbles in one region an image, a form code, a region the collection of marked labels; for example {C}, {A, C}, or the empty collection {}
Detect how the image is oriented an image Upright, Upside down, or Unknown
Rotate an image an image, an angle the rotated image
Write rows to a file a file name, a list of rows nothing; one line per row

A region is one group of bubbles: student-number column c or question row q. Recognizing marks needs the form code because the bubbles are in different places on the two forms. A bubble that the student filled and then crossed out, as the form’s adjustment instructions allow, is not reported.

Scoring. A question scores 1 point when the answer equals the key. Scores are always reported as percentages:

\[ \text{score} = 100\% \times \frac{\text{points}}{N}, \qquad \text{grade} = 1 + 9 \times \frac{\text{score}}{100\%}. \]

Compute the grade from the unrounded score, then round the score and the grade to one decimal place, with halves rounded up.

Levels. Table 2 splits the task into levels. Levels 1 to 4 build on each other. Levels 5 to 7 are independent extensions of level 4 and can be answered in any order. From level 2 onward, show only the procedures you add or change and label them with their level.

Table 2: Levels with their weight in the score of this question. Levels 1 to 4 build on each other; levels 5 to 7 extend level 4 independently.
Level Situation Your design must Weight
1. One sheet One scanned student sheet on form 38431; every group of bubbles has exactly one mark. Read the student number and the N answers. 15%
2. Score one sheet The key is given as a list of N letters. Score the answers and compute the score and the grade. 15%
3. A stack of sheets A stack with one key sheet, which may be anywhere in the stack. Find the key sheet, grade every other sheet, and write the results file and the key file. 20%
4. Which form is used The exam may use either form. The form of the key sheet is the exam’s form. Read the form code before reading any mark, and use it for every mark. Flag a sheet on a different form. 10%
5. Imperfect marks A question row may be blank or have several marks. Score such a row 0 points and count the blank and multiple-mark rows per student. 5%
6. Missing or inconsistent data A student-number column is blank or has several marks, a student number was already graded, or the stack has no key sheet or more than one. Flag a sheet with the reason instead of grading it; for a duplicate, grade the first sheet and flag the later one. Without exactly one key sheet, write no files and output the reason. 15%
7. Scanning problems A sheet is upside down or in an unknown orientation, or its form code is unreadable. Rotate an upside-down image before reading it; flag the other cases. 5%

Naming (5%) and test cases (10%). Name your procedures. Create test cases for your design: for every level you answered, give an input and the output you expect.

Work backwards from the results file. List every value you need about one sheet, and write one small procedure that obtains each value. A procedure that reads one group of bubbles can be reused for a digit and for an answer.