stacscheck: the St Andrews Computer Science CHECKer

software
stacscheck

Stacscheck can be found Stacscheck is designed to be a lightweight automated tester It was designed for automatically testing University student’s practicals.

Stacscheck is purposefully designed to be as simple as possible (although this still turns out to be less simple than I hoped!) It is designed only for programs which run in the terminal (no graphical user interfaces!). It will build a student’s code, and then run it with various inputs, checking each output against a “correct” output.

How the student’s code is built, and run, is given as a series of shell scripts, and the inputs and outputs are text files. The advantage of this is you can, in principle, test programs in any language. The disadvantages are you need to be happy running shell scripts, and output needs to match exactly – stacscheck expects an exact match, except for “invisible differences” (for example, stacscheck will ignore whitespace at the end of lines, or a tab vs 8 spaces).

Stacscheck was designed with four founding principles – although I am not claiming they were all met!

Philosophy

  • Transparency: It should be as clear how the tester is running a particular test.
  • Extensibility: The tester should be easy to extend with new types of tests.
  • Simplicity: Writing tests (particularly of common existing types) should be as simple as possible.
  • Fallback: The tester should be as light as possible – the more complex a system, and the more pieces it has, the easier it is for it to break.

Several of these requirements are in conflict – most particularly Extensibility and Simplicity.

After considering various options, we decided to base Stacscheck on shell scripts. While shell scripts have many annoying features, they are a general and powerful way of running any program on Unix.

The use of shell scripts does mean that Stacscheck does not run on Windows (unless you install a Linux later, like WSL). However achieving any useful Extensibility and Simplicity while supporting Unix and Windows would be extremely difficult (maybe impossible!)