benchmark: Add README, missing scripts

benchmark/README.md

+# Benchmarking our project's code on the Gemini server
+
+This directory contains several scripts needed to submit our project's code to
+the job system on the Gemini server, which we used to benchmark our project in
+a reproducible environment.
+
+The scripts in this directory are intended to be executed via the SGE job
+system on this server and do not require administrative rights for any step.
+
+This requires a slightly unusual workaround to allow the Nix package manager to
+work without having administrative rights on the system, this is achieved using
+`proot`, a wrapper tool that "fakes" the location of the `/nix` directory for
+processes it starts.
+
+To use these scripts they will first need to be copied to the Gemini server, as
+they need to be processed with a command on this server.
+
+## Setting up the Nix environment
+
+To run our project's code it is required to set up an environment with the Nix
+package manager. This can be done using two scripts:
+`bootstrap-nix-environment.sh` and `bootstrap-project-environment.sh`.
+
+The Nix environment can be bootstrapped on the server by running the `bootstrap-nix-environment.sh`:
+
+```console
+$ qsub -M "YOUR_EMAIL@students.uu.nl" bootstrap-nix-environment.sh
+```
+
+This will result in a Nix environment in the server's local scratch space under `/scratch`.
+
+The job system will report the script's output to the directory this script was run from in two
+files `bootstrap-nix-environment.o_____` and `bootstrap-nix-environment.e_____`. Inspect these files
+if something goes wrong. You will also be alerted via email about when this script is started and
+when it completes.
+
+To download our project's code from this repository the second script
+`bootstrap-project-environment.sh` can be executed. This script will install all of the project's
+dependencies except for the Gurobi solver, as the license terms for this solver prohibit this.
+
+Execute this script once the previous script is done using:
+
+```console
+$ qsub -M "YOUR_EMAIL@students.uu.nl" bootstrap-project-environment.sh
+```
+
+The job system will again create files in the current directory with the script's output and will
+send emails to inform you of the script's start and completion.
+
+## Executing our project's code
+At this point our project can be used to run benchmarks using the code for the online variant of the
+problem, for the combinatorial algorithm of the offline variant of the problem and for the ILP-based
+algorithm using the free CBC solver.
+
+### Running a benchmark for the online variant
+To execute our program on one of the provided benchmark cases, run the following command:
+
+```console
+$ qsub -M "YOUR_EMAIL@students.uu.nl" run-online-benchmark.sh NUMBER MODE
+```
+
+Here `NUMBER` should be the number of the benchmark to execute (between 1 and 18) and `mode` should be one of
+
+ - `first-fit`, to use the first-fit strategy to seat people,
+ - `last-fit`, to use the last-fit strategy to seat people,
+ - `greedy`, to use the greedy/best-fit strategy to seat people.
+
+To run all benchmarks sequentially (i.e. at most one at a time), run the following command:
+
+```console
+$ qsub -tc 1 -t 1-18 -M "YOUR_EMAIL@students.uu.nl" run-online-benchmark.sh \$SGE_TASK_ID MODE
+```
+
+Here `\$SGE_TASK_ID` will be filled in by the job system with the number of the benchmark to execute.
+
+It is also possible to override the default runtime limit of five minutes by passing the `-l`
+parameter to `qsub`, e.g. `-l 0:15:00` will limit the runtime of the submitted jobs to fifteen
+minutes.
+
+### Running a benchmark for the combinatorial algorithm for the offline variant
+To execute our program on one of the provided benchmark cases, run the following command:
+
+```console
+$ qsub -M "YOUR_EMAIL@students.uu.nl" run-edge-benchmark.sh NUMBER
+```
+
+Here `NUMBER` should be the number of the benchmark to execute (between 1 and 21).
+
+To run all benchmarks sequentially (i.e. at most one at a time), run the following command:
+
+```console
+$ qsub -tc 1 -t 1-21 -M "YOUR_EMAIL@students.uu.nl" run-edge-benchmark.sh \$SGE_TASK_ID
+```
+
+Here `\$SGE_TASK_ID` will be filled in by the job system with the number of the benchmark to execute.
+
+It is also possible to override the default runtime limit of ten minutes by passing the `-l`
+parameter to `qsub`, e.g. `-l 0:20:00` will limit the runtime of the submitted jobs to twenty
+minutes.
+
+### Running a benchmark for the ILP-based algorithm for the offline variant using CBC
+To execute our program on one of the provided benchmark cases, run the following command:
+
+```console
+$ qsub -M "YOUR_EMAIL@students.uu.nl" run-cbc-benchmark.sh NUMBER
+```
+
+Here `NUMBER` should be the number of the benchmark to execute (between 1 and 21).
+
+To run all benchmarks sequentially (i.e. at most one at a time), run the following command:
+
+```console
+$ qsub -tc 1 -t 1-21 -M "YOUR_EMAIL@students.uu.nl" run-cbc-benchmark.sh \$SGE_TASK_ID
+```
+
+Here `\$SGE_TASK_ID` will be filled in by the job system with the number of the benchmark to execute.
+
+The job system will create files in the current directory containing the algorithm's output.
+
+## Running a benchmark for the ILP-based algorithm for the offline variant using Gurobi
+To run our benchmark using the Gurobi solver it first needs to be made available to our project's code.
+We are unfortunately unable to provide a script for this as Gurobi is not free software and its
+license terms do not allow us to provide an automated installer.
+
+To make Gurobi available to our project's code:
+
+1. Follow the instructions in [Gurobi's installation guide][gurobi-guide] to create an account and
+   obtain an academic license.
+2. Open an SSH connection to the Gemini server and execute the following commands:
+    ```console
+    $ cd /scratch/$USER
+    $ wget https://packages.gurobi.com/9.0/gurobi9.0.3_linux64.tar.gz
+    $ tar xf gurobi9.0.3_linux64.tar.gz
+    $ ./proot -b /scratch/${USER}/nix-store:/nix -0
+    [ A new shell will open under the root user. Execute these commands in this shell: ]
+    # export GRB_LICENSE_FILE=./gurobi.lic
+    # ./gurobi903/linux64/bin/grbgetkey
+    ```
+
+   Follow the instructions to activate your Gurobi license.
+   When asked whether it is intended to activate the license as `root` "Yes" must be chosen.
+
+[gurobi-guide]: https://www.gurobi.com/documentation/9.0/quickstart_linux/quickstart_linux.html
+
+Once this has been done the terminal can be closed by executing `exit` twice.
+
+After this a benchmark using the Gurobi solver can be executed with this command:
+
+```console
+$ qsub -M "YOUR_EMAIL@students.uu.nl" run-gurobi-benchmark.sh NUMBER
+```
+
+Here `NUMBER` must again be between 1 and 21.
+
+To run all benchmarks sequentially (i.e. at most one at a time), run the following command:
+
+```console
+$ qsub -tc 1 -t 1-21 -M "YOUR_EMAIL@students.uu.nl" run-gurobi-benchmark.sh \$SGE_TASK_ID
+```
+
+## Running arbitrary commands in the Nix environment
+For debugging or running additional examples a helper script is included to run an arbitrary command
+in the Nix environment.
+
+```console
+$ qsub -M "YOUR_EMAIL@students.uu.nl" run-command.sh COMMAND
+```
+
+Here `COMMAND` will be interpreted as a Bash command.
+
+The Astor Movie Theatre benchmark can then be run as follows:
+
+```console
+$ qsub -M "YOUR_EMAIL@students.uu.nl" run-command.sh cli offline instances/astor-movie-theatre
+```

benchmark/bootstrap-project-environment.sh

@@ -2,9 +2,6 @@
 
 # This script will use the Nix environment bootstrapped by bootstrap-nix-environment.sh to clone the
 # Cinematinator project to the scratch directory and set up all required dependencies.
-# It requires a SSH key with access to the repository to be present in the user's home directory.
-# SSH keys can be granted access here:
-# https://git.science.uu.nl/mads-2020-cinematinator/python-prototype/-/settings/repository#js-deploy-keys-settings
 
 # This script can be submitted via SGE with: qsub -M "YOUR_EMAIL@students.uu.nl" bootstrap-project-environment.sh
 
@@ -13,19 +10,23 @@
 #$ -N bootstrap-project-environment
 #$ -m bes
 
+# Enable "Bash strict mode": exit on error, undefined variables, also log executed commands
 set -euxo pipefail
 
+# Set up environment
 scratch_dir=/scratch/${USER}
 export PROOT_NO_SECCOMP=1
 
+# Enter Nix environment
 $scratch_dir/proot -b ${scratch_dir}/nix-store:/nix bash <<EOF
   set -euxo pipefail
   . ~/.nix-profile/etc/profile.d/nix.sh
 
+  # Navigate to temporary directory and download the project's code via Git
   cd ${scratch_dir}
-  git clone git@git.science.uu.nl:mads-2020-cinematinator/python-prototype.git --depth 1 --branch benchmark
+  git clone https://git.science.uu.nl/mads-2020-cinematinator/python-prototype.git --depth 1
 
+  # Open the project and download all dependencies, then build the project's code.
   cd python-prototype
-
   nix-build -A package
 EOF

benchmark/run-cbc-benchmark.sh

+#!/bin/bash
+
+# This script will run a offline benchmark testcase using CBC and will log the output and memory/CPU statistics.
+# Use with `qsub -M 'YOUR_EMAIL@students.uu.nl' run-offline-benchmark.sh BENCHMARK_NUMBER MODE`.
+
+#$ -S /bin/bash
+#$ -cwd
+#$ -N run-cbc-benchmark
+#$ -m bes
+
+# Limit runtime to ten minutes
+#$ -l h_rt=0:10:00
+
+set -euxo pipefail
+
+scratch_dir=/scratch/${USER}
+export PROOT_NO_SECCOMP=1
+
+benchmark_number=$1
+
+$scratch_dir/proot -b ${scratch_dir}/nix-store:/nix bash <<EOF
+  set -euxo pipefail
+  . ~/.nix-profile/etc/profile.d/nix.sh
+
+  cd ${scratch_dir}/python-prototype
+  nix run -f default.nix package -c cli offline ./testcases/Exact${benchmark_number}.txt
+EOF

benchmark/run-command.sh

+#!/bin/bash
+
+# This script will run an arbitrary command in the Nix environment, see README.
+# Use with `qsub -M 'YOUR_EMAIL@students.uu.nl' run-offline-benchmark.sh COMMAND`.
+
+#$ -S /bin/bash
+#$ -cwd
+#$ -N run-command
+#$ -q all
+#$ -m bes
+
+# Limit runtime to ten minutes
+#$ -l h_rt=0:10:00
+
+set -euxo pipefail
+
+scratch_dir=/scratch/${USER}
+export PROOT_NO_SECCOMP=1
+
+$scratch_dir/proot -b ${scratch_dir}/nix-store:/nix bash <<EOF
+  set -euxo pipefail
+  . ~/.nix-profile/etc/profile.d/nix.sh
+
+  cd ${scratch_dir}/python-prototype
+  nix run -f default.nix package -c $@
+EOF

benchmark/run-edge-benchmark.sh

+#!/bin/bash
+
+# This script will run a offline benchmark testcase using the combinatorial algorithm and will log
+# the output and memory/CPU statistics.
+# Use with `qsub -M 'YOUR_EMAIL@students.uu.nl' run-offline-benchmark.sh BENCHMARK_NUMBER`.
+
+#$ -S /bin/bash
+#$ -cwd
+#$ -N run-edge-benchmark
+#$ -m bes
+
+# Limit runtime to ten minutes
+#$ -l h_rt=0:10:00
+
+set -euxo pipefail
+
+scratch_dir=/scratch/${USER}
+export PROOT_NO_SECCOMP=1
+
+benchmark_number=$1
+
+$scratch_dir/proot -b ${scratch_dir}/nix-store:/nix bash <<EOF
+  set -euxo pipefail
+  . ~/.nix-profile/etc/profile.d/nix.sh
+
+  cd ${scratch_dir}/python-prototype
+  nix run -f default.nix package -c cli edge-algo ./testcases/Exact${benchmark_number}.txt
+EOF

benchmark/run-gurobi-benchmark.sh

+#!/bin/bash
+
+# This script will run a offline benchmark testcase using Gurobi and will log the output and memory/CPU statistics.
+# Use with `qsub -M 'YOUR_EMAIL@students.uu.nl' run-offline-benchmark.sh BENCHMARK_NUMBER MODE`.
+
+#$ -S /bin/bash
+#$ -cwd
+#$ -N run-gurobi-benchmark
+#$ -q all
+#$ -m bes
+
+# Limit runtime to ten minutes
+#$ -l h_rt=0:10:00
+
+set -euxo pipefail
+
+scratch_dir=/scratch/${USER}
+export PROOT_NO_SECCOMP=1
+
+benchmark_number=$1
+
+# Additional workaround: Gurobi does not like the system used on the Gemini server to translate
+# Solis users to Linux users, so we pretend to be root here as the license check otherwise fails.
+# This is achieved using the `-0` flag.
+$scratch_dir/proot -b ${scratch_dir}/nix-store:/nix -0 bash <<EOF
+  set -euxo pipefail
+  . ~/.nix-profile/etc/profile.d/nix.sh
+
+  cd ${scratch_dir}/python-prototype
+  export GUROBI_HOME=${scratch_dir}/gurobi903/linux64
+  export GRB_LICENSE_FILE=${scratch_dir}/gurobi.lic
+  export PATH=\${PATH}:\${GUROBI_HOME}/bin
+  export SOLVER_NAME=GUROBI
+  nix run -f default.nix package -c cli offline ./testcases/Exact${benchmark_number}.txt
+EOF