Where did my task go? Monitoring Nextflow on Slurm without Tower

2026-08-20T00:00:00+00:00

If you run Nextflow pipelines on a university HPC cluster, your monitoring stack probably looks like mine did: one terminal tailing .nextflow.log</code>, another running squeue</code> every thirty seconds, and a growing suspicion that somewhere between "submitted" and "completed" your tasks are spending a lot of time doing nothing at all.

The frustrating part is that the information exists. Nextflow knows when it submitted each task and when Slurm actually started it. It knows how much memory you requested and how much the process actually used. It just doesn't show you any of this while the run is going — and by the time you're staring at the trace file after a failed 14-hour run, aggregating submit-to-start deltas per process with awk</code> is nobody's idea of observability.

The options, as of 2026</h2> The obvious answer is Seqera Platform (formerly Tower), and if your lab runs in the cloud with a budget, it's a good one. On a shared on-premise cluster the picture is different: Seqera Cloud's free tier is sized for trying things out — a handful of users and concurrent runs, limited run history. Fine for one person; not for a core facility with dozens of users.</li> Self-hosting Seqera Platform is an Enterprise license conversation.</li> The open-source nf-tower was archived in January 2025 and is unmaintained.</li> -with-weblog</code> gives you a firehose of JSON events and a "build the rest yourself" exercise.</li> </ul> Meanwhile, every HPC cluster I've touched already runs a Prometheus and Grafana stack somewhere — the sysadmins use it for node health. What's missing is the bridge between Nextflow's head job and that stack. The awkward thing about head jobs</h2> Here's what makes the bridge less trivial than "expose an HTTP endpoint": on a well-run cluster your Nextflow head job isn't on the login node — it's a batch job itself, running on a compute node behind a firewall, with an IP your Prometheus server has never heard of and no inbound connectivity at all. Classic pull-based scraping simply doesn't reach it. On air-gapped clusters, nothing does. But there is one thing every HPC node can always do: write a file to the shared filesystem. And it turns out Prometheus grew an idiom for exactly this situation years ago — the node_exporter textfile collector, built for batch jobs and cron scripts. Write your metrics to a .prom</code> file atomically, point a node_exporter at the directory, done. No network, no credentials, no agent on the cluster. That observation became nf-prometheus</a>: a small (zero-dependency, Apache 2.0) Nextflow plugin that listens to workflow events and exports them in Prometheus format, textfile-first. A Pushgateway mode covers live updates during the run, and a classic HTTP /metrics</code> endpoint exists for the cases where the head node is actually reachable. The metric I built it for is per-process scheduler queue wait — the submit-to-start time your squeue</code> never aggregates for you. On the dashboard it sits next to requested-vs-used CPUs and peak RSS, which together answer the two questions every cluster admin eventually asks a pipeline user: why are you queued so long and why did you request 32 GB for a process that peaks at 4. Setup genuinely takes about five minutes: the walkthrough is in the repo</a>. Three things the test cluster taught me</h2> I developed the plugin against a single-node Slurm cluster in a container (Rocky 9, Slurm from EPEL — the setup ships in the repo under dev/slurm/</code>). Even a toy cluster manages to reproduce the greatest hits of real HPC operations: 1. The node that drains itself. After every container restart, and occasionally after successfully finishing jobs, slurmctld would put the node in drain</code> — reason: Kill task failed</code>. With proctrack/pgid</code> (no cgroups in a container), slurmd is slow to reap task process groups and drains the node out of caution. Jobs sit in PD</code> forever and nothing in Nextflow's output tells you why. sinfo</code> first, always. (UnkillableStepTimeout</code> and an auto-resume watchdog fixed the dev environment.) 2. The metrics that were silently empty. My first dashboard had a beautiful peak-RSS panel showing nothing. Nextflow only collects per-task resource metrics if an observer explicitly asks for them by returning true</code> from enableMetrics()</code> — which the plugin template doesn't mention. One overridden method later, a task that allocated 60 MB dutifully reported a 73 MB peak. An empty dashboard panel is an integration test, if you let it be one. 3. cgroups v2 without systemd. Slurm's cgroup plugin wants to create its scope via dbus, which doesn't exist in a container. IgnoreSystemd=yes</code> in cgroup.conf</code> is the documented escape hatch, and one more entry in the "things that are obvious after you know them" file that HPC runs on. None of these are exotic. They're Tuesday on a real cluster — which is rather the point of having a dashboard that shows you where the time went instead of a log that shows you everything except that. What's next</h2> The plugin does one thing right now: run-level observability. The obvious next layer — and the reason the metrics carry the labels they do — is showback: CPU-hours per pipeline, per user, per PI, the number a core facility needs at the end of the quarter. That's where this is headed. If you run Nextflow on Slurm (or PBS, or LSF) and this scratches an itch — or conspicuously fails to — I'd genuinely like to hear about it: issues and feature requests welcome</a>, or find me on nf-core Slack.

Callisto Labs

Where did my task go? Monitoring Nextflow on Slurm without Tower