Shutdown, retries and lost messages

Some bugs show up only in production operations. I fixed five of them in workflow, messaging, HTTP and Kubernetes projects, each through a full maintainer review.

5Pull requests merged in Dagu
3Pull requests merged in Kamaji
+840 / -27Lines in the Atmos fix
Year
2026
Sector
Infrastructure
Client
Dagu, Hookdeck Outpost, urllib3, Kamaji, Atmos
Duration
Open source, 2026
Stack
Go, Python, Kubernetes
Role
Contributor
Updated

The problem

A SIGTERM from systemd or Docker killed running Dagu workflow runs with no handlers and no final status.

In Hookdeck Outpost, a failed delivery whose attempt log failed to publish was acknowledged, so the attempt record and its retry were lost. urllib3 followed redirects without honoring Retry-After. In Kamaji, a deleted tenant control plane could stay in Terminating.

What I built

Dagu in Go. Installed signal handling in the worker, drained running tasks before cancelling anything, forwarded SIGTERM to each agent once, then added a configurable shutdown timeout.

Hookdeck Outpost in Go. Traced the error wrapping that made the queue acknowledge, returned the message when the attempt log was not published, and added the test the reviewer asked for.

urllib3 in Python. Waited for Retry-After before the redirected request, with tests for the wait, the cap and a past date.

Kamaji in Go. Returned the finalizer error so the controller requeues with backoff, and guarded a nil spec.

Atmos in Go. Made one directive remove a whole component section across 16 section types, and extended the tests when the maintainer asked for 85 percent coverage.

How the work ran

Each change went through the project's own review. The Outpost fix merged after one review round.

Result

5 pull requests merged in Dagu, including graceful shutdown at 221 lines added and a bounded drain at 332 lines added.

The Outpost fix merged at 234 lines added and 11 removed.

3 pull requests merged in Kamaji.

The Atmos fix merged at 840 lines added and 27 removed.

“The drain design is the right one, and the writeup made it easy to verify”

Dagu maintainer, in the review of PR 3035, on GitHub

These projects are public, so they are named. The quote stops where the maintainer's sentence turns into a list.

[+]More case studies

More case studies

All case studies

Next step

If you have a problem that needs careful engineering, send me a written brief.

Start a brief

I reply in writing within one working day.

How it starts

  1. [1]Send a written brief
  2. [2]Sign the agreement
  3. [3]Week 1 starts