Shutdown, retries and lost messages
Some bugs show up only in production operations. I fixed five of them in workflow, messaging, HTTP and Kubernetes projects, each through a full maintainer review.

The problem
A SIGTERM from systemd or Docker killed running Dagu workflow runs with no handlers and no final status.
In Hookdeck Outpost, a failed delivery whose attempt log failed to publish was acknowledged, so the attempt record and its retry were lost. urllib3 followed redirects without honoring Retry-After. In Kamaji, a deleted tenant control plane could stay in Terminating.
What I built
Dagu in Go. Installed signal handling in the worker, drained running tasks before cancelling anything, forwarded SIGTERM to each agent once, then added a configurable shutdown timeout.
Hookdeck Outpost in Go. Traced the error wrapping that made the queue acknowledge, returned the message when the attempt log was not published, and added the test the reviewer asked for.
urllib3 in Python. Waited for Retry-After before the redirected request, with tests for the wait, the cap and a past date.
Kamaji in Go. Returned the finalizer error so the controller requeues with backoff, and guarded a nil spec.
Atmos in Go. Made one directive remove a whole component section across 16 section types, and extended the tests when the maintainer asked for 85 percent coverage.
How the work ran
Each change went through the project's own review. The Outpost fix merged after one review round.
Result
5 pull requests merged in Dagu, including graceful shutdown at 221 lines added and a bounded drain at 332 lines added.
The Outpost fix merged at 234 lines added and 11 removed.
3 pull requests merged in Kamaji.
The Atmos fix merged at 840 lines added and 27 removed.
“The drain design is the right one, and the writeup made it easy to verify”
Evidence
- github.com/dagucloud/dagu/pull/3035Dagu PR 3035, graceful shutdown
- github.com/dagucloud/dagu/pull/3062Dagu PR 3062, shutdown timeout
- github.com/hookdeck/outpost/pull/1134Hookdeck Outpost PR 1134
- github.com/urllib3/urllib3/pull/5330urllib3 PR 5330
- github.com/clastix/kamaji/pull/1337Kamaji PR 1337
- github.com/cloudposse/atmos/pull/3289Atmos PR 3289
These projects are public, so they are named. The quote stops where the maintainer's sentence turns into a list.


