The delay and absence webhooks reject a request without an Idempotency-Key header (422).

A known key returns the existing incident with 200 instead of 201, so a sender that retries never opens a second incident or runs the playbook twice, for example calling the contractor agency twice for a single absence. When two requests race with the same new key, a unique constraint lets a single insert through; the losing request rolls back and returns the winning incident. Purchase-order amendments use a version compare-and-swap instead of a lock: the UPDATE bumps the version and only matches while the version is unchanged since the read, so a concurrent edit produces a conflict error rather than an overwrite.

A customer-facing action, here the customer email, is created as needs_approval next to a pending row in the approvals table, under a setting that is on by default.

The gate follows the action type, not a model's sense of risk, and neither agent has a tool that approves. A decision is a single UPDATE that only matches while the approval is still pending, so a double click or a racing request gets 409 instead of a second email. Approving moves the action to pending and dispatches the action; rejecting marks the action skipped. Asking either agent to email the customer produces a gated approval instead of an email.

By default, actions run inline, inside the API request that created or approved the actions.

With queued mode switched on in configuration, the actions are marked queued instead, and a background worker claims the oldest queued action with SELECT … FOR UPDATE SKIP LOCKED, runs that incident's queued actions, and requeues failed actions once the backoff window has passed. A queue-status endpoint reports queue depth, retriable and dead-lettered counts, and worker health. A written evaluation in the repo compares the queue with a durable workflow engine such as Temporal and recommends keeping the queue until defined adoption gates are met; selecting the durable mode of the workflow-engine interface raises an error on purpose.

The chat panel sends the operator's message, the conversation so far and a snapshot of the incident to Gemini 2.5 Flash with three function tools: get_incident_status, list_active_shipments and execute_command.

The backend runs each tool call and returns the result to the model, for up to five rounds per message. The execute_command tool accepts only a fixed list of commands: internal steps run immediately through the same executor as the playbook and report success or the connector's error, while a request to email the customer becomes an approval card instead of an email. Status answers are grounded in the action rows, because get_incident_status returns each step's status and last error.

A playbook is a Python list, so a delay always becomes four action runs (Slack alert, supplier call, PO amendment, customer email) and an absence three (labor-system update, contractor call, site-manager alert in Slack).

A deterministic executor walks the runs in sequence order, sends each pending action to the matching connector, saves the request and response payloads on the row, and writes every action in a separate database transaction, so a later failure cannot roll back an earlier PO change. An incident resolves once every run is completed or skipped, and escalates when any run is dead-lettered. No model decides what a playbook does next.

Slack, Twilio and Resend calls each run behind a dedicated timeout (defaults: 10 s, 15 s and 10 s) and a circuit breaker that opens after five straight failures, lets calls through again after 60 seconds and closes after two successes (defaults).

A failed action retries with exponential backoff and full jitter (200 ms base, 10 s cap) and is dead-lettered after the third failed attempt, with a fallback note stored on the row. Timeouts get stricter treatment: a timed-out Slack post, Twilio call or manager alert may still have gone through, and no idempotency key reaches Slack or Twilio, so the action goes straight to the dead-letter list for a person to check. Email stays retriable because every attempt sends Resend the same idempotency key. Dead-lettered actions can be listed and requeued through the API.

Inbound calls, and outbound calls started through the voice API, get TwiML that opens a Twilio Media Stream to a WebSocket on the backend.

A Pipecat pipeline converts the 8 kHz telephone audio and streams the audio to and from Gemini Live, with Gemini's voice-activity detection tuned (high start and end sensitivity, 260 ms of silence to end a turn; defaults). When the caller talks over the agent, a custom Pipecat processor sends Twilio a clear event so buffered agent audio stops playing. The agent marks four call objectives (cause, new ETA, mitigation, risk) through a tool and closes the call after a brief closing statement once all four are confirmed; the transcript is saved and summarized into Slack. The playbook's supplier and contractor calls are different: one-way Twilio calls that read a fixed message.

Groundwork for running several backend servers, built but never deployed to more than a single server.

Each active call records the owning server's ID and a heartbeat, and checkpoints state to Redis every five seconds and on each progress or end-call tool call: objective flags, close state and the transcript tail, with an expiry. A control command (only end_call is supported) runs locally when the call lives on the receiving server; otherwise the command goes onto a per-call Redis list that the owning server polls every second. Routing fails closed: no Redis bus returns 503, no checkpoint returns 404, and a checkpoint older than 20 seconds returns 409 (timings are defaults). Kubernetes and NGINX sticky-routing templates and a multi-pod smoke-test runbook are in the repo; the smoke test has not been run.