Skill · Observability
Background Job Reliability Auditor
Audit scheduled jobs, queue consumers, workers, and asynchronous workflows for delivery assumptions, idempotency, retries, poison messages, concurrency, timeouts, checkpoints, shutdown, and observability. Use when a job duplicates work, silently stops, falls behind, fails only at scale, or needs review before production.
User-invocablev1.0.0
Updated Aug 4, 2026npx agentscamp add skills/background-job-reliability-auditorInstall to ~/.claude/skills/background-job-reliability-auditor/SKILL.md
Audits asynchronous work end to end: enqueue, delivery, deduplication, processing, external side effects, acknowledgement, retry, dead-letter handling, shutdown, and recovery. It ties each failure mode to observable evidence and distinguishes queue guarantees from the application's required outcome.
Trace asynchronous work from intent to verified outcome.
Workflow
- Define the required outcome. Identify the user or business effect, acceptable delay, duplication tolerance, loss tolerance, ordering need, and recovery objective.
- Map the lifecycle. Trace producer commit, enqueue, broker delivery, lease or visibility timeout, worker processing, database writes, external effects, acknowledgement, retry, dead-letter handling, and reconciliation.
- State delivery assumptions. Verify what the scheduler or broker actually guarantees under crashes, timeouts, partitions, and redelivery. Do not infer exactly-once effects from product terminology.
- Audit idempotency and transactions. Find the stable operation key, deduplication store, uniqueness boundary, transaction scope, outbox or inbox pattern, and behavior when an external side effect succeeds before local state commits.
- Review failure policy. Check exception classification, backoff, jitter, maximum attempts, retry budget, timeout hierarchy, poison-message isolation, dead-letter ownership, and replay procedure.
- Inspect concurrency and lifecycle. Verify prefetch, worker count, per-key ordering, rate limits, locks, heartbeat or lease extension, graceful shutdown, deploy draining, checkpointing, and resource cleanup.
- Assess observability. Require correlation from request or schedule to job and effect. Measure enqueue rate, completion rate, latency, oldest age, attempts, terminal failures, dead letters, throughput, and saturation.
- Exercise recovery. Define tests for crash before and after side effects, duplicate delivery, broker outage, dependency timeout, malformed payload, deploy interruption, backlog drain, and dead-letter replay.
WARNING
A retry policy without idempotency can amplify an outage into duplicate charges, emails, allocations, or data corruption.
Output
Return a lifecycle diagram in text, verified delivery assumptions, prioritized findings with code or configuration evidence, missing telemetry, failure-injection cases, and a remediation plan. Separate containment from durable fixes.
Frequently asked questions
- Does an exactly-once queue guarantee exactly-once business effects?
- Not automatically. Redelivery, producer retries, transaction boundaries, and external side effects can still duplicate outcomes. Design idempotency at the business operation boundary.
- What should alert for a background job?
- Alert on user-impacting delay or loss: oldest-message age, overdue schedule, terminal failures, dead-letter growth, or completion SLO. Queue depth alone may rise normally with traffic.
Related
- 10 Best Claude Skills for DevOps and SREUse Claude skills for incidents, rollbacks, SLOs, alerts, dashboards, logs, traces, containers, CI, and safer releases.
- Alerting Rules TunerCut alert noise and make every page mean something — rewrite alerting rules to fire on user-felt symptoms (error rate, latency SLO burn, failed requests) instead of causes (high CPU, full disk), with duration windows and severity routing so only urgent, actionable conditions reach a human. Use when on-call is fatigued by low-value pages, when real incidents get missed in the noise, or when alerts fire on causes rather than impact.
- Dashboard DesignerDesign a service dashboard that answers one question at a glance — is the service healthy, and if not, where's the problem? — by structuring panels around RED/USE instead of dumping every metric. Use when a service has no dashboard, when the existing one is an unreadable metric wall, or during incident-readiness prep.
- Safe Data Backfill PlannerPlan and implement a resumable production data backfill with bounded batches, checkpoints, rate limits, correctness verification, observability, and a pause or rollback path. Use when populating a new column, repairing historical rows, re-keying records, migrating derived data, or updating millions of records without overwhelming the database or replicas.
- Structured Logging DesignerDesign a structured (JSON) logging strategy with a stable field schema, correlation-ID propagation, and a disciplined level policy — then migrate ad-hoc string logs toward it. Use when logs are unsearchable plain text, when debugging a request across services means grepping multiple log streams by hand, or when standing up logging for a new service.
- Idempotency DesignerMake unsafe, retryable API operations idempotent so a client retry or a network hiccup can't double-charge, double-create, or double-send — design a client-supplied idempotency key, an atomic store-and-check (unique constraint or conditional write), in-flight conflict handling, and a retention policy. Use when a POST/mutation can be retried (payments, order creation, sends, webhooks), or when duplicate side effects have already shown up in production.