Deployment

Standalone Worker

For traditional deployments, run one or more standalone worker processes. This is the most robust and scalable option.

# Run with default settings
python manage.py runworker

# Run with specific concurrency and queues
python manage.py runworker --concurrency 100 --queue high_priority --queue default

# Run with scheduler enabled
python manage.py runworker --scheduler

# Run a worker for a batch queue
python manage.py runworker --queue=batch_queue

Scaling

It's safe to run multiple standalone worker instances. Both backends use atomic operations to prevent multiple workers from picking up the same task:

• Postgres: SELECT ... FOR UPDATE SKIP LOCKED
• Valkey: BLMOVE

Each worker has an ID of {hostname}-{pid} — deterministic rather than random, precisely so a restarted worker can recognise what its previous incarnation left behind. If a worker process is terminated uncleanly, its in-flight tasks are abandoned.

On startup a worker reclaims abandoned tasks from dead siblings on the same host (same hostname, no fresh heartbeat) and, on the Valkey backend, from its own previous incarnation at the same ID. A periodic sweep repeats the sibling half only: it deliberately leaves the worker's own in-flight tasks alone, because while the worker is running those are executing normally rather than abandoned.

Note

The database backend does not currently reclaim its own previous incarnation's tasks — only dead siblings. Recovering after a restart at the same {hostname}-{pid} needs per-task ownership rather than inference from the worker ID, and is planned as part of a broader failure/retry rework.

One worker per ID

The ID identifies a slot, not a run: at most one live worker holds a given ID, so any in-flight record found under your own ID belongs to a previous incarnation and is safe to take over. {hostname}-{pid} satisfies this automatically — the OS will not hand the same PID to two live processes on one host.

If you override it with VTASKS_WORKER_ID / --id, keeping it unique per live worker becomes your responsibility; give each replica its own value (an ordinal from a StatefulSet, or the pod name) rather than one shared string. Two live workers sharing an ID breaks the assumption above, and each will treat the other's running tasks as abandoned.

Health Checks (Kubernetes)

The worker can report its status by updating a file's modification time every 5 seconds. This is useful for Liveness Probes.

python manage.py runworker --health-check-file /tmp/worker_health

Kubernetes Liveness Probe:

livenessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - 'test -f /tmp/worker_health && [ $(($(date +%s) - $(stat -c %Y /tmp/worker_health))) -lt 15 ]'
  initialDelaySeconds: 10
  periodSeconds: 10

Embedded Worker (All-in-One)

For simpler deployments, run the worker inside your ASGI web server's event loop. This reduces the number of processes you need to manage.

1. Create an Embedded ASGI Entrypoint

# myproject/asgi_embedded.py
import os
from django.core.asgi import get_asgi_application
from django_vtasks.asgi import get_worker_application

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")

# Get the standard Django ASGI application
django_asgi_app = get_asgi_application()

# Wrap it with the worker application
application = get_worker_application(django_asgi_app)

2. Run with an ASGI Server

Use any ASGI-compliant server that supports the lifespan protocol:

# Example with Granian
granian --interface asgi myproject.asgi_embedded:application --host 0.0.0.0 --port 8000

Scaling Embedded Workers

You can run multiple instances of the embedded configuration. Each web server process has its own worker, and they coordinate through the shared backend (Postgres or Valkey).

Shutdown Behavior

The ASGI server owns process signals, so the embedded worker stops in two ways:

1. ASGI lifespan shutdown — the normal path: the server tells the app it is shutting down, and the worker drains its active tasks before the process exits.
2. Stop signal (SIGTERM/SIGINT) — a chained handler stops the worker and scheduler as soon as the process is told to stop, without replacing the server's own signal handler.

The second path matters on servers that deliver lifespan shutdown only after every in-flight HTTP request has drained (granian). There, a request that never completes — a client that stops sending mid-body is enough — would otherwise leave a "stopping" process consuming queue work indefinitely, competing with its own replacement on the shared queue. With the chained handler, the worker stops consuming within milliseconds of the signal regardless of what the HTTP side is doing. It is installed automatically; no configuration is needed.

For granian specifically, two settings are worth setting alongside this:

• GRANIAN_WORKERS_KILL_TIMEOUT (granian, disabled by default): without it, granian waits forever for a worker that cannot finish draining, and never escalates to SIGKILL. Set it (e.g. 60) so a wedged worker is force-killed instead of holding its memory next to its replacement until the OOM killer intervenes. Tasks lost to the SIGKILL are picked up by task rescue on the next worker start — re-queued if VTASKS_MAX_RESCUES is above 0, but sent to the dead-letter queue rather than retried at the default of 0.
• VTASKS_SHUTDOWN_CANCEL_GRACE (vtasks, disabled by default): opt-in refinement of the same situation. After the stop signal and worker drain, waits this many seconds and then cancels any asyncio tasks still pending, so the server's request drain can complete and the process exits cleanly rather than waiting to be SIGKILLed. This aborts any HTTP request still running when the grace expires — set it above your slowest legitimate request.

Metrics

Standalone Worker

Enable the metrics server:

python manage.py runworker --metrics-port 9100

Metrics are available at http://localhost:9100/.

Embedded Worker

When running in embedded mode, the worker shares the same process as your web server. All metrics are exposed via your application's standard metrics endpoint (e.g., /metrics provided by django-prometheus).

Do not use --metrics-port in embedded mode.

Memory Optimization

Reduce worker memory by removing unneeded INSTALLED_APPS:

# settings.py
if os.environ.get("VTASKS_IS_WORKER") == "true":
    INSTALLED_APPS = prune_installed_apps(INSTALLED_APPS)
    ROOT_URLCONF = "django_vtasks.empty_urls"  # Omit if tasks require "reverse"

Set VTASKS_IS_WORKER=true in your worker's environment variables.

Reliability

Valkey Reliable Queue Pattern

When using Valkey, django-vtasks implements the Reliable Queue Pattern:

1. Worker waits for a task
2. Task is atomically moved from q:default to processing:<worker_id> via BLMOVE
3. Task is processed and acknowledged (removed from processing list)

If a worker crashes (OOM kill, power failure), the task remains in its processing: list. On the next startup, the same worker (or a new one with the same ID) reclaims it. What happens next depends on VTASKS_MAX_RESCUES: above 0 the task goes back onto the main queue, while at the default of 0 it is sent to the dead-letter queue instead — crash-recovery is therefore opt-in, and the default prefers recording the task over re-running something that may have killed the worker.

Database Backend

The Database backend uses SELECT ... FOR UPDATE SKIP LOCKED for safe concurrent processing. Failed tasks are moved to a Dead Letter Queue for inspection.

Example Docker Compose

version: '3.8'

services:
  web:
    build: .
    command: granian --interface asgi myproject.asgi:application --host 0.0.0.0 --port 8000
    ports:
      - "8000:8000"
    depends_on:
      - db
      - valkey

  worker:
    build: .
    command: python manage.py runworker --scheduler --health-check-file /tmp/health
    environment:
      - VTASKS_CONCURRENCY=50
    depends_on:
      - db
      - valkey
    healthcheck:
      test: ["CMD", "sh", "-c", "test -f /tmp/health && [ $(($(date +%s) - $(stat -c %Y /tmp/health))) -lt 15 ]"]
      interval: 10s
      timeout: 5s
      retries: 3

  db:
    image: postgres:16
    environment:
      POSTGRES_DB: myapp
      POSTGRES_USER: myapp
      POSTGRES_PASSWORD: secret

  valkey:
    image: valkey/valkey:7.2

Example Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vtasks-worker
spec:
  replicas: 3
  selector:
    matchLabels:
      app: vtasks-worker
  template:
    metadata:
      labels:
        app: vtasks-worker
    spec:
      containers:
      - name: worker
        image: myapp:latest
        command: ["python", "manage.py", "runworker", "--scheduler", "--health-check-file", "/tmp/health"]
        env:
        - name: VTASKS_CONCURRENCY
          value: "50"
        livenessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - 'test -f /tmp/health && [ $(($(date +%s) - $(stat -c %Y /tmp/health))) -lt 15 ]'
          initialDelaySeconds: 10
          periodSeconds: 10
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "512Mi"
            cpu: "500m"