Production runs on a Kubernetes cluster (k3s) managed with GitOps: Argo CD watches a private operator repository and applies everything — chart, per-site values, secrets wiring, ingress, blue-green standbys — from git. Nothing is deployed by hand against the cluster.
This guide describes the deployment flow for developers working on
date-website. It intentionally does not include cluster addresses, access
paths, or secret locations — those live in the private operator repository
and its docs. If you are on the ops team, the operator repository is the
authoritative source; keep this file in sync with it when the flow changes.
Use this together with:
charts/date-website/ — the Helm chart this repository publishesdocs/dev/operations.md — day-to-day app operationsREADME.md — local development and Docker Compose workflowsmain is the development branch. Every push to main builds a
container image and publishes it to GHCR as <sha> and qa.qa first, then promoted to production.vX.Y.Z) are the production promotion
mechanism. A release tag promotes the already-built image to the
production aliases (prod, latest, and the vX.Y.Z/vX.Y/vX
family) — no rebuild.promote_production.yaml workflow can also promote an arbitrary
existing tag (e.g. a QA-tested SHA) to prod/latest manually.| Trigger | Image tags pushed |
|---|---|
push to main |
:qa, :<commit-sha> |
SemVer tag vX.Y.Z |
:vX.Y.Z, :vX.Y, :vX, :prod, :latest |
manual promote_production |
:prod, :latest from a chosen existing tag |
Notes:
qa, prod, latest, vX, and vX.Y are moving aliases. Production values
set image.pinInProduction: true; these aliases must be paired with
image.digest.vX.Y.Z release tags and 40-character commit SHA tags are accepted as
immutable tag forms. A digest can also be supplied to retain both the
readable tag and the exact manifest identity.main build,
then reuses that digest instead of rebuilding — tag the commit whose
main build went green.docker_build.yaml, release_tag.yaml,
promote_production.yaml.The chart lives in charts/date-website/ and is published as an OCI chart
to GHCR by helm_chart.yaml whenever chart files change on main. Bump
charts/date-website/Chart.yaml version on any template or values change;
deployments pin the immutable chart version.
The chart deploys:
core.asgiIngress, or Gateway API resourcesMigrations run as a single migration Job per release. Two modes:
post-install,post-upgrade hooks
and is deleted after success. This works for plain helm
install/upgrade, but hook timing under GitOps reconciliation is less
predictable and the Job does not strictly gate application startup.migrations.job.hook: "" and add argocd.argoproj.io/sync-wave: "0" via
migrations.job.annotations, with argocd.argoproj.io/sync-wave: "1" on
the web/celery Deployments (.Values.<component>.annotations).
Argo then runs migrations to completion before rolling the application.
The Job gets a name suffixed with the sanitized tag + a short hash of the
tag/digest/chart-version tuple (kept under 63 bytes: Kubernetes mirrors the Job name
into a pod-template label), stays completed as desired state (no TTL),
and the previous image’s Job is pruned on the next sync. (Helm’s
Release.Revision cannot suffix the name: Argo CD renders it as 1, so the
name would never change and the second sync would fail on the immutable
Job pod template.) Including the chart version also makes chart-only changes
create a fresh Job instead of trying to mutate the completed Job.When runtime database traffic uses a transaction pooler, set
migrations.job.databaseHost and migrations.job.databasePort to a direct
PostgreSQL endpoint. These values override DB_HOST/DB_PORT only in the
migration Job; web and celery continue using database.external. Leaving the
migration host empty preserves the existing behavior. Direct routing protects
schema migrations that need session-level PostgreSQL state from transaction
pooling semantics.
Also set DB_DISABLE_SERVER_SIDE_CURSORS=True for web and worker processes
that use transaction pooling. Django’s named server-side cursors are scoped to
one PostgreSQL session, while a transaction pooler may assign a different
session when the queryset fetches its next batch. Direct database connections
retain the default (False).
Never enable web.migrateOnStartup in production: it couples schema
mutation to pod readiness, re-runs on every pod restart, and races when the
replica count is above one. Keep expand-contract migration rules so old and
new pods can overlap during rollouts.
PROJECT_NAME, hosts,
settings module, media prefixes, database, and secrets.postgresql.enabled: false, database.external used).media.s3.privateCustomDomain /
media.s3.publicCustomDomain, host only, no scheme) switch media URLs to a
domain served by a Cloudflare Worker in the operator repository. The Worker
forwards requests to the S3 endpoint, rewriting the Host header to the S3
endpoint host (the hostname the URL was signed against) and preserving path
and query. Public media URLs become unsigned https://<domain>/<location>/<key>;
private media URLs stay presigned, with only the host swapped, so the SigV4
signature still validates.promote_production is run) — the image is now
available as prod/latest.image.tag) and, for moving or custom tags, the image digest
(image.digest). A release that includes database migrations carries that
pin together with its migrations./healthz/ and /readyz/ are green.Every production association’s static is collected into the image at build
time, one tree per variant under /code/static-collected/<PROJECT_NAME>.
Settings pick the tree matching the runtime PROJECT_NAME, so every site
serves build-time static and no web pod runs collectstatic at startup
(web.collectstaticOnStartup defaults to false; keep it only for images
built before this layout).
Separate trees are required because variants define the same logical paths
with different content (e.g. date/css/homepage.css differs per
association); a single merged collection would silently overwrite assets.
This keeps one generic image for all variants: no per-association images,
no startup collection, no way for an image and its values to disagree on
the variant.
The build collects each variant’s static concurrently into its own tree, and
collects django-unfold static only for the variants that run the unfold
admin skin (USE_UNFOLD=True: date incl. qa, pulterit, sf, impuls); kk and
biocum keep the stock admin. collectstatic only discovers app static dirs
for apps in INSTALLED_APPS, so a variant collected without USE_UNFOLD=True
renders the unfold admin with 500s on the missing
unfold/fonts/inter/styles.css manifest entry (2026-08-30).
The image also carries compiled translations. Both the collected trees and
the .mo files are only read by an image that serves the app from its own
filesystem: dev and PR CI bind-mount the checkout over /code, so they build
with BUILD_ASSETS=0 and skip the collection and compilation entirely.
Anything built for a deployment must leave the default (BUILD_ASSETS=1).
If static-on-S3 (see issue) lands, collection moves from the image build to a release-time upload into the existing per-association media bucket, and pods stop carrying static entirely.
With static-on-S3 the runtime {% static %} lookups
resolve through the manifest stored in the bucket (hashed filenames,
immutable CDN caching).
Two operational rules:
StaticStorage tolerates a missing manifest: it falls back to computing
hashed names per {% static %} tag, which costs one
S3 round trip per tag (measured on qa 2026-08-30: ~70 serial calls, ~2.3 s
per page render). The fallback result itself is not cached; only manifest
reload attempts are gated to one per manifest_retry_interval (60 s). A
worker that booted too early stays slow until the next retry boundary
after the upload appears, then self-heals without a restart.The web deployment probes are values-templated
(web.livenessProbe/web.readinessProbe): default timeout 10 s and 5
failures, so a busy render cannot kill the pod through the probe budget
(measured death spiral on qa 2026-08-30 with the old hardcoded 5 s/3).
The operator repository holds:
PostgreSQL is shared across associations per database, and Redis must be treated the same way. Two sanctioned layouts:
redis.enabled: true; the standby sets
redis.enabled: false and redis.externalUrl to the live release’s
Redis service (e.g. redis://<live-release>-redis:6379). Live and
standby must never get separate Redis instances: a separate standby
broker strands queued Celery tasks and splits Channels group state
during cutover. When roles switch, the Redis service itself does not
move: the old live’s Redis keeps serving, and only the deployments
change.redis.enabled: false + a pathless redis.externalUrl per
site, with a unique redis.database number per association (0-15, the
default Redis range). The database number is applied to the cache,
Channels, and Celery broker and result backend, so queues and keys
cannot collide.Example: with one cluster Redis at redis://redis.internal:6379, site A
uses redis.database: 0, site B redis.database: 1, and both releases of
each site point at the same URL.
Ephemeral Redis implies an accepted task-loss model on broker loss: the database backup does not preserve queued Celery tasks. Tasks that must survive need a durable source of truth with reconciliation/re-enqueue, or a durable broker; keep the backup/restore pipeline for database data only.
Every site has a standby release (same chart, fullnameOverride,
shares the site’s database and secrets, no ingress) that is scaled to zero
between deploys. A deploy:
python manage.py redact_harassment_logs
after the release that stopped writing harassment report text into the admin
log (see operations.md)Database migration rule (important): the standby shares the site’s database, so a destructive migration breaks the live site the moment the standby migrates — not at cutover. Therefore:
Each component gets a deliberate shutdown policy:
terminationGracePeriodSeconds: 70, preStopSleepSeconds: 2):
the short preStop sleep gives the ingress time to observe endpoint
termination and stop routing new requests; on SIGTERM gunicorn drains
in-flight requests for gunicorn.gracefulTimeout (60s, configured
explicitly; the default graceful timeout is only 30s) before killing
workers. Requests longer than the graceful timeout are cut off at the cap
so a deploy is never held open indefinitely. WebSockets are closed when
the worker is killed at the graceful timeout: the Uvicorn worker stops
accepting new connections on SIGTERM and there is no application-level
drain hook (the removed separate asgi pod behaved the same way).CELERY_TASK_ACKS_LATE and CELERY_TASK_REJECT_ON_WORKER_LOST, work that
was acknowledged but not finished is requeued on worker loss instead of
being dropped; tasks must tolerate redelivery. The grace period bounds
the roll stall: short tasks finish in place, longer ones requeue to the
new worker instead of being waited out.Before relying on this, verify once per environment:
web logs during a rollout; confirm gunicorn logs graceful
worker shutdown and no 5xx for in-flight requests.web rollout and
confirm it closes cleanly (client sees a close frame, not a hang).secret.existingSecret. Nothing secret goes in
values files or git.Static files can be served from the same per-association public bucket as
media (prefix static, CDN in front via the public custom domain) instead
of the pod filesystem. Enable per site:
media:
s3:
staticEnabled: true
staticCustomDomain: "" # defaults to publicCustomDomain
migrations:
job:
enabled: true # required: the Job performs the upload
Rollout ordering (critical): the migration Job (sync wave 0, from
#1113) uploads static before application pods start. Enabling
staticEnabled and the upload must happen in the same release, so pods
never render S3 URLs before the bucket is populated. Rollback: flip
staticEnabled back; the image still carries the build-time static trees,
so old pods serve from the pod filesystem again.
Bucket requirements (one-time):
Access-Control-Allow-Origin: * on the
bucket (or the CDN origin) for @font-face.Cache-Control: public, max-age=31536000, immutable on the static
prefix. Old hashes expire naturally, no CDN invalidation needed.static prefix; the public custom domain already
serves media/public, so the same domain works.startingDeadlineSeconds / activeDeadlineSeconds /
timeZone to bound missed and hung runs.values-hetzner.yaml are based on observed
production usage: web is the largest process (~300-450Mi, plus a small
increment for the Channels stack now that it also serves WebSockets),
Celery ~250Mi. Revisit after sustained traffic or big uploads.core.asgi),
so the /ws ingress/gateway route and the asgi Service are gone. Per-site
operator values must drop their asgi: overrides; WebSocket traffic
follows the normal / path to the web service.web.asgi.maxConcurrentHttpRequestsPerWorker (default 8); WebSockets do
not consume slots. With the default 3 Gunicorn workers, one pod therefore
runs at most 24 concurrent Django HTTP executor threads. Replica count and
rollout surge multiply that bound. A slot covers the complete ASGI HTTP
exchange, including request input and response streaming, so slow clients
consume capacity rather than bypass the memory/connection bound. Configure
finite ingress request and response timeouts accordingly.CONN_MAX_AGE=0: a connection cannot be reused after
Django destroys its request executor, so keeping it open would strand the
thread-local backend until PostgreSQL reaps it. The outer
date.middleware.ConnectionLifecycleMiddleware runs
close_old_connections() on the owning thread at request start and end;
it remains defense-in-depth for exceptional disconnect/error paths beyond
Django’s normal request-signal cleanup. Celery, migrations and
administrative sessions are outside the web concurrency bound. PgBouncer
is the recommended way to amortize PostgreSQL connection setup without
persistent application connections.