Backup & upgrade

Day-2 operational procedures — backup, restore, upgrade, rollback, rotate secrets.

8 min readUpdated Mar 12, 2026

Two data stores hold state that matters: Postgres (everything structured — users, channels, messages, webhook deliveries) and MinIO (attachment files). Redis only holds ephemeral state (rate-limit counters, presence markers, typing) — you can throw it away and restart without data loss.

Postgres

Daily dump

terminal

docker compose exec -T postgres \
    pg_dump -U cariosan -d cariosan --format=custom \
    > "/backups/cariosan-$(date +%F).dump"

Compression — chat content compresses well (~5-10× on typical traffic). Rotate weekly or monthly based on compliance needs.

Restore

terminal

docker compose down cariosan-server
docker compose exec -T postgres \
    pg_restore -U cariosan -d cariosan --clean < cariosan-YYYY-MM-DD.dump
docker compose up -d cariosan-server

Stop the server first — taking cariosan-server down before the restore avoids racing with live INSERTs. Postgres will gladly let you restore over a busy database, but you'll get a corrupted final state.

Point-in-time recovery

The bundled Postgres container doesn't ship WAL archiving. If RPO matters, switch to a managed Postgres (RDS, Neon, Supabase) that handles PITR natively, or replace the container with a postgres:15 image configured with archive_mode=on.

MinIO

Mirror to another bucket

terminal

docker run --rm --network cariosan_cariosan \
    -v /backups/minio:/target minio/mc sh -c '
        mc alias set local http://minio:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD"
        mc mirror --remove local/cariosan-attachments /target
    '

For geographical redundancy, replace /target with a remote S3 bucket alias. mc mirror --remove keeps the backup in sync with deletions on the source.

Upgrade

Pin the image tag in .env (see docker-compose), bump to the next release, and pull:

terminal

# Edit .env: CARIOSAN_IMAGE=ghcr.io/cariosan/server:v0.2.0
docker compose pull
docker compose up -d

The new container applies pending migrations on boot. It only starts serving traffic after its healthcheck passes (~10-15s start period), so there's a brief cutover window during which the old container is stopping and the new one is migrating.

Rollbacks are safe — MVP migrations are additive only (no DROP / ALTER COLUMN). Rolling back is just CARIOSAN_IMAGE=...:v0.1.0 + docker compose up -d.

Rotating secrets

`CARIOS_JWT_SECRET`

Restarting with a new value invalidates every outstanding client JWT in one go. Users must sign in again; the client SDK surfaces this as UNAUTHORIZED.

Coordinate with your frontend — for a smooth rotation, push a new token-refresh flow first, wait a deploy, then rotate. Otherwise active users see a logout storm.

Workspace API secret

Use the server CLI:

terminal

docker compose exec cariosan-server /cariosan workspace rotate-secret \
    --api-key pk_live_abc

Update your backend first — the old secret stops working immediately. Set the new CARIOS_API_SECRET in your backend env before running rotate-secret, not after.

`MINIO_ROOT_PASSWORD`

Redeploy with the new password in .env. Existing presigned URLs (15-minute TTL) remain valid until expiry; new uploads use the new credentials.

`POSTGRES_PASSWORD`

Changing env var alone is NOT enough — Postgres doesn't re-hash its password from the env var on restart. Either run ALTER USER ... WITH PASSWORD ... from inside Postgres and then update .env, or recreate the postgres_data volume from a fresh backup under the new credentials.

Monitoring

Basic health signal: GET /healthz returns 200 when the server is up. The Docker HEALTHCHECK directive uses the built-in /cariosan healthcheck subcommand — don't try to replace it with curl, the distroless image doesn't ship one.

Rich observability — Prometheus metrics at /metrics, structured JSON logs to stdout, request traces — ships in the MVP server.

Was this page helpful?

On this page