Backup & upgrade

Two data stores hold state that matters: Postgres (everything structured — users, channels, messages, webhook deliveries) and MinIO (attachment files). Redis only holds ephemeral state (rate-limit counters, presence markers, typing) — you can throw it away and restart without data loss.

Postgres

Daily dump

docker compose exec -T postgres \
    pg_dump -U cariosan -d cariosan --format=custom \
    > "/backups/cariosan-$(date +%F).dump"

Restore

docker compose down cariosan-server
docker compose exec -T postgres \
    pg_restore -U cariosan -d cariosan --clean < cariosan-YYYY-MM-DD.dump
docker compose up -d cariosan-server

Point-in-time recovery

The bundled Postgres container doesn't ship WAL archiving. If RPO matters, switch to a managed Postgres (RDS, Neon, Supabase) that handles PITR natively, or replace the container with a postgres:15 image configured with archive_mode=on.

MinIO

Mirror to another bucket

docker run --rm --network cariosan_cariosan \
    -v /backups/minio:/target minio/mc sh -c '
        mc alias set local http://minio:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD"
        mc mirror --remove local/cariosan-attachments /target
    '

For geographical redundancy, replace /target with a remote S3 bucket alias. mc mirror --remove keeps the backup in sync with deletions on the source.

Upgrade

docker compose exec -T postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" \
  -c "SET statement_timeout = 0; CREATE INDEX CONCURRENTLY idx_messages_channel_created_id ON messages (channel_id, created_at, id);"
# must print t; if it prints f the build failed — drop it with DROP INDEX CONCURRENTLY and retry
docker compose exec -T postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" \
  -c "SELECT indisvalid FROM pg_index WHERE indexrelid = 'idx_messages_channel_created_id'::regclass;"

Pin the image tag in .env (see docker-compose), bump to the next release, and pull:

# Edit .env: CARIOSAN_IMAGE=ghcr.io/cariosan/server:v0.2.0
docker compose pull
docker compose up -d

The new container applies pending migrations on boot. It only starts serving traffic after its healthcheck passes (~10-15s start period), so there's a brief cutover window during which the old container is stopping and the new one is migrating.

Rotating secrets

CARIOS_JWT_SECRET

Restarting with a new value invalidates every outstanding client JWT in one go. Users must sign in again; the client SDK surfaces this as UNAUTHORIZED.

Workspace API secret

Use the server CLI:

docker compose exec cariosan-server /cariosan workspace rotate-secret \
    --api-key pk_live_abc

MINIO_ROOT_PASSWORD

Redeploy with the new password in .env. Existing presigned URLs (15-minute TTL) remain valid until expiry; new uploads use the new credentials.

POSTGRES_PASSWORD

Monitoring

Basic health signal: GET /healthz returns 200 when the server is up. The Docker HEALTHCHECK directive uses the built-in /cariosan healthcheck subcommand — don't try to replace it with curl, the distroless image doesn't ship one.

Rich observability — Prometheus metrics at /metrics, structured JSON logs to stdout, request traces — ships in the MVP server.

Realtime delivery health

Messages are persisted before their realtime frame is sent, so a lost frame never loses data; it only leaves connected clients stale until they re-sync. The following counters distinguish lost channel events from deliberately skipped ephemeral frames:

In a healthy deployment both stay at zero. A sustained non-zero rate() of publish_failed means Redis is unreachable or overloaded; inbox_full means one server is saturated. Example alert:

- alert: CariosanRealtimeLoss
  expr: sum(rate(cariosan_realtime_events_dropped_total{reason=~"inbox_full|publish_failed|authz_unavailable"}[5m])) > 0
  for: 10m
  annotations:
    summary: "Realtime frames are being dropped; clients are re-syncing from history"

Clients recover automatically (see Staying in sync); frames for clients on other servers cannot be signalled while the Redis bus itself is down, and those clients recover on their next reconnect.

When Redis is down, sending a message fails with 503 SERVICE_UNAVAILABLE and a Retry-After header (the rate limiter needs Redis). The rate-limit check is bounded to 2 seconds, so an unreachable or unresponsive Redis produces the 503 within about 2 seconds (a stopped Redis refuses the connection and answers at once). Nothing is stored, so the client can retry safely; the SDK marks the error retryable. Messages already stored are unaffected. A Redis publish that takes longer than one second counts as failed (publish_failed) and the channel's clients re-sync from history.