Operations and troubleshooting
This chapter covers running a KasaDESK installation after first setup: key custody, backups and tested restores, upgrades and rollback, scaling, monitoring, retention and troubleshooting. It ends with a release checklist for operators.
Commands assume the Compose deployment from Install KasaDESK, run from the release checkout.
Protect the encryption keyLink to this section
ENCRYPTION_KEY seals channel passwords and tokens, widget identity secrets, webhook secrets, S3 credentials, browser push subscriptions and staff authenticator seeds. It is never included in backups.
- Store it in a secret manager with its own access control and backup, separate from backup storage.
- Record its key ID and the dates it was in use. The key ID is the first 8 hex characters of the SHA-256 of the base64 key text. Backups record only this ID.
- Keep retired keys for as long as you keep backups that need them.
Rotation. Set the new key as ENCRYPTION_KEY and the old one as ENCRYPTION_KEY_PREVIOUS, then restart. New values are sealed with the new key and old ones still open. There is no bulk re-seal command, so keep ENCRYPTION_KEY_PREVIOUS until every sealed value has been re-saved. Each backup manifest lists sealed secrets by key ID so you can track progress.
If the key is lost, conversations, customers, attachments and settings remain, but every sealed secret is lost: re-enter mailbox passwords and provider credentials, rotate widget identity and webhook secrets and update their consumers, and have every staff member enroll their authenticator again.
Back up and restoreLink to this section
A KasaDESK backup is one directory containing a consistent PostgreSQL dump, an archive of STORAGE_DIR, file checksums and a manifest. The manifest records migrations, row counts, artifact SHA-256 values and the encryption key ID, never a key or secret. Backups run while the server is up.
Run backupsLink to this section
# One run now: create, re-check checksums, off-site copy if configured, prune, write last-run.json
docker compose --env-file deploy/.env -f deploy/docker-compose.yml --profile backup run --rm backup
# Or a long-running scheduler: first run at start, then every BACKUP_INTERVAL_SECONDS (default 86400)
docker compose --env-file deploy/.env -f deploy/docker-compose.yml --profile backup-schedule up -d backup-schedulerOr schedule the one-shot service from host cron, for example daily at 02:15 UTC:
15 2 * * * cd /srv/kasadesk && docker compose --env-file deploy/.env -f deploy/docker-compose.yml --profile backup run --rm backup >> /var/log/kasadesk-backup.log 2>&1Each run creates kasadesk-<UTC timestamp>/ in the backups volume and keeps the newest backup of each of the last BACKUP_KEEP_DAILY days (default 7) and BACKUP_KEEP_WEEKLY weeks (default 4). A failed run removes its partial directory.
Monitor it. Each run writes last-run.json with ok, timestamps, the backup name and the off-site result. KasaDESK does not send backup alerts itself. Alert when ok is false or the file is older than your interval plus a margin:
docker compose --env-file deploy/.env -f deploy/docker-compose.yml --profile backup run --rm --entrypoint sh backup \
-c 'cat /var/lib/kasadesk/backups/last-run.json'Keep a copy off the hostLink to this section
- Admin-managed: in Admin → Settings → Storage, choose a private S3-compatible bucket, enable Send scheduled off-site backups here, then Verify and save. The scheduler then uploads each backup. This applies when
BACKUP_OFFSITEis empty. - Environment-managed: set
BACKUP_OFFSITE=s3withOFFSITE_S3_ENDPOINT,OFFSITE_S3_BUCKET,OFFSITE_S3_ACCESS_KEY_IDandOFFSITE_S3_SECRET_ACCESS_KEY, plus optional region, prefix and path-style settings.BACKUP_OFFSITE=offdisables scheduled copies. - Manual: copy the
backupsvolume to other storage on a timer.
Before relying on an S3 copy, upload one backup, download an object and compare its SHA-256 with the manifest. Use a write-only key, bucket encryption and a lifecycle rule; backup prune deletes local backups only. The dump contains customer data in clear text, so protect backup storage accordingly.
Restore into a clean databaseLink to this section
-
Stop the app and its workers on the target.
-
Provision an empty database and an empty storage directory.
-
Configure
DATABASE_URL,STORAGE_DIRandENCRYPTION_KEY, plusENCRYPTION_KEY_PREVIOUSif needed. Together the keys must cover every key ID in the backup’s manifest. -
Use KasaDESK code of the same release as the backup, or a newer one that ships every migration listed in its manifest.
-
Check the backup offline, then restore. With Compose, do this in a project whose
pgdataandstoragevolumes are new and empty, withdbrunning and the backup directory copied into itsbackupsvolume:backup_dir='kasadesk-20261121T120000Z' # replace with your backup directory name docker compose --env-file deploy/.env -f deploy/docker-compose.yml --profile backup run --rm --entrypoint node backup \ dist/cli.js backup check "/var/lib/kasadesk/backups/$backup_dir" docker compose --env-file deploy/.env -f deploy/docker-compose.yml stop app docker compose --env-file deploy/.env -f deploy/docker-compose.yml --profile backup run --rm --entrypoint node backup \ dist/cli.js backup restore "/var/lib/kasadesk/backups/$backup_dir" -
If the output says the backup predates migrations in this code, run
docker compose --env-file deploy/.env -f deploy/docker-compose.yml run --rm migrate. -
If you need time to review retention settings, set
RUN_WORKERS=falsefor the first app start. Review the rules, then re-enable workers and recreate the service.
The restore verifies checksums and migrations before writing anything, refuses a non-empty target, and checks afterwards that row counts, files and sealed secrets match.
| Exit code | Meaning |
|---|---|
| 0 | Restored and verified. |
| 1 | Configuration, checksum, migration, tool or verification failure. |
| 2 | Refused because the target database or storage is not empty. |
| 3 | Wrong key: the configured keys cannot open this backup’s secrets. |
Prefer restoring into a new database and switching DATABASE_URL, so the old database stays available. backup restore … --force overwrites an existing database but leaves objects that are not in the dump.
After restoring, check by hand:
-
/healthzreturns 200 and/readyzreturnsready. - A staff member signs in with password and a current authenticator code.
- The inbox lists conversations and an attachment downloads.
- Email channels show saved secrets, and Test connection passes where the mail host is reachable.
- API keys and webhooks are listed in Developer, and a test webhook delivery succeeds.
- Review Retention before workers run. A restore brings back data that retention had deleted since the backup.
- Rotate any credential the incident may have exposed.
Test restores regularlyLink to this section
Restore the latest backup into a disposable database and storage directory, with workers off, on a schedule you choose, at least before every upgrade. Time it to know your real recovery time. Keep workers off in any restored copy that can reach your live S3 buckets or providers.
Point-in-time recoveryLink to this section
For recovery between logical snapshots, add deploy/docker-compose.pitr.yml to the base Compose deployment. It turns on PostgreSQL WAL archiving and adds the walarchive and pgbasebackups volumes. Adding it can recreate the database container, so schedule the change. Take regular base backups and preserve every WAL segment you need from the oldest retained base backup.
docker compose --env-file deploy/.env \
-f deploy/docker-compose.yml -f deploy/docker-compose.pitr.yml up -d --wait
docker compose --env-file deploy/.env \
-f deploy/docker-compose.yml -f deploy/docker-compose.pitr.yml exec -u postgres db \
pg_basebackup -U kasadesk -D "/basebackups/base-$(date -u +%Y%m%dT%H%M%SZ)" -Ft -z -Xs -c fast
docker compose --env-file deploy/.env \
-f deploy/docker-compose.yml -f deploy/docker-compose.pitr.yml exec -u postgres db \
psql -U kasadesk -Atc "SELECT archived_count, failed_count, last_archived_wal, last_failed_wal FROM pg_stat_archiver"Copy both the base backups and archived WAL off the host. The overlay stores these locally. Restore into a new empty PostgreSQL data directory using tools/ops/pitr-restore.sh from the matching release. The command below is for an isolated drill. Replace the example time with a timestamp covered by your archive and use the same major PostgreSQL version as the backup.
tools/ops/pitr-restore.sh \
--base /srv/restore-drill/base-backup \
--wal-archive /srv/restore-drill/wal-archive \
--target-time '2026-11-21 12:00:00+00' \
--data-dir /srv/restore-drill/new-pgdata --disable-archivingWithout --start, the script prepares the recovered cluster for replay. Start it in the isolated environment, confirm it reached the target and inspect the newest records before planning a cutover. A drill can use --start --port and --socket-dir to replay over a Unix socket. If the archive ends before the target, recover from a complete archive or choose an earlier target. Rehearse a Compose volume cutover separately before depending on it.
WAL restores the database only. Restore attachment storage from snapshots or a logical backup and verify that representative attachments open. Keep background workers disabled during recovery checks. A completed recovery starts a new timeline, so take a new base backup after cutover.
Upgrade KasaDESKLink to this section
-
Read the release notes for new migrations and configuration.
-
Back up first and note the encryption key ID. Test that the backup restores.
-
Build or pull the new image:
docker compose --env-file deploy/.env -f deploy/docker-compose.yml buildOr set
KASADESK_IMAGEto a tag you publish. -
Apply and start:
docker compose --env-file deploy/.env -f deploy/docker-compose.yml up -d --waitThe
migrateservice runs with the new image and must succeed beforeappis recreated. -
Check
/readyz, sign in, and run the smoke checks.
If /readyz reports migrations_pending, run docker compose --env-file deploy/.env -f deploy/docker-compose.yml run --rm migrate. A migration file changed after it was applied is refused as migrations_mismatch.
Rollback boundariesLink to this section
- Each migration has a down step, run with the new image:
node dist/cli.js migrate down [steps]. - Down steps can drop data created by the newer release. For example, rolling back the staff phone app, staff notification and widget callback migrations (
0034_staff_mobile,0035_staff_inbox_notifications,0036_widget_callback) removes phone sessions, alert preferences and the callback setting. - When in doubt, restore the pre-upgrade backup into a fresh database instead of migrating down.
- Older app instances keep reporting ready while the database is ahead, as long as migrations are additive. Do not run old and new releases side by side for long.
Scale and run workersLink to this section
The Compose file runs one app process. Before adding more:
- Run background workers (
RUN_WORKERS=true) in exactly one process. Run extra HTTP-only replicas withRUN_WORKERS=false; their/readyzreports workers asdisabled. - Attachments stored on the local filesystem need shared storage across hosts, or switch new attachments to S3-compatible storage in Admin → Settings → Storage.
- Outbound messages and webhooks are delivered at least once. Webhook receivers should deduplicate by event ID.
- Keep
SHUTDOWN_TIMEOUT_SECONDS(default 25) below your orchestrator’s stop timeout. On stop, the app drains requests, live streams and workers.
Monitor the installationLink to this section
- Readiness: probe
/readyz. Reason codes are listed in Check health and readiness. - Metrics: Prometheus text at
/metricson the internal listener (port 9464 on the container network), withAuthorization: Bearer <METRICS_TOKEN>. Useful series includekasadesk_http_requests_total,kasadesk_worker_last_success_timestamp_seconds,kasadesk_outbox_queue_depth,kasadesk_webhook_queue_depth,kasadesk_sse_connectionsandkasadesk_db_up. - Logs: JSON lines on stdout. Each request line has a request ID, route template, status and duration. Raw URLs, query strings, cookies and secret-looking fields are never logged. Ship stdout with your log driver.
KasaDESK does not include alert rules or dashboards. Alert at least on /readyz failures, stale worker successes, growing outbox depth and backup status.
Data retentionLink to this section
Admin → Settings → Retention sets per-workspace deletion rules and legal holds. All rules are off until an admin enables one, and enabling or shortening a rule requires confirming a dry-run preview. Deleted data remains in older backups until they age out, so your effective retention is the policy period plus your backup retention. The audit log is append-only.
TroubleshootLink to this section
| Symptom | Likely cause | What to do |
|---|---|---|
app never becomes healthy |
Missing variable, unreachable database or failed migration | docker compose --env-file deploy/.env -f deploy/docker-compose.yml logs --tail=100 app migrate |
/readyz 503 migrations_pending |
App started without migrating | Run the migrate service. |
| Console sign-in loops or fails over the internet | Console served over HTTP, or wrong origin | Serve over HTTPS and set PUBLIC_API_ORIGIN and AGENT_CONSOLE_ORIGIN to it. |
| Rate limits or audit entries show the proxy’s IP | More or fewer than one proxy hop | Use exactly one reverse proxy in front of the app. |
| Inbox or chat updates are delayed | Proxy buffering or short timeouts on the event streams | Disable buffering and raise read timeouts for /api/v1/inbox/events and /api/v1/widget/events. |
| Chat frame does not load on a site | Origin not allowlisted, or the page’s CSP blocks it | Add the exact origin, including any port, and allow the installation in script, frame and font sources. |
| Chat says the site key is unknown | Channel disabled, removed, or its workspace archived | Check the channel in Connected channels. |
| Native app chat refused | Native app access is disabled, or verified_only without a token |
Change the channel setting or send a signed identity. |
| Verified customer shows as guest | Token rejected: wrong secret, wrong site key in aud, expiry more than one hour ahead, or an invalid claim |
Fix the backend signer. Tokens are checked as a whole. |
| Provider messages never arrive | Webhook URL not registered, PUBLIC_API_ORIGIN not public HTTPS, or the channel is not assigned and activated |
Run Verify connection and Run live check. Assign under Brands on this channel and activate. |
| Replies stay “Accepted by provider” | Normal for Telegram. Elsewhere, receipt webhooks are not subscribed. | Check the provider’s receipt subscriptions. |
| Email arrives twice or not at all with several replicas | Workers running in more than one process | Run workers in one process only. |
Backup fails with a pg_dump version error |
Client tools older than the server | Use the image’s tools, or set PG_BIN_DIR to tools of the same or a newer major version. |
| Restore exits with code 3 | Wrong encryption key | Configure the key, or ENCRYPTION_KEY_PREVIOUS, matching the manifest’s key IDs. |
| Staff must re-enroll authenticators after a restore | Restored with a different key | Restore again with the original key. |
Release checklist for operatorsLink to this section
Run this on a disposable environment with synthetic data before each first install or upgrade, then repeat the customer-facing items on production with test accounts.
Deployment
- Fresh install from the release reaches
/readyzreadythrough HTTPS. - Upgrade from your current release applies only the expected migrations and keeps existing data.
- Owner setup with a one-time code, authenticator enrollment, first workspace and workgroup.
- Stop and restart keeps sign-in, conversations and attachments.
Data safety
-
backup runsucceeds andlast-run.jsonreportsok. - The latest backup restores into an empty database with exit code 0 and passes the manual checks.
- The off-site copy, if used, is present and its checksum matches.
- The encryption key and its key ID are stored separately from backups.
Customer surfaces
- Website embed on an allowlisted origin: guest message, staff reply, internal note not visible to the customer.
- A non-allowlisted origin is refused.
- Verified identity: the customer is marked Verified and resumes the same open ticket from a second surface on the same channel.
- A second brand’s channel does not show the first brand’s conversations.
- Callback request appears in the ticket timeline when enabled, and is absent when disabled.
- Published widget colours appear after reload; the inbox badge colour is unchanged.
- WordPress, hosted Help Center WebViews and the SDK, if you use them, on your own stack and physical devices.
Channels
- For each provider you enable: verification, live check, inbound message in the right brand and workgroup, and a reply received on a test device.
Staff
- Each role lands on the right page and sees only its scope.
- Inbox alerts, sound, and background browser alerts if configured, on the browsers your staff use.
- Desktop install opens the normal sign-in.
- Staff phone app sign-in, once available, against your HTTPS origin.
- Phone calling and background mobile alerts only after their own device tests pass.
AI
- Every provider connection passes Test connection with your real key.
- Autopilot answers only from matching published articles and hands off otherwise.