Skip to main content

Troubleshooting

Connection Issues​

"Cannot connect to Consul"​

Error: list kv prefix "config/": Get "http://localhost:8500/v1/kv/config/": dial tcp 127.0.0.1:8500: connect: connection refused

Cause: Consul is not running or the address is wrong.

Fix:

# Check Consul is running
curl http://localhost:8500/v1/status/leader

# Use the correct address
consul-guardian watch --consul-addr http://your-consul:8500

"ACL token not found" or "Permission denied"​

Cause: Missing or insufficient ACL token.

Fix:

# Set token via environment variable
export CONSUL_GUARDIAN_CONSUL_TOKEN=your-token

# Or via CLI flag
consul-guardian watch --consul-token your-token

Minimum ACL policy for watching:

key_prefix "" { policy = "read" }

Watch Issues​

"git commit: cannot create empty commit: clean working tree"​

Cause: Guardian detected changes but no files actually changed (data was identical).

This is harmless. It happens on startup when the Git repo already has the current state. The message appears once and stops after new changes arrive.

Watch detects initial state as "Added" on every restart​

Expected behavior. On startup, Guardian compares against an empty tracker and sees all keys as new. If the Git repo already has these files, the commit is skipped (empty commit error above).

Changes are not detected​

Check:

  1. Is the prefix correct? Key config/production/db requires prefix config/ not config/production/.
  2. Is the ACL token authorized for that prefix?
  3. Is Consul responding? curl http://consul:8500/v1/kv/config/?keys

Git Issues​

"git push failed, local commit preserved"​

Cause: Git remote is unreachable or SSH key is not configured.

Your data is safe. Local commits are preserved. When connectivity is restored, you can push manually:

cd consul-backup
git push origin main

Git repo grows too large​

Over time, the Git repo accumulates history. Typical growth:

  • 100 keys, 10 changes/day: ~50MB/year
  • 1000 keys, 100 changes/day: ~500MB/year

Mitigations:

# Shallow clone for CI/CD
git clone --depth 100 ./consul-backup

# Periodic garbage collection
cd consul-backup && git gc --aggressive

Dashboard Issues​

Dashboard shows blank page​

Check browser console (Cmd+Option+J on Mac) for JavaScript errors.

Common causes:

  • Stale cache: Hard refresh with Cmd+Shift+R
  • API not responding: Check curl http://localhost:9090/api/status
  • Frontend not built: Rebuild with cd frontend && npm run build

WebSocket disconnects frequently​

Cause: Network proxy or load balancer closing idle connections.

Fix: Configure your proxy to allow WebSocket upgrades:

# nginx
location /ws {
    proxy_pass http://guardian:9090;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_read_timeout 86400;
}

Restore Issues​

"CAS conflict in batch"​

Cause: Another process modified the key between Guardian reading the current state and writing the restore value.

Fix: Retry the restore. Guardian uses Check-And-Set to prevent silent overwrites. If the conflict persists, someone is actively writing to the same keys.

Restore does not delete extra keys​

Expected behavior with prefix-based restore. Guardian only restores keys that exist in the Git backup. Keys that were added directly to Consul after the backup are not deleted unless they were in the backup at some point.

Snapshot Issues​

"save snapshot: unexpected response code: 403"​

Cause: ACL token does not have snapshot permissions.

Fix: Snapshot requires elevated permissions:

acl = "write"
key_prefix "" { policy = "write" }
session_prefix "" { policy = "write" }

Performance​

High memory usage​

Guardian keeps the last 500 change events in memory. With large values, this can grow.

Mitigations:

  • Watch fewer prefixes
  • Use exclude_prefixes for high-churn keys
  • Increase the poll interval: --wait-time 10m

Slow initial sync with many keys​

First sync downloads all keys under watched prefixes. For 10,000+ keys, this takes 10-30 seconds.

This is a one-time operation. After initial sync, only changed keys are processed.