Skip to main content

Watch

The watch command monitors Consul KV prefixes for changes and commits every change to a Git repository. It runs as a long-lived process, typically as a systemd service or a Kubernetes deployment.

How it works​

Guardian uses Consul's blocking queries (long-polling) to detect changes. Instead of polling every N seconds and comparing, blocking queries hold the HTTP connection open until something changes -- or until a timeout (default: 5 minutes).

Here's the flow:

  1. Guardian sends a request with the last known X-Consul-Index.
  2. Consul holds the connection open until the index changes (a key was created, updated, or deleted).
  3. Guardian receives the new state, diffs it against the previous state, and detects what changed.
  4. Changed keys are written to the filesystem and committed to Git.
  5. Repeat from step 1.

This gives you near-real-time detection (typically under 5 seconds) with minimal load on the Consul cluster.

Basic usage​

consul-guardian watch \
  --consul-addr http://localhost:8500 \
  --prefix "config/" \
  --git-repo ./consul-backup

Configuration options​

Prefixes​

Watch multiple KV prefixes by passing a comma-separated list:

consul-guardian watch \
  --prefix "config/,env/,feature-flags/"

Each prefix gets its own blocking query goroutine, so changes to different prefixes are detected independently.

Exclude prefixes​

Skip keys that match specific prefixes:

consul-guardian watch \
  --prefix "config/" \
  --exclude-prefix "config/temp/,config/cache/"

Excluded keys are filtered client-side after the blocking query returns. Useful for high-churn keys you don't need in version control.

Wait time​

The blocking query timeout. After this duration with no changes, the connection resets and a new one starts:

consul-guardian watch --wait-time "5m"

Valid values: 10s to 10m. Longer values mean fewer reconnections but slower failure detection. The default of 5m works well for most setups.

Auto-push​

Automatically push commits to a remote Git repository:

consul-guardian watch --auto-push

If the push fails, the local commit is preserved and push retries on the next change. Configure a remote first:

cd consul-backup
git remote add origin git@github.com:yourorg/consul-backup.git

Commit author​

consul-guardian watch \
  --commit-author "consul-guardian" \
  --commit-email "guardian@consul.local"

Git sync behavior​

File organization​

Keys map directly to filesystem paths. A key like config/database/host becomes a file at config/database/host.json (or .yaml, .txt, etc., based on content detection).

consul-backup/
  config/
    database/
      host.json
      port.json
    cache/
      ttl.json
  feature-flags/
    dark-mode.json
    new-checkout.json

Commit messages​

Each commit describes what changed:

consul-guardian: 2 keys changed

Modified: config/database/host
Added: config/database/max_connections

Timestamp: 2026-04-04T14:30:00Z

Initial sync​

On first run, Guardian fetches all existing keys under the watched prefixes and commits them as the initial state. This baseline commit contains your full KV tree.

Running as a background service​

For production, run Guardian as a systemd service or in a container. See:

Error handling and retries​

Guardian handles transient failures with adaptive backoff:

  • Consul connection lost: Retries with exponential backoff (up to 15 seconds).
  • Index goes backward: This can happen after a Raft snapshot restore or leader failover. Guardian resets its index and does a full re-sync.
  • Git commit fails: Logged as an error. The next successful sync will include the missed changes since Guardian always diffs against its tracked state.

All errors are logged with structured fields (prefix, error, backoff). Set --log-format json for machine-readable logs.

Real-world example​

A team uses Consul KV for feature flags across 20 microservices. A developer accidentally sets feature-flags/payment-processing to false, disabling payments.

Without Guardian: The team spends 45 minutes figuring out what changed and when.

With Guardian:

cd consul-backup
git log --oneline -- feature-flags/payment-processing
# a1b2c3d consul-guardian: 1 keys changed    (2 minutes ago)
# e4f5g6h consul-guardian: 1 keys changed    (3 days ago)

git diff a1b2c3d~1 a1b2c3d -- feature-flags/payment-processing
# -true
# +false

The team sees the exact change, when it happened, and can restore it immediately.