Hermes Agent: Updating Without Breaking Running Crons or the Live Gateway

hermes update pulls the latest NousResearch commit into your install directory and reinstalls Python dependencies in place. On a live install with 10 scheduled cron jobs, a Telegram gateway, a cloudflared tunnel, and two watchdog sweeps every 5 minutes, that same one-line operation can take the entire operator surface offline in a way that is invisible until the next user message fails. This guide is the discipline for that case.

Why most updates get botched

Hermes’ active installation state is everything the scheduler is holding open: 10 cron jobs, a Telegram gateway with auto-applied polling, a tunnel daemon, and the consolidated watchdog sweeps. Pulling a new commit into that runtime without first checking those surfaces means an update that breaks the gateway also makes the error invisible until the next user message fails. The cron subsystem retries on a delay. The Telegram polling wrapper does not.

The fix is not a fancier update command. It is a snapshot taken before the pull and a smoke-test pattern taken after. If the snapshot still matches at the 10-minute mark, the update stays. If anything has drifted, you roll back. There is no middle path.

Inventory the live state before the pull

Before hermes update does anything, you need a picture of what “healthy” looks like on this box. Today’s hermes cron list shows 68 jobs — twenty-eight state: paused, forty state: scheduled. The 10 active jobs are the operational core; if silently broken by an upgrade, they take down the operator surface. Confirm all are green via hermes cron list:

• Gateway Watchdog — Telegram polling (lets users reach you).
• a2a-relay-watchdog — keeps the A2A relay between VPS profiles alive.
• a2a-poller-watchdog — keeps the A2A poller alive.
• Mockup Studio Watcher — reacts to mockup queue events.
• Daily cloudflared tunnel restart — cycles the tunnel before stale-conn errors accumulate.
• site-watchdog-sweep — consolidates 5 service watchdogs + cascade probe.
• aux-services-sweep — consolidates 3 aux watchdogs.
• ABS daily Telegram briefing — morning push; visibly broken on day 1 if missed.
• ATS RD nightly R2 backup — silently broken if cron subsystem breaks.
• hourly-health-sweep — combines site uptime + matrix R2 gap.

Save the snapshot:

cp /opt/data/cron/jobs.json /tmp/jobs.json.pre-update
git -C /opt/hermes rev-parse HEAD > /tmp/hermes-pre-update-sha
hermes cron list > /tmp/cron-pre-update-$(date -u +%Y%m%dT%H%M%SZ).txt

abs-backend-watchdog is one of the 10 and is intentionally paused since 2026-07-12. It stays paused through the update; do not “tidy up” by resuming it.

Disable the unsafe path

Pause only the no_agent sweeps that would fire mid-update and step on the in-progress reinstall. Do not pause the gateway, do not pause the watchdogs, do not pause the user-facing surface.

hermes cron pause <matrix-r2-gap-check-id>
hermes cron pause <nightly-backup-id>      # if it fires within the update window

# Verify the pause actually took (P33: scheduler pause can lie)
ls -la /opt/data/cron/output/<job-id>/ | tail -3   # confirm no fresh .md files

The output-directory check matters: state: paused is a UI status, not a hard kill switch (cron-health-monitor P33). If the job has an output artifact newer than the pause timestamp, it fired anyway.

Three things to deliberately NOT disable:

1. The Telegram gateway. Killing it is the classic “now I can’t even report the breakage” failure mode.
2. The consolidated watchdog sweeps. The only reliable signal that the update broke something.
3. The dashboard watchdog. Annoying when it triggers mid-update. The difference between “I know the dashboard broke” and “the dashboard has been down for six hours and I shipped a broken update.”

Pull upstream

Note the pre-update hermes --version output (today’s on this VPS: Hermes Agent v0.18.2 (2026.7.7.2) · upstream 2d0f2185 · 416 commits behind). Run the update with explicit inspection of what changed:

hermes update --check                                    # confirm update is available
git -C /opt/hermes fetch upstream
git -C /opt/hermes log --oneline --since=1day upstream/main | head -50
hermes update --yes --backup                             # --backup keeps a tarball
hermes --version                                         # confirm receipt

The first upstream commit that touches cron/jobs.json, the gateway adapter, or the watcher sweeper is the one to watch the smoke tests against. --backup is off by default; override on the first run of a long-running box.

Smoke-test the new runtime

The 10-minute window after hermes update is where broken updates reveal themselves. Four commands:

hermes chat -q "Reply with OK"                           # agent alive
diff /tmp/cron-pre-update-*.txt <(hermes cron list)      # cron store intact
hermes gateway status --deep                             # adapter healthy
hermes curator --dry-run                                 # skill store parses

If all four return green, the snapshot still matches. If any drift, the rollback rule applies — do not try to fix forward.

Drift patterns to recognize on sight:

• hermes chat -q returns ModuleNotFoundError → lost a dependency on reinstall. Roll back.
• Cron shows the right jobs but last_status: error on a job that worked pre-update → no_agent script path resolver changed (P32). Roll back; diagnose in a branch.
• hermes gateway status --deep shows connected but polling token expired → token format rolled. Roll back, refresh, re-update.
• hermes curator --dry-run errors on a SKILL.md you did not touch → frontmatter validator stricter. Roll back; let upstream fix the validator before mass-editing skills.

Re-enable disabled jobs

Only after all four smoke tests pass:

hermes cron resume <matrix-r2-gap-check-id>
hermes cron resume <nightly-backup-id>

Verify with hermes cron list. The abs-backend-watchdog stays paused — that pause predated the update and is intentional. The cron-health-monitor skill’s P36 rule applies: any post-update cron-management script must operate on a hardcoded ID list, not a sweeping filter on state == 'paused'. The 28 currently-paused jobs are not garbage to be collected.

What goes wrong

The same regressions repeat across upgrades. Knowing their shape means you spot them in the smoke test instead of three days later.

Symptom	Cause	First place to look
hermes chat -q hangs >30s, returns 502	Model provider config drift or global inference guard tripped	~/.hermes/config.yaml inference: block
hermes cron list shows right jobs but last_status: error post-update	No_agent script path resolver changed (P32)	/opt/data/cron/output/<job-id>/<newest>.md — usually “Script not found”
Gateway disconnects on first post-update message	Token rotation or polling URL change	hermes gateway status --deep; rotate if expired
Cron deliver messages don’t arrive	deliver field enum changed (telegram:<chat_id> accepted, telegram not)	hermes cron edit <id> --deliver telegram:<chat_id>
top shows hermes at 200% CPU	New code path missing a backoff (common in watcher reorgs)	ps auxf | grep hermes; if confirmed, roll back
Tunnel restarts but every hostname returns 1033	Cloudflared config schema changed	/etc/cloudflared/config.yml; check upstream changelog

The rollback rule

Any update that does not pass the smoke test within 10 minutes of the pull gets rolled back. Period.

The 10 minutes is the budget for transient issues (stale venv cache, race on first hermes invocation, gateway reconnect delay) — not the budget for “let me try to fix it.” A fix-forward is a separate decision; it goes in a branch with a test, a smoke test on the fixed branch, and only then promoted. The rule exists because half of fix-forward attempts end up turning a 5-minute rollback into a 2-hour outage.

ROLLBACK_SHA=$(cat /tmp/hermes-pre-update-sha)
git -C /opt/hermes checkout "$ROLLBACK_SHA"
hermes update --yes
hermes --version                       # expect pre-update receipt
hermes chat -q "Reply with OK"
hermes cron list > /dev/null           # scheduler alive
hermes gateway status --deep

Tell the user: “Update at <commit> broke <smoke-test>. Rolled back to <pre-sha>. No state lost.”

If the rollback itself fails (pre-update tree no longer reinstalls cleanly because dependencies diverged too far), escalate — that is a sign the install needs a clean reinstall, not another update.

What not to do

• Do not update during peak hours. The user’s first message at 9 AM is a bad time to discover the gateway adapter regressed.
• Do not update without inspecting git log --since=1day on upstream.
• Do not skip the post-update smoke test.
• Do not disable the dashboard watchdog just because it triggers mid-update.
• Do not pause a job and forget to resume it. The 28 paused jobs are paused for unrelated reasons — they are not “temporarily paused for the upgrade.” Pause only the 0–3 jobs in the unsafe path.
• Do not “fix forward” within the 10-minute window.
• Do not skip --check on a box you have not touched in weeks.

Verification checklist

#	Command	What it proves
1	hermes --version shows new build	Update took
2	hermes chat -q "Reply with OK"	Agent runtime alive
3	diff /tmp/cron-pre-update-*.txt <(hermes cron list)	Cron store intact
4	hermes gateway status --deep	Adapter healthy
5	/opt/data/cron/output/<id>/<newest>.md is fresh after resume	Pause was a real pause, not a UI lie

Sources

• Hermes Agent update docs
• ABS: Hermes Agent Cron Job Management
• ABS: Hermes Agent Setup v2

Last verified 2026-07-14: Hermes v0.18.2 (2026.7.7.2), ‘416 commits behind’.