> For the complete documentation index, see [llms.txt](https://docs.pharos.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pharos.ai/open-telemetry/configuration/operations.md).

# Operations & Validation

Hermes health endpoints, a post-deployment validation checklist, how to apply configuration changes, and troubleshooting for the most common setup failures.

Once Hermes is deployed — Salesforce setup done, Terraform applied, `OTel_Configuration__mdt` set — this page covers how to confirm it's actually working, how to apply changes, and what to check when it isn't. Cloud-specific commands live in the [AWS](https://docs.pharos.ai/open-telemetry/configuration/aws.md) and [Azure](https://docs.pharos.ai/open-telemetry/configuration/azure.md) runbooks; the metrics Hermes reports about itself are catalogued in [Self-Instrumentation Metrics](https://docs.pharos.ai/open-telemetry/configuration/self-metrics.md).

***

## Health Model

Hermes exposes a small set of unauthenticated HTTP endpoints for liveness, readiness, and operational visibility. In the combined-supervisor deployment shape, these are served on `HEALTH_PORT` (default `8080`) and aggregate across whichever runtimes the supervisor is running.

| Endpoint   | Purpose                                                                                                                                                                                                                                 |
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `/health`  | Liveness. In a multi-org deployment, `HEALTH_MULTI_ORG_MODE` controls how per-org health rolls up into the single endpoint result — `all` (default; every org must be connected), `any` (at least one), or `majority` (more than half). |
| `/ready`   | Readiness — all orgs initialized (distinct from mere liveness). Per-org credential failures deliberately do not fail `/ready`: failing it would restart the container and take down the orgs that were fine.                            |
| `/stats`   | Per-org operational stats — useful for confirming which orgs are actually connected and processing.                                                                                                                                     |
| `/metrics` | Prometheus-format metrics, including scaling signals like `elf_lag_seconds` (the same signal cloud autoscalers reference as `argus_elf_lag_seconds`).                                                                                   |

![Hermes operational metrics — converter\_\*, elf\_lag\_seconds, and dedup — from the /metrics endpoint, in Grafana.](https://4264222071-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fdl9tmJ56lvwg5XY63QYr%2Fuploads%2Fgit-blob-25e3df3907d2abdf4cfc3bb1781876400b9c09a4%2Fgrafana-metrics-converter.png?alt=media)

{% hint style="warning" %}
**Prometheus-family backends need two adjustments for Hermes metrics.** If you scrape `/metrics` into a self-hosted Prometheus or Mimir (or route Hermes metrics into Grafana Cloud):

* **ELF metrics carry historical event timestamps**, not current wall-clock time — a Daily ELF file can land hours late. Set `out_of_order_time_window` to at least **72 hours**, or samples that arrive "in the past" relative to the default (much shorter) window are silently dropped with no ingest error. RTEM metrics use current wall-clock timestamps and aren't affected.
* **Hermes exports counters with delta temporality.** Prometheus/Mimir expect cumulative counters and silently drop delta sums by default — add a `deltatocumulative` Collector processor on the Prometheus-facing metrics pipeline. See [Grafana Cloud & LGTM](https://docs.pharos.ai/open-telemetry/configuration/backends/grafana.md) for where this fits in the Collector pipeline.
  {% endhint %}

**Per-org fault isolation.** Each org runs under its own supervisor-managed process group. If one org's connection fails — a bad credential, a transient Salesforce outage — that org restarts on its own **30-second cool-down** without affecting the other orgs in the same deployment.

{% hint style="warning" %}
**Health endpoints are unauthenticated.** `/health`, `/ready`, `/stats`, and `/metrics` don't require credentials to call. Keep them on a private network — don't expose `HEALTH_PORT` (or the per-runtime health ports in the combined-supervisor shape) to the public internet. The AWS and Azure modules give Hermes no public ingress.
{% endhint %}

***

## Validation Checklist

After deploying (or after a config change), confirm each of the following:

* [ ] **Container is healthy.** `/health` returns `200`; the startup log shows `hermes starting`, one `supervisor: starting runtime` line per runtime, `health: server listening`, and `SF access token refreshed`.
* [ ] **No auth errors in logs.** No repeated Pub/Sub authentication failures — if you see them, check the [JWT gotcha](#the-jwt-gotcha) below first. `hermes_org_connected` reads `1` for every org and channel.
* [ ] **At least one trace is visible in the backend.** Confirm a trace (or, for Argus no-span types, a log or metric) has landed in your configured OTLP backend.
* [ ] **Replay cursor is persisted.** Confirm the replay/cursor state store is actually being written to, not silently falling back to an empty or ephemeral store.
* [ ] **Pipeline config loaded as intended.** If you use a custom pipeline-config file, the `pipeline config loaded` log line reports the expected `metric_rules` count — a custom base file replaces the default, and a missing `metrics:` section means zero Iris metrics.

***

## Applying Configuration Changes

`ORG_CONFIGS` is read once at startup, so every change to it — a new org, a rotated key, a redaction flag — takes effect on the next deployment:

* **AWS (ECS):** update the secret value, then force a new deployment of the service. Changing the image is a task-definition change; see [AWS Deployment](https://docs.pharos.ai/open-telemetry/configuration/aws.md#applying-changes-upgrading-and-rolling-back).
* **Azure (Container Apps):** update the secret in Key Vault **and** create a new revision (`az containerapp update --revision-suffix …`). A plain restart re-uses the revision's already-resolved secret values; see [Azure Deployment](https://docs.pharos.ai/open-telemetry/configuration/azure.md#applying-changes-upgrading-and-rolling-back).

Pipeline-config YAML changes are picked up the same way: they are files read at startup, so a rolling restart applies them. Image upgrades follow [Versioning & Upgrades](https://docs.pharos.ai/open-telemetry/configuration/versioning-and-upgrades.md).

***

## Troubleshooting

### The JWT gotcha

By far the most common first-setup failure: Hermes authenticates but the Pub/Sub API rejects every request. This almost always means the External Client App's **"Issue JWT-based access tokens"** setting is enabled when it needs to be disabled — see the full explanation in [Salesforce Setup](https://docs.pharos.ai/open-telemetry/configuration/salesforce-setup.md#setup-steps).

### TLS / plaintext warnings

OTLP export is TLS by default. If Hermes logs a warning about a plaintext connection to a non-loopback endpoint, either your backend endpoint needs to be TLS, or — if you've deliberately pointed at a Collector service over plaintext — set `insecure: true` on that `collector` or `otlp` block to acknowledge it.

### Dedup silently disabled

The runtime reads the dedup state-store selector as `ARGUS_DEDUP_STORE` (with its `ARGUS_DEDUP_*` companions). A variable named `DEDUP_*` is ignored, and Hermes starts fine with cross-surface dedup off. If dedup doesn't seem to be doing anything, check the exact env-var names your deployment sets against [`ORG_CONFIGS` Reference](https://docs.pharos.ai/open-telemetry/configuration/org-configs.md#state-store-selectors), not just whether *some* dedup-shaped variable exists.

### Azure: rotated a secret, but Hermes still uses the old value

Azure Container Apps only re-reads Key Vault-referenced secrets when a **new revision** is created — a plain container restart reuses the current revision's already-resolved secret values. If you rotate a Salesforce private key or backend token in Key Vault and only restart the Container App, Hermes keeps authenticating (or exporting) with the stale value. Deploy a new revision instead — see [Azure Deployment](https://docs.pharos.ai/open-telemetry/configuration/azure.md#secrets-management).

### RTEM replay-window gaps

RTEM's Pub/Sub replay window is **72 hours**. If Hermes is down (or an org is unreachable) for longer than that, the events that would have replayed are gone — there's no backfill path for RTEM the way ELF has one. A gap beyond 72 hours is unrecoverable, not just delayed.

### An org is missing from `/stats`

An org whose entry fails validation is **skipped with a warning at startup** while the other orgs keep running — a missing `service_name` in a multi-org document, a backend block missing its required field, an unknown `trace_stitching.mode`, or a bad `redaction_overrides` entry. Search the startup log for `skipping org due to config error`; the message names the org and the field.

### Spans dropped or flattened

`hermes_export_dropped_total` counts spans and log records that could not be encoded — most often a Triton trace ID that is not 32 hex characters. `hermes_export_reparented_total` counts spans exported as roots because their parent span ID was unusable; the trace is flattened, not lost. Both are absent until the first occurrence, so alert on `increase() > 0`. See [Self-Instrumentation Metrics](https://docs.pharos.ai/open-telemetry/configuration/self-metrics.md).

***

## Related

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>📡 Self-Instrumentation Metrics</strong></td><td>Connectivity, auth-failure, drop, and lag metrics with alerting guidance</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/self-metrics.md">Self-Instrumentation Metrics</a></td></tr><tr><td><strong>🔔 Dashboards &#x26; Alerts</strong></td><td>The shipped dashboard suite and alert set</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/dashboards-and-alerts.md">Dashboards &amp; Alerts</a></td></tr><tr><td><strong>🟠 AWS Deployment</strong></td><td>ECS Fargate runbook, including verification and rollback commands</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/aws.md">AWS Deployment</a></td></tr><tr><td><strong>🔵 Azure Deployment</strong></td><td>Container Apps runbook, including verification and rollback commands</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/azure.md">Azure Deployment</a></td></tr></tbody></table>
