> For the complete documentation index, see [llms.txt](https://docs.pharos.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pharos.ai/open-telemetry/configuration/self-metrics.md).

# Self-Instrumentation Metrics

The metrics Hermes reports about itself — connectivity, export drops, leadership, ELF lag, dedup health — how they reach your backend, and what to alert on.

Alongside the Salesforce telemetry it converts, Hermes reports on its own health: whether each org is reachable, whether a credential was rejected, whether anything was dropped at export, and how far behind the ELF pipeline is. These self-metrics are the basis of the operational alerts in [Dashboards & Alerts](https://docs.pharos.ai/open-telemetry/configuration/dashboards-and-alerts.md).

***

## How They Are Emitted

* Every runtime serves them in Prometheus format on its `/metrics` endpoint.
* When `hermes.self_metrics` is present in an org's [`ORG_CONFIGS`](https://docs.pharos.ai/open-telemetry/configuration/org-configs.md) block (even `{}`), the runtime also **pushes** them to that org's OTLP backend every `interval_seconds` (default 30), so they land next to the customer telemetry — the same on AWS and Azure, Honeycomb and Splunk.
* Metric names are identical in Prometheus and OTLP form (there is no dotted mapping-rule name).
* Counters that record a problem (`hermes_export_dropped_total`, `hermes_export_reparented_total`, `dedup_miss_due_to_late_rtem`) appear only on the first occurrence — **absent is the healthy state**, so alert on `increase(...) > 0`, never on `absent()`.

{% hint style="warning" %}
**Splunk direct mode receives no self-metrics.** The direct OTLP/gRPC path accepts traces only, so `signals` resolves to `traces` and the metrics are never sent — an alert on `hermes_org_connected` cannot fire, and an alert on missing data fires forever. If you are on Splunk direct and want per-org health alerts, route that org through a `collector` block. ELF **backfill** runs export no self-metrics either; their failures surface only in logs.
{% endhint %}

***

## Catalogue

### Per-org health

| Metric                           | Type    | Labels              | Meaning and use                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| -------------------------------- | ------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `hermes_org_connected`           | gauge   | `org_id`, `channel` | `1` = Hermes can currently reach this org with its credential. For Iris and RTEM that means a Pub/Sub stream is open (it reads `0` through every 30-second reconnect); for ELF live, the most recent poll authenticated. On every runtime it is also pinned to `0` after **two consecutive** credential rejections and un-pinned by the next accepted call — one rejection alone is the signature of a benign token-rotation race. **Primary alert: `== 0` sustained for 10 minutes.** |
| `hermes_org_auth_failures_total` | counter | `org_id`, `channel` | Salesforce refused a credential — at startup (an invalid consumer key, an expired certificate, a dead pasted token) or in operation (gRPC `UNAUTHENTICATED` on a stream, HTTP 401 or `INVALID_SESSION_ID` on an ELF poll). Fires once per credential presentation, in every auth mode. **Secondary alert: `increase(...[15m]) > 0`.**                                                                                                                                                  |

`channel` is `iris`, `argus-elf`, or the RTEM topic name. Read the two together:

| `hermes_org_connected` | `increase(hermes_org_auth_failures_total[15m])` | Reading                                                                            |
| ---------------------- | ----------------------------------------------- | ---------------------------------------------------------------------------------- |
| `0` sustained          | `> 0`                                           | The org is refusing the credential — re-issue it                                   |
| `0` sustained          | `0`                                             | Transport, lease, or an unlicensed topic — not the credential                      |
| `1`                    | `> 0`                                           | Rejections that recovered; a rotation race or a flapping credential worth watching |

What these two do **not** detect: transport failures (`UNAVAILABLE`, timeouts, DNS — a network blip cannot masquerade as a dead credential; on ELF use `elf_lag_seconds` instead), a 403 without `INVALID_SESSION_ID` (Salesforce uses 403 for API limits and field-level security too), and a live credential that lost object access (the queries simply return nothing). Neither feeds `/health` or `/ready` — failing readiness for one org's credential would restart the container and take down the orgs that were fine.

### Export integrity

| Metric                           | Type    | Labels                                                                                                                                       | Meaning and use                                                                                                                                                                                                                                                                                                                           |
| -------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `hermes_export_dropped_total`    | counter | `org_id`, `channel`, `signal` (`span` \| `log_record`), `reason` (`trace_id` \| `span_id` \| `link_trace_id` \| `link_span_id` \| `unknown`) | A span or log record could not be encoded for OTLP and was dropped — most often a Triton trace ID that is not 32 hex characters. Fix the instrumentation in the org; alert on `increase(...) > 0`.                                                                                                                                        |
| `hermes_export_reparented_total` | counter | `org_id`, `channel`, `reason` (`parent_span_id` \| `unknown`)                                                                                | A span's parent ID was unusable, so it was exported as a root. **Not data loss** — the trace is flattened, not missing. Investigate a sustained rate rather than paging on the first sample.                                                                                                                                              |
| `session_epoch_cuts_total`       | counter | `org_id`                                                                                                                                     | A session trace crossed `max_spans_per_trace` and a new trace epoch was started (written by Iris only). A steadily rising rate on one org means its sessions are span-heavy — consider a time segment (`HERMES_SESSION_TRACE_SEGMENT`) so cuts land on predictable boundaries, or raise the cap if your backend's per-trace limit allows. |
| `session_root_claims_total`      | counter | `org_id`, `channel` (`iris` \| `argus-rtem` \| `argus-elf`), `outcome` (`granted` \| `suppressed` \| `fail_open`)                            | Which runtime won the claim to emit a trace's session root. `fail_open` means a trace-map store error let the span through and duplicate session roots are possible — alert on a sustained rate.                                                                                                                                          |

### Stream ownership

| Metric                          | Type  | Labels             | Meaning and use                                                                                                                                                                                                                                                                       |
| ------------------------------- | ----- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `hermes_stream_lease_is_leader` | gauge | `org_id`, `stream` | `1` while this process holds the stream-ownership lease for an Iris topic or RTEM channel, `0` otherwise (including while waiting to acquire). Only meaningful when `STREAM_LEASE_STORE` is configured; during a deploy expect a brief hand-over from the old replica to the new one. |

### Argus ELF pipeline

| Metric                                      | Type      | Labels                                                   | Meaning and use                                                                                                                                                                                                                                    |
| ------------------------------------------- | --------- | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `elf_lag_seconds`                           | gauge     | `org_id`, `event_type`, `interval` (`daily` \| `hourly`) | How far behind the live cursor is from the newest available file. Published to the cloud autoscaler and lag alarm as **`argus_elf_lag_seconds`**; the autoscaler targets ≈60 s (`scaled`) or ≈300 s (`standard`). Alert above 600 s for 5 minutes. |
| `elf_lag_refreshed_at_seconds`              | gauge     | `org_id`, `event_type`, `interval`                       | Unix time the lag series was last refreshed — diagnostic only, not a lag value; the cloud publisher uses it to exclude a series whose poll loop stopped from the autoscaling maximum. Do not chart it as lag.                                      |
| `elf_pending_files`, `elf_pending_bytes`    | gauge     | `org_id`, `event_type`, `interval`                       | Files discovered but not yet processed, and their total size — the backlog behind the lag.                                                                                                                                                         |
| `queue_depth`                               | gauge     | —                                                        | Global export-queue depth shared by all orgs; a rising value means the backend or Collector is the bottleneck.                                                                                                                                     |
| `export_latency_ms`                         | histogram | `org_id`, `event_type`                                   | Exporter call latency. Over the OTLP bridge the buckets arrive as `_bucket` / `_count` / `_sum` series — reconstruct percentiles from `_bucket`, not with native-histogram functions.                                                              |
| `argus_elf_interval_dedup_suppressed_total` | counter   | `org_id`, `event_type`                                   | Daily-file rows suppressed because the same row was already emitted from an Hourly file. Expected to grow steadily when the Hourly loop is on.                                                                                                     |
| `argus_elf_cross_surface_suppressed_total`  | counter   | `org_id`, `event_type`, `mode`                           | ELF rows suppressed or supplemented by RTEM ↔ ELF dedup. Live path only — backfill always emits its own signal.                                                                                                                                    |

### Dedup store

| Metric                                               | Type    | Labels                    | Meaning and use                                                                                                  |
| ---------------------------------------------------- | ------- | ------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| `dedup_cache_hit_rate`                               | gauge   | `backend`, `org_id`       | Rolling one-minute hit ratio (0–1).                                                                              |
| `dedup_cache_size`, `dedup_cache_evictions_per_hour` | gauge   | `backend`, `org_id`       | In-memory backend only; absent with DynamoDB or Table Storage.                                                   |
| `dedup_backend_errors_total`                         | counter | `backend`, `org_id`, `op` | Store errors absorbed by the fall-open policy — duplicates, never loss. Alert above 10 per minute.               |
| `dedup_miss_due_to_late_rtem`                        | counter | `backend`, `org_id`       | The ELF row arrived before its RTEM fingerprint, so it could not be suppressed. Series appear on the first miss. |

Business metrics derived from Salesforce events — `argus.login.count`, `argus_elf_api_rest_call_count_total`, the `apex_*` governor gauges, and the rest of the mapping-derived families — are a different layer, catalogued with the dashboards that chart them in [Dashboards & Alerts](https://docs.pharos.ai/open-telemetry/configuration/dashboards-and-alerts.md).

***

## Related

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>🔔 Dashboards &#x26; Alerts</strong></td><td>The Hermes Health dashboard and the alert set built on these metrics</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/dashboards-and-alerts.md">Dashboards &amp; Alerts</a></td></tr><tr><td><strong>🩺 Operations &#x26; Validation</strong></td><td>Health endpoints and troubleshooting</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/operations.md">Operations &amp; Validation</a></td></tr><tr><td><strong>🧩 ORG_CONFIGS Reference</strong></td><td>hermes.self_metrics and the state-store selectors</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/org-configs.md">ORG_CONFIGS Reference</a></td></tr></tbody></table>
