> For the complete documentation index, see [llms.txt](https://docs.pharos.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pharos.ai/open-telemetry/configuration/dashboards-and-alerts.md).

# Dashboards & Alerts

The Hermes dashboard suite and alert set — what each dashboard answers, how they ship per backend, and which alerts watch per-org health.

Hermes ships one dashboard suite and one alert set, ported to every supported backend. The suite is built entirely on metric families grouped by labels — no panel keys on a span name — so a mapping change that renames a span never breaks a dashboard.

***

## The Dashboard Suite

| Dashboard                      | Answers                                                                                                                                                                                                                                                                         | Source and lag                              |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- |
| **Hermes Overview**            | Fleet landing page: is every org connected and authenticating; what is happening right now (RTEM logins, API calls, navigation, threats); what is loading the backend (ELF top-N); links to every other dashboard with the template variables carried along                     | Self-metrics + RTEM (seconds) + ELF (hours) |
| **Argus Security**             | Authentication, API and data access, browser activity, and the threat-detection channels near-real-time, plus the security-relevant EventLogFile types — InsufficientAccess, PlatformEncryption, Login, Logout, ChangesetOperation                                              | RTEM (seconds) + ELF (hours to a day)       |
| **Argus Performance**          | Where the org spends server time: Apex execution, CPU and DB time, triggers, SOAP and REST API, Aura and Visualforce, GraphQL, async and callout latency — which entry points, triggers, pages, and callouts are slow                                                           | ELF only (hours to a day)                   |
| **Argus Adoption & Analytics** | Who uses the org and how it performs for them: Lightning navigation and page time, reports and dashboards, content, knowledge, and CRM Analytics activity                                                                                                                       | RTEM first row (seconds), ELF below (hours) |
| **Argus Data & Integrations**  | Who moves data, how much, and at what failure rate: Bulk API record counts and job outcomes, API data volume by type, operation, and status, record- and event-level activity                                                                                                   | RTEM + ELF                                  |
| **Iris Apex Runtime**          | The Triton application-telemetry families — Apex governor limits as percent-of-limit gauge pairs, Flow, Integration, Triton errors, and LWC browser vitals                                                                                                                      | Iris (near-real-time)                       |
| **Hermes Health**              | Hermes itself, no customer telemetry: per-org connectivity and auth failures (the two metrics behind the production alerts), ELF lag by event type and interval, pending files and bytes, export queue depth, interval-dedup suppression, dedup fall-open errors, poll coverage | Self-metrics                                |
| **Metric Coverage (QA)**       | Of every metric family Hermes is configured to emit, which ones exist in the store right now — expected counts derived from the two pipeline configs versus observed families over 24 h and 7 d                                                                                 | Self-check                                  |

Every dashboard carries an `$org` template variable and filters on `service_name`; the Health dashboard's `hermes_org_*` panels aggregate by `org_id` and `channel` instead.

***

## How the Suite Ships

| Backend                            | Delivery                                                                                                                            |
| ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| **Grafana** (Cloud or self-hosted) | Reference implementation — all eight dashboards as provisioning JSON                                                                |
| **Splunk Observability Cloud**     | Terraform module creating the "Hermes Suite" dashboard group with metric dashboards and charts, plus the optional org-down detector |
| **Honeycomb**                      | Terraform module creating seven metrics boards plus Iris, RTEM, and ELF boards and saved queries                                    |
| **Datadog**                        | Terraform module creating all eight dashboards                                                                                      |

Metrics reach Splunk, Datadog, and Grafana through a Collector; a Splunk **direct** deployment carries traces only and therefore has no metrics for these dashboards.

***

## Production Alerts

Two per-org health self-metrics drive the production alerts — identical on AWS and Azure and in every backend that receives metrics. Their meaning and caveats are on [Self-Instrumentation Metrics](https://docs.pharos.ai/open-telemetry/configuration/self-metrics.md).

**Primary — org disconnected.** `hermes_org_connected == 0` sustained for **10 minutes**, per `(org_id, channel)`. The 10-minute window matters: the two stream runtimes legitimately read `0` through every 30-second reconnect, and a rolling deploy or a lease hand-over reads `0` briefly on all of them. Importable definitions ship for Splunk (detector JSON) and Honeycomb (trigger JSON).

**Secondary — auth failures.** `increase(hermes_org_auth_failures_total[15m]) > 0`. Read together with the primary: connected `0` and failures `> 0` means re-issue the credential; connected `0` and failures `0` means transport, lease, or an unlicensed topic; connected `1` and failures `> 0` means rejections that recovered — a rotation race or a flapping credential.

{% hint style="warning" %}
**Splunk direct mode receives neither metric**, so neither alert can fire, and an alert on missing data fires forever. Move the org to a `collector` block for per-org health alerting.
{% endhint %}

***

## Alert Candidates

PromQL as written for Grafana; the same conditions port to any backend that receives the metrics. `F` is the standard filter `service_name=~"$org"`.

| Condition                        | PromQL                                                                                                                                    | Threshold       | Severity                                   |
| -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | --------------- | ------------------------------------------ |
| Org disconnected                 | `hermes_org_connected == 0`                                                                                                               | sustained 10 m  | Critical                                   |
| Any auth failure                 | `increase(hermes_org_auth_failures_total[1h]) > 0`                                                                                        | `> 0`           | Warning                                    |
| Login-failure spike              | `sum(increase(argus_login_failure_count_total{F}[1h])) > 10`                                                                              | `> 10` per hour | Warning; Critical if sustained two windows |
| Insufficient-access denial burst | `increase(argus_elf_security_insufficient_access_count_total{F}[24h]) > 0`                                                                | `> 0`           | Warning                                    |
| Governor usage near limit        | `100 * max_over_time(apex_governor_<res>{F}[1h]) / max_over_time(apex_governor_<res>_limit{F}[1h]) > 80` (each of the governor resources) | `> 80 %`        | Warning; Critical at `> 95`                |
| Bulk API record failures         | `increase(argus_elf_bulkapi_records_failed_total{F}[24h]) > 0`                                                                            | `> 0`           | Warning                                    |
| RTEM threat event detected       | `increase(argus_rtem_threat_count_total{F}[1h]) > 0`                                                                                      | `> 0`           | Critical                                   |
| ELF lag high                     | `max(elf_lag_seconds{F}) > 600`                                                                                                           | 5 m             | Warning                                    |
| Export drops                     | `increase(hermes_export_dropped_total[1h]) > 0`                                                                                           | `> 0`           | Warning                                    |

The problem counters (`hermes_export_dropped_total`, `hermes_export_reparented_total`, `dedup_miss_due_to_late_rtem`) have no series until the first occurrence — alert on `increase() > 0`, never on `absent()`.

***

## Cloud Log-Based Bridges

Where a backend does not receive metrics, the same conditions can be approximated from container logs in the cloud's own monitoring — the AWS and Azure Terraform alert stages provision the infrastructure-level versions (see the [AWS](https://docs.pharos.ai/open-telemetry/configuration/aws.md#monitoring) and [Azure](https://docs.pharos.ai/open-telemetry/configuration/azure.md#monitoring) runbooks).

* **Azure — Log Analytics scheduled query:** over `ContainerAppConsoleLogs_CL` for the Hermes container, match `invalid_client_id`, `credential rejected by the org`, `hermes: subscription down`, or `exited with error; restarting`; fire when rows appear in a 15-minute window.
* **AWS — CloudWatch Logs metric filter + alarm:** filter pattern `?"invalid_client_id" ?"credential rejected by the org" ?"subscription down" ?"exited with error; restarting"` on the Hermes log group, then an alarm on `Sum >= 1` over three 5-minute periods.

***

## Related

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>📡 Self-Instrumentation Metrics</strong></td><td>The metrics behind the Health dashboard and the alerts</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/self-metrics.md">Self-Instrumentation Metrics</a></td></tr><tr><td><strong>🎯 Backend Configuration</strong></td><td>Per-backend runbooks, including how each imports the suite</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/backends.md">Backend Configuration</a></td></tr><tr><td><strong>🩺 Operations &#x26; Validation</strong></td><td>Health endpoints and troubleshooting</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/operations.md">Operations &amp; Validation</a></td></tr></tbody></table>
