> For the complete documentation index, see [llms.txt](https://docs.pharos.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pharos.ai/open-telemetry/configuration/backends/grafana.md).

# Grafana Cloud & LGTM

Operating Hermes telemetry in Grafana Cloud or a self-hosted LGTM stack — wiring, data model, queries, the reference dashboards, and troubleshooting.

One runbook covers both **Grafana Cloud** and a **self-hosted LGTM** stack (Loki, Grafana, Tempo, Mimir). The Hermes side is identical — a `collector` block — and the query layer (TraceQL, PromQL, LogQL) is the same; only the egress side of the Alloy or Collector differs. Grafana Cloud's OTLP gateway is HTTP-only while Hermes exports gRPC, which is why the Cloud track also goes through a Collector.

***

## Wiring (Both Tracks)

```json
{
  "collector": {},
  "signals": "traces,metrics,logs"
}
```

A bare `{}` resolves to the sidecar at `http://127.0.0.1:4317`. Point `collector.endpoint` at a separate Alloy/Collector service (with `insecure: true` on a private network) when it does not run as a sidecar. There is no `grafana` block in the export schema.

***

## Track A — Grafana Cloud

1. **Create or open a stack** and note its zone.
2. **Find the OTLP gateway endpoint and instance ID** on the stack's OpenTelemetry configuration page.
3. **Create a Cloud Access Policy token** scoped to `metrics:write`, `logs:write`, and `traces:write`.
4. **Build the auth header** — `Authorization: Basic base64(<instanceID>:<token>)` — and place it in the Alloy/Collector secret store, not in `ORG_CONFIGS`.
5. **Configure egress** — an `otlphttp` exporter to the gateway (`/otlp/v1/traces`, `/otlp/v1/metrics`, `/otlp/v1/logs`) carrying that header, fed by the OTLP/gRPC receiver Hermes sends to.

***

## Track B — Self-Hosted LGTM

1. **Bring up** Tempo, Mimir, Loki, Grafana, and Alloy (or a Collector).
2. **Wire Grafana data sources** for the three stores.
3. **Configure the fan-out** — OTLP/gRPC receiver → `otlp` exporter to Tempo (traces), `prometheus.remote_write` to Mimir (metrics), `loki.write` to Loki (logs).

Two settings on the metrics path are mandatory for Hermes:

* **`deltatocumulative` processor** before the Prometheus-facing exporter — Hermes counters use delta temporality and Mimir/Prometheus silently drop delta sums otherwise. Gauges, spans, and logs are unaffected.
* **`out_of_order_time_window` of 72h or more** on Mimir — Argus ELF metrics carry historical event timestamps, hours to a day old, and the default window rejects them silently. RTEM metrics are real-time and unaffected.

Grafana Cloud Metrics needs the same `deltatocumulative` conversion; raise the out-of-order window in your stack's settings if ELF metrics are missing.

***

## Data Model

| Signal          | Where `service.name` lands                                                                                                |
| --------------- | ------------------------------------------------------------------------------------------------------------------------- |
| Traces (Tempo)  | Resource attribute — `resource.service.name` in TraceQL                                                                   |
| Metrics (Mimir) | Translated to labels on ingestion — commonly `service_name` (and a `job` label), depending on the processor configuration |
| Logs (Loki)     | Translated to a label — commonly `service_name`                                                                           |

Verify the exact label names against your stack on first ingest rather than assuming they match Tempo's attribute name. Org identity is `salesforce.org_id` on Iris spans and `salesforce.org.id` on Argus spans (`span.salesforce.org_id` in TraceQL); select a surface with `resource.service.name="hermes-argus-elf"` or, in an org-named deployment, `resource.hermes.channel="argus-elf"`.

***

## Queries

* **Traces (TraceQL)** — `{ resource.service.name = "hermes-iris" && span.salesforce.technology = "Apex" && duration > 2s }`; one user's session: `{ span.session.id = "<id>" }`.
* **Metrics (PromQL)** — health: `hermes_org_connected == 0`, `increase(hermes_org_auth_failures_total[1h]) > 0`; Iris performance: `quantile_over_time(0.95, triton_operation_duration{salesforce_technology="Apex"}[5m])` — mapping-derived `histogram` rules reach the wire as one gauge datapoint per event, so quantiles are computed over the samples rather than from `_bucket` series; Argus: `sum(increase(argus_login_failure_count_total[1h]))`, `increase(argus_rtem_threat_count_total[1h]) > 0`.
* **Logs (LogQL)** — `{service_name="hermes-argus-rtem"} | json | event_name =~ "salesforce.security.*"`.

***

## Dashboards

Grafana is the reference implementation of the Hermes dashboard suite: the eight dashboards ship as provisioning JSON — Overview, Argus Security, Argus Performance, Argus Adoption & Analytics, Argus Data & Integrations, Iris Apex Runtime, Hermes Health, and Metric Coverage (QA). Every panel is a PromQL query grouped by labels with an `$org` template variable and a `service_name` filter, so the suite is independent of span names. See [Dashboards & Alerts](https://docs.pharos.ai/open-telemetry/configuration/dashboards-and-alerts.md); the alert-candidate table there is written in PromQL and applies directly as Grafana alert rules.

***

## Troubleshooting

| Symptom                                                 | Check                                                                                                                                                   |
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Cloud: 401 / wrong zone                                 | The basic-auth value is `instanceID:token`, the token has all three write scopes, and the gateway endpoint matches the stack's zone                     |
| Self-hosted: nothing in any store                       | The Alloy/Collector OTLP receiver is reachable from Hermes at the configured endpoint; Hermes log shows no exporter errors                              |
| Traces in Tempo but no metrics in Mimir or logs in Loki | The fan-out routes each signal; `deltatocumulative` is on the metrics path; the out-of-order window is raised for ELF metrics                           |
| Metric names not found in the picker                    | Counters end in `_total`; label translation turned dots into underscores (`salesforce_org_id`); delta sums were dropped for want of `deltatocumulative` |
| Hermes-side checks                                      | `/stats` lists the org, no `skipping org` log line, `hermes.self_metrics` present if you expect health metrics                                          |
| Per-signal diagnosis                                    | In Explore, query each store separately with the same time range and `service_name` to find which hop drops the signal                                  |

***

## Related

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>🎯 Backend Configuration</strong></td><td>Block selection, endpoint rules, Collector wiring</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/backends.md">Backend Configuration</a></td></tr><tr><td><strong>🎯 Backends &#x26; Routing</strong></td><td>Metric temporality and the out-of-order window explained</td><td><a href="https://docs.pharos.ai/open-telemetry/hermes/backends.md">Backends &amp; Routing</a></td></tr><tr><td><strong>🔔 Dashboards &#x26; Alerts</strong></td><td>The reference dashboard suite and PromQL alert candidates</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/dashboards-and-alerts.md">Dashboards &amp; Alerts</a></td></tr></tbody></table>
