> For the complete documentation index, see [llms.txt](https://docs.pharos.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pharos.ai/open-telemetry/configuration/backends/honeycomb.md).

# Honeycomb

Operating Hermes telemetry in Honeycomb — the two API keys, dataset strategy, data model, queries, the shipped boards, and troubleshooting.

Honeycomb receives all three Hermes signals directly over OTLP/gRPC with a `honeycomb` block, and is equally reachable through a Collector. Direct mode is the simplest path and the recommended first deployment; choose Collector mode when you already run an OpenTelemetry gateway, must keep the ingest key out of Hermes configuration, or need sampling and filtering before data reaches Honeycomb.

***

## Setup: Two API Keys

Hermes work in Honeycomb uses two keys with distinct roles. Create both in the Honeycomb UI under the environment you will export to.

| Key                   | Prefix   | Used by                                                            | Permissions                                     | Where it lives                                                                                      |
| --------------------- | -------- | ------------------------------------------------------------------ | ----------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| **Ingest key**        | `hcaik_` | Hermes (or the Collector) to send telemetry                        | Send events, create datasets                    | `honeycomb.api_key` in `ORG_CONFIGS` (Secrets Manager / Key Vault), or the Collector's secret store |
| **Configuration key** | `hcalk_` | Terraform, to provision boards, saved queries, and derived columns | Manage boards, queries, columns, triggers, SLOs | Your Terraform runner's secret store — never in `ORG_CONFIGS`, never in the image                   |

Rotate the ingest key by creating a new key, updating the secret, rolling a new deployment or revision, then deleting the old key.

***

## Wiring

Direct mode — the minimal top-level block:

```json
{
  "honeycomb": {
    "api_key": "hcaik_REPLACE_WITH_INGEST_KEY",
    "dataset": "hermes-production"
  },
  "signals": "traces,metrics,logs"
}
```

Hermes derives the endpoint (`https://api.honeycomb.io`) and the `x-honeycomb-team` / `x-honeycomb-dataset` headers. Set `"endpoint": "https://api.eu1.honeycomb.io"` for an EU environment. Drop `dataset` and each org's dataset becomes its own `service_name`. Extra `headers` on the block are merged in and win over the derived ones — useful for an authenticating proxy.

Collector mode — Hermes sends to the sidecar, the Collector's OTLP/gRPC exporter carries the ingest key:

```json
{ "collector": {}, "signals": "traces,metrics,logs" }
```

**Expect two datasets.** Honeycomb stores OTLP metrics in a dataset named by the metrics' own routing, so a `metrics` dataset appears beside the one you configured. It is not a leftover — do not delete it.

***

## Dataset Strategy

* `honeycomb.dataset`, when set, is both the routing dataset and the `service.name` resource attribute; the org's original `service_name` is preserved as `service.display_name` for filtering within the dataset. When it is not set, the dataset falls back to the org's `service_name`, then to `hermes`.
* For a **combined Iris + Argus deployment, route every surface to one environment dataset** (for example `hermes-prod`, `hermes-sandbox`) so cross-channel traces stitch — spans for one request share a `trace_id` and only stitch within a single dataset. Distinguish surfaces with `service.name` or `hermes.channel` as a filter, not as the routing dataset.
* Multiple production orgs in one dataset is the recommended layout; partition queries by `salesforce.org_id`.

***

## Data Model

| Attribute                                                | Notes                                                                                                                                                                    |
| -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `service.name`                                           | Surface identity (`hermes-iris`, `hermes-argus-rtem`, `hermes-argus-elf`) or per-org `service_name` with the surface on `hermes.channel` — one convention per deployment |
| `salesforce.org_id` (Iris) / `salesforce.org.id` (Argus) | Span-level org identifier; use `WHERE salesforce.org_id = "00D…" OR salesforce.org.id = "00D…"` across channels                                                          |
| `salesforce.bridge.mapping_version`                      | Resource attribute — which pipeline config produced the data                                                                                                             |
| `trace.trace_id`, `trace.parent_id`                      | Deterministic — the same Salesforce session or request yields the same trace on every channel                                                                            |
| `duration_ms`                                            | Span duration in query context                                                                                                                                           |

Pipeline-config metric rules of `type: histogram` reach Honeycomb as one gauge value per event rather than bucketed histograms; heatmaps are built from those points. `export_latency_ms`, a Hermes self-metric, is a real histogram whose `_bucket` series support percentiles. ELF timestamps (`salesforce.elf.timestamp`) reflect the time recorded in the EventLogFile and lag real time by up to the poll interval; RTEM timestamps are real-time.

***

## Queries

Useful starting points, all filtered by `service.name` (or `hermes.channel`) and `salesforce.org_id`:

* **Health** — `COUNT` of spans by `service.name` over time; a surface that drops to zero is a stream that closed. `hermes_org_connected` and `hermes_org_auth_failures_total` arrive as metrics when `hermes.self_metrics` is on.
* **Apex performance** — `HEATMAP(duration_ms)` and `P95(duration_ms)` where `salesforce.technology = "Apex"`, grouped by `name`; governor consumption from `salesforce.limits.cpu_time` against `salesforce.limits.cpu_time.limit`.
* **Login and API monitoring** — `COUNT` grouped by `salesforce.login.status` / `salesforce.api.operation` on the RTEM surface.
* **Threat detection** — log records where `event.name` starts with `salesforce.security` or `salesforce.threat`, grouped by `salesforce.threat.type`.
* **Session timeline** — filter `session.id = "<id>"` and open the trace: session → transactions → requests → work, across Iris and Argus.
* **Cost / volume** — `COUNT` grouped by `service.name` and `salesforce.org_id` per day shows which orgs and surfaces drive event volume.

***

## Boards, Triggers, and SLOs

A Terraform pack provisions the reference boards and saved queries with the Configuration key: an Iris board (session, transaction, and Apex performance), an RTEM board (authentication, API, threats), an ELF board (EventLogFile lag and volume), and seven metrics boards mirroring the shared Hermes dashboard suite (Overview, Argus Security, Argus Performance, Argus Adoption, Argus Data, Iris Apex Runtime, Hermes Health). Recommended trigger conditions — org disconnected for 10 minutes, any auth failure, login-failure spike, threat event detected — and SLO definitions for ingestion availability, latency, error rate, and login success accompany the pack; the org-down trigger is also importable as JSON. See [Dashboards & Alerts](https://docs.pharos.ai/open-telemetry/configuration/dashboards-and-alerts.md).

***

## Troubleshooting

| Symptom                                                              | Check                                                                                                                                                                                                     |
| -------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| No spans appearing                                                   | `/health` 200 and the startup log show `SF access token refreshed`; the ingest key is the `hcaik_` key, not the configuration key; the dataset filter matches `honeycomb.dataset`; the org emits activity |
| 401 / 403 from Honeycomb                                             | Wrong key type, revoked key, or a key from a different environment                                                                                                                                        |
| Data lands in a new or wrong dataset                                 | `honeycomb.dataset` changed, or an org's `service_name` changed with `dataset` unset — datasets are created on first write                                                                                |
| EU vs. US                                                            | An EU environment needs `honeycomb.endpoint` set to the EU host; the default is US                                                                                                                        |
| Some orgs missing                                                    | Check `/stats` and the startup log for `skipping org due to config error`; in a multi-org document every org needs `service_name`                                                                         |
| `trace.trace_id` differs between Iris and Argus for the same request | Session unification is off for that org and trace stitching is not enabled — turn one of them on                                                                                                          |
| Unexpected `salesforce.bridge.mapping_version`                       | An override file or a new image changed the mapping; compare with the `pipeline config loaded` log line                                                                                                   |
| ELF lag high for one event type                                      | That type's files are large or the Hourly loop is off; watch `elf_lag_seconds{event_type}` and raise `hermes.elf.parallelism`                                                                             |
| Threat events missing                                                | The RTEM channel is not in `hermes.rtem.channels`, or the org lacks the Event Monitoring threat-detection add-on                                                                                          |
| Dedup suppressing expected spans                                     | Check the family's `dedup.mode`; `surface-select` keeps only the authoritative surface by design                                                                                                          |

***

## Related

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>🎯 Backend Configuration</strong></td><td>Block selection, endpoint rules, Collector wiring</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/backends.md">Backend Configuration</a></td></tr><tr><td><strong>🔔 Dashboards &#x26; Alerts</strong></td><td>The dashboard suite and alert set shipped for every backend</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/dashboards-and-alerts.md">Dashboards &amp; Alerts</a></td></tr><tr><td><strong>📡 Self-Instrumentation Metrics</strong></td><td>The Hermes health metrics that arrive alongside your telemetry</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/self-metrics.md">Self-Instrumentation Metrics</a></td></tr></tbody></table>
