> For the complete documentation index, see [llms.txt](https://docs.pharos.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pharos.ai/open-telemetry/configuration/backends/datadog.md).

# Datadog

Operating Hermes telemetry in Datadog — the Collector route and why, wiring, facets, the span-age limit, the shipped dashboards, and troubleshooting.

Datadog's agentless OTLP intake is HTTP-only and Hermes exports OTLP/gRPC, so every signal reaches Datadog **through an OpenTelemetry Collector** running the `datadog` exporter. Traces land in APM, metrics in Metrics, logs in Logs. Datadog accepts delta-temporality sums natively, so no temporality conversion is needed.

***

## Setup

1. **Create an API key** in Datadog under Organization Settings → API Keys. The key is site-scoped.
2. **Pick your site** — it determines the intake endpoint the Collector uses:

   | Site                | Region  |
   | ------------------- | ------- |
   | `datadoghq.com`     | US1     |
   | `datadoghq.eu`      | EU1     |
   | `us3.datadoghq.com` | US3     |
   | `us5.datadoghq.com` | US5     |
   | `ap1.datadoghq.com` | AP1     |
   | `ddog-gov.com`      | US1-FED |
3. **Wire `ORG_CONFIGS`** with a `collector` block and **deploy the Collector** with the `datadog` exporter configured with your `api.key` and `site`.
4. **Set a session trace window.** Datadog rejects spans older than 18 hours, and an unsegmented session trace can span a workday; set `HERMES_SESSION_TRACE_SEGMENT` (for example `6h`) identically on every runtime. See [Cross-Surface Correlation](https://docs.pharos.ai/open-telemetry/hermes/correlation.md#session-trace-segmentation-and-span-caps).

***

## Wiring

```json
{
  "collector": {},
  "signals": "traces,metrics,logs"
}
```

`{}` is complete: the endpoint derives as `http://127.0.0.1:4317`, the sidecar address. The Datadog API key is never a Hermes field — it belongs to the Collector's `datadog` exporter. You may place it as an extra key in the same `collector` object for the deploy tooling to hand to the sidecar; Hermes reads only `endpoint`, `headers`, and `insecure` and ignores the rest.

A minimal Collector pipeline: an `otlp` receiver on `4317`; `batch` processor; the `datadog` exporter with `api.key` and `api.site` for traces, metrics, and logs.

***

## Data Model

| Attribute                                                   | In Datadog                                                                                                                         |
| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `service.name`                                              | The `service` tag — `hermes-iris`, `hermes-argus-rtem`, `hermes-argus-elf`, or per-org names with the surface on `@hermes.channel` |
| `salesforce.org_id` (Iris) / `salesforce.org.id` (Argus)    | Span attributes — **create facets** for both before they are searchable in the APM UI                                              |
| `session.id`, `salesforce.request.id`, `salesforce.user.id` | Worth faceting for session and request drill-down                                                                                  |
| `trace_id`                                                  | Shared across Iris and Argus for one session or request                                                                            |

***

## Queries

* **Health** — `hermes_org_connected` and `hermes_org_auth_failures_total` as metrics (with `hermes.self_metrics` on); APM traffic per `service` in the service list.
* **Iris performance** — APM latency distribution for `service:hermes-iris` filtered to `@salesforce.technology:Apex`, grouped by resource name; governor consumption from the `apex_*` metric families.
* **Argus monitoring** — `argus_login_failure_count_total`, `argus_rtem_threat_count_total`, and the `argus_elf_*` families as timeseries; log search on `@event.name:salesforce.security.*` for security logs.
* **Session view** — filter APM by `@session.id:<id>` to see one user's session across channels.

Substitute `@hermes.channel:iris` for `service:hermes-iris` in an org-named deployment.

***

## Dashboards

A Terraform module provisions the eight Hermes dashboards in Datadog — Overview, Argus Security, Argus Performance, Argus Adoption, Argus Data, Iris Apex Runtime, Hermes Health, and Metric Coverage — mirroring the shared suite described in [Dashboards & Alerts](https://docs.pharos.ai/open-telemetry/configuration/dashboards-and-alerts.md). Build the org-down monitor on `hermes_org_connected` staying `0` for 10 minutes.

***

## Troubleshooting

| Symptom                                         | Check                                                                                                                                                                                                                                                                       |
| ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Nothing arrives                                 | `/health` 200 and `SF access token refreshed` in the Hermes log; the Collector's `datadog` exporter log for `403` (API key); the `site` matches the key's organization; `signals` includes the signal you expect; the Collector pipeline routes that signal to the exporter |
| Traces arrive but attributes are not searchable | Create facets for `salesforce.org_id`, `salesforce.org.id`, and any other span attribute you filter on                                                                                                                                                                      |
| Metrics are missing                             | The Collector's metrics pipeline must include the `datadog` exporter; `hermes.self_metrics` must be present for Hermes health metrics                                                                                                                                       |
| Spans rejected as too old                       | Set `HERMES_SESSION_TRACE_SEGMENT` under 18 hours; ELF spans carry historical event times and can also exceed the window when Daily files land late                                                                                                                         |
| Hermes-side checks                              | Before assuming a Datadog problem: `/stats` lists the org, no `skipping org` log line, the Collector is reachable at the configured endpoint                                                                                                                                |

***

## Related

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>🎯 Backend Configuration</strong></td><td>Block selection, endpoint rules, Collector wiring</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/backends.md">Backend Configuration</a></td></tr><tr><td><strong>🔗 Cross-Surface Correlation</strong></td><td>Session trace segmentation and span caps</td><td><a href="https://docs.pharos.ai/open-telemetry/hermes/correlation.md">Cross-Surface Correlation</a></td></tr><tr><td><strong>🔔 Dashboards &#x26; Alerts</strong></td><td>The dashboard suite and alert set shipped for every backend</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/dashboards-and-alerts.md">Dashboards &amp; Alerts</a></td></tr></tbody></table>
