> For the complete documentation index, see [llms.txt](https://docs.pharos.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pharos.ai/open-telemetry/configuration/backends/splunk.md).

# Splunk Observability Cloud

Operating Hermes telemetry in Splunk Observability Cloud — direct vs. Collector mode, data model, APM patterns, the shipped dashboards, and troubleshooting.

Splunk Observability Cloud has two modes. **Direct mode** — a `splunk` block — sends traces over OTLP/gRPC to Splunk's ingest endpoint with the realm-derived address and `X-SF-Token` header; that endpoint accepts traces only. **Collector mode** — a `collector` block — sends everything to an OpenTelemetry Collector whose Splunk exporters forward traces and metrics to Splunk's OTLP/HTTP endpoints, and Salesforce log records to Splunk's log ingestion path. Choose direct mode only when traces are enough; choose Collector mode for production deployments that need metrics, central retry and batching, or logs.

***

## Setup

1. **Find your realm** — shown in your Splunk Observability Cloud profile (for example `us0`, `us1`, `eu0`). The ingest endpoint is derived from it.
2. **Create an organization access token** with the `ingest` scope. Store it in your cloud's secrets vault: in `ORG_CONFIGS` for direct mode, in the Collector's secret store for Collector mode. A separate Splunk API token is used only by Terraform to manage dashboards and detectors.
3. **Choose the mode** and write the block below.

***

## Wiring

Direct mode — traces only, so set `signals` explicitly:

```json
{
  "splunk": {
    "realm": "us1",
    "access_token": "REPLACE_WITH_SPLUNK_INGEST_TOKEN"
  },
  "signals": "traces"
}
```

Hermes derives `ingest.<realm>.observability.splunkcloud.com:443` and the `x-sf-token` header. `realm` is required. If `signals` is omitted Hermes resolves it to `traces`; if it names metrics or logs, those exports are attempted, warned about, and fail.

Collector mode — the sidecar topology the AWS and Azure modules support:

```json
{
  "collector": {},
  "signals": "traces,metrics"
}
```

A bare `collector` block derives `http://127.0.0.1:4317`. Set `collector.endpoint` (with `insecure: true` on a private network) only when the Collector runs as a separate service. The Collector's Splunk exporters carry the ingest token, realm, and — for logs — the HEC endpoint and token; those values are placed as extra keys in the same `collector` object for the deploy tooling to hand to the sidecar, and Hermes ignores them.

***

## Data Model

| Attribute                                                | Notes                                                                                                                                                                                                                    |
| -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `service.name`                                           | Surface identity (`hermes-iris`, `hermes-argus-rtem`, `hermes-argus-elf`) or per-org `service_name` with the surface on `hermes.channel` — filter on `hermes.channel` when you mean a surface in an org-named deployment |
| `salesforce.org_id` (Iris) / `salesforce.org.id` (Argus) | Org identifier; include both when filtering across channels                                                                                                                                                              |
| `trace_id`                                               | Shared across Iris and Argus for one session or request — keep all surfaces in the same realm and tenant so they can be viewed together                                                                                  |

Signals: traces in direct and Collector mode; metrics in Collector mode (never to the direct gRPC trace endpoint); logs through a Collector plus a Splunk log-ingestion path (Splunk Cloud Platform / Log Observer Connect).

***

## APM and Charts

* **Service map** — one node per `service.name`; with the surface convention you see Iris, RTEM, and ELF as separate services joined by shared traces.
* **Indexed span tags** — index `salesforce.org.id`, `salesforce.org_id`, `hermes.channel`, and `salesforce.user.id` to make them available in Tag Spotlight and Trace Analyzer filters.
* **Tag Spotlight recipes** — security: RTEM spans grouped by `salesforce.login.status` and `salesforce.threat.type`; performance: ELF Apex spans by `salesforce.apex.entry_point`; data-loss prevention: content and Bulk API spans by operation.
* **Trace Analyzer** — search by `session.id` to open one user's session across Triton and Event Monitoring; search by `salesforce.request.id` to join Iris and Argus for one request.
* **Charts (Collector mode)** — health from `hermes_org_connected` and `hermes_org_auth_failures_total`; Iris performance from the `apex_*` families; Argus RTEM/ELF monitoring from `argus_*` counters; volume from span counts per `service.name` and org.

***

## Dashboards and Detectors

A Terraform module provisions a **"Hermes Suite"** dashboard group in Splunk Observability Cloud — the eight shared Hermes dashboards (Overview, Argus Security, Argus Performance, Argus Adoption, Argus Data, Iris Apex Runtime, Hermes Health, Metric Coverage) as metric dashboards and charts — plus an optional **org-down detector** (`hermes_org_connected` at `0` for 10 minutes), also importable as JSON. The dashboards and detectors assume Collector mode, which carries metrics; a direct deployment receives no self-metrics, so the detector cannot fire there. See [Dashboards & Alerts](https://docs.pharos.ai/open-telemetry/configuration/dashboards-and-alerts.md).

***

## Troubleshooting

| Symptom                               | Check                                                                                                                                                          |
| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| No spans appear                       | `/health` 200 and `SF access token refreshed` in the log; exporter errors in the log; the block's `realm` and `access_token`; the token has the `ingest` scope |
| 401 Unauthorized                      | Wrong or revoked token, or a token from another organization                                                                                                   |
| Wrong realm / connection refused      | The derived endpoint uses the realm in the block — compare with the realm shown in your profile                                                                |
| Logs do not appear                    | Logs never travel the direct path; they need a Collector plus a Splunk log ingestion path                                                                      |
| Metrics are missing                   | Direct mode is traces-only; move the org to a `collector` block                                                                                                |
| Some orgs are missing                 | Check `/stats` and the startup log for `skipping org due to config error`                                                                                      |
| Cross-channel stitching does not work | Both surfaces must land in the same realm and tenant; confirm session unification is on for the org, or enable `trace_stitching.mode: iris_map`                |

***

## Related

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>🎯 Backend Configuration</strong></td><td>Block selection, endpoint rules, Collector wiring</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/backends.md">Backend Configuration</a></td></tr><tr><td><strong>🔔 Dashboards &#x26; Alerts</strong></td><td>The dashboard suite and alert set shipped for every backend</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/dashboards-and-alerts.md">Dashboards &amp; Alerts</a></td></tr><tr><td><strong>📡 Self-Instrumentation Metrics</strong></td><td>Why direct mode carries no health metrics</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/self-metrics.md">Self-Instrumentation Metrics</a></td></tr></tbody></table>
