> For the complete documentation index, see [llms.txt](https://docs.pharos.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pharos.ai/open-telemetry/hermes/correlation.md).

# Cross-Surface Correlation

How Hermes keeps one Salesforce session and request as one trace across Iris and Argus, built on deterministic idconv identity.

Iris and Argus each convert their own event stream into OTel signals independently — but a single Salesforce session or request can produce telemetry on more than one surface at once. Cross-surface correlation is what keeps those independent conversions from splintering into unrelated traces. Every mechanism on this page is built on the same foundation: `idconv` deterministic identity, where the same Salesforce identifier — a session ID, a Triton transaction ID, or a `REQUEST_ID` — derives to the same `trace_id` no matter which runtime does the derivation (see [Architecture](https://docs.pharos.ai/open-telemetry/hermes/architecture.md#deterministic-identity-idconv)).

***

## Cross-Channel Session Unification

Iris builds a session umbrella span over a user's Triton telemetry (see [Iris](https://docs.pharos.ai/open-telemetry/hermes/iris.md)). Cross-channel session unification extends that same umbrella across Argus RTEM and ELF spans too, so "everything this user did in this session" spans Triton and Salesforce Event Monitoring telemetry together. It is **on by default** for every org; opt out per org with `"session_unification": {"enabled": false}` in [`ORG_CONFIGS`](https://docs.pharos.ai/open-telemetry/configuration/org-configs.md), which turns the session root off on Iris, RTEM, and ELF alike.

The mechanism anchors on the same resolved `AuthSession` Iris already uses, keyed by the Salesforce user id (`USER_ID`) — deliberately **not** `SESSION_KEY`, which is frequently blank on the ELF event types whose activity window overlaps Iris's. Every channel derives the same session span id and the same session-segment trace id from the same `AuthSession` id, so RTEM, ELF, and Iris spans all land in one trace under one umbrella without a shared store to coordinate. A small claim in the trace-map store stops two runtimes from both emitting the same trace's session root.

Two consequences of anchoring on `AuthSession`:

* The integration user must be able to read `AuthSession`. That is an **Argus** requirement, not an Iris one — Iris takes the session from the log payload, while Event Monitoring events carry none — and it comes from the `Hermes_Argus` permission set (see [Security](https://docs.pharos.ai/open-telemetry/hermes/security.md#authentication)). Where the user cannot read `AuthSession`, no session resolves and each span keeps its own trace — grouping degrades, telemetry is never lost.
* Historical ELF rows can reference a session Salesforce has already purged, so backfilled spans can miss the umbrella their live peers got.

![A cross-technology Salesforce trace (LWC to Apex) with an error span, in Datadog APM.](https://4264222071-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fdl9tmJ56lvwg5XY63QYr%2Fuploads%2Fgit-blob-31d7a8ab3095c183d45a2218ee12ebc2746eb4ce%2Fdatadog-apm-flame-graph.png?alt=media)

***

## Request Anchor (ELF-Only Traces)

An ELF file row becomes one span, and every span of one Salesforce request lands in one trace because the trace id derives from `REQUEST_ID`. Argus also emits one synthetic **request span** per (trace, request) so those rows have a single root instead of arriving as flat siblings. Its span id is `MD5(REQUEST_ID)[:16]` — byte-identical to the request anchor Iris emits — so an ELF-only trace and an Iris trace for the same request agree on the anchor's identity. When a session does resolve, the session overlay re-parents this one anchor, yielding session → request → events. The cost is one extra span per request.

***

## Trace Stitching (Iris ↔ Argus)

Trace stitching adds an explicit request-level handoff for cases where Argus should land on the exact trace Iris already started: it's **opt-in per org** (`hermes.trace_stitching.mode=iris_map`; the default is `off`).

When enabled, Iris writes each `(org_id, REQUEST_ID) → trace_id` mapping it derives to a shared trace-map store as it processes events. Argus then reads that same store when it encounters a matching `REQUEST_ID`, so an Argus span lands in exactly the trace Iris already started, parented to the Iris request anchor. Entries expire after a configurable TTL (`map_ttl_hours`, default 48h — long enough to outlast peak ELF lag).

If the trace-map store is unavailable or a lookup misses, Argus doesn't block or drop the event — it **falls open** to its own deterministic derivation, so the trace still lands somewhere sensible; the shared store is an optimization on top of deterministic identity, not a dependency it requires.

Trace stitching only ever rewrites **spans** — metrics and logs are never touched, so correcting a `REQUEST_ID` mapping later can't cause a metric or log timeseries to jump between trace contexts.

***

## Session Trace Segmentation and Span Caps

A session has no hard upper bound — an active workday can be one session — and a trace that long is a problem for most backends. Hermes bounds session traces two ways:

* **Span-count cap** (`max_spans_per_trace`, default 5000, configurable per org or deployment-wide in `ORG_CONFIGS`; `0` disables): once a session trace crosses the cap, the next span starts a new trace "epoch". Orgs whose spans carry the full governor-limit attribute set should use 2000–3000. Tempo refuses traces over its per-trace byte limit and Honeycomb recommends splitting around 10k spans.
* **Time segmentation** (`HERMES_SESSION_TRACE_SEGMENT`, a Go duration such as `6h`; default off): cuts a session into fixed time windows, each its own trace, with the session root of every segment after the first carrying `session.previous_trace_id` and a span link back to the prior segment. Set it identically on every runtime so Iris and Argus cut at the same instants. **Datadog rejects spans older than 18 hours, so Datadog deployments should set a window.**

```mermaid
flowchart LR
  subgraph s["One Salesforce session (session.id = S)"]
    direction LR
    t1["trace 1<br/>segment 00:00–06:00<br/>session root"] --> t2["trace 2<br/>segment 06:00–12:00<br/>session root<br/>session.previous_trace_id = trace 1"]
    t2 --> t3["trace 3<br/>epoch cut at 5000 spans<br/>session root<br/>session.previous_trace_id = trace 2"]
  end
  style t1 fill:#37474f,stroke:#212121,color:#fff
  style t2 fill:#37474f,stroke:#212121,color:#fff
  style t3 fill:#2e7d32,stroke:#1b5e20,color:#fff
```

Each arrow is both the `session.previous_trace_id` attribute and an OTel span link on the new segment's session root, so a session split across traces stays navigable by querying `session.id` and walking the links.

***

## RTEM and ELF Dedup

Real-Time Event Monitoring and EventLogFile can both describe the same underlying Salesforce event — RTEM sees it near-real-time, ELF sees it again later in a batch log. Left alone, that's the same event reported twice. RTEM ↔ ELF dedup is **opt-in** and resolves it per event family, using a strategy declared in the shared YAML mapping engine.

The default strategy is **surface-select**: one surface is designated authoritative for a given family, and the other surface's matching event is suppressed. Surface-select requires no state store at all — it's a static routing decision, and it's the default `dedup.mode` when the field is left unset. An alternative fingerprint-based strategy is also available, holding RTEM fingerprints (an `xxhash64` over cross-surface-stable fields) in a dedup store, selected via `dedup.mode`: `suppress`, `emit-as-supplement`, or `emit-always`.

Whichever strategy is active, **all dedup backends fall open**: if the dedup store is unreachable, Hermes does not hold back or drop events waiting for it. The outage produces duplicate telemetry — the same event reported by both surfaces — never a missing one.

{% hint style="info" %}
**Fall-open, always in the same direction.** Every correlation mechanism on this page treats a store outage as "miss and continue," not "block and wait." The failure mode is always a duplicate or a less-enriched trace — never dropped telemetry.
{% endhint %}

***

## Async Threshold Nesting

Some Argus events describe work that finishes well after the transaction that started it — an async Apex job, a scheduled report run. Async threshold nesting decides whether a late-arriving event still belongs inside its originating trace, or needs a trace of its own. It's opt-in per event type — no shipped event type declares `async` by default, so an org turns on threshold-aware nesting only for the channels or ELF types it configures in its [pipeline config](https://docs.pharos.ai/open-telemetry/configuration/pipeline-config.md).

The mechanism compares the event's arrival time against a stored origin time for the same `REQUEST_ID`, against a configurable per-type threshold (`async.threshold_hours`, defaulting to the org-wide `hermes.elf.async_link_threshold_hours`, default 18h):

* **Within the threshold** — the event nests into the originating trace, as a child of the work that spawned it.
* **Beyond the threshold** — the event gets its own trace, connected back to the origin by a backward span link (`salesforce.async.linked_request_id`, `salesforce.async.link_reason`) rather than nesting. The 18-hour default is a general-purpose "nest vs. fork" tuning value rather than a fixed rule — an org can raise or lower it to fit how long its own async work realistically runs.

The origin store behind this is opt-in and, like the others, **falls open**: first-writer-wins on origin records, and a store outage means an event that should have nested instead gets its own trace with a span link — a less-tidy trace, not a lost one.

***

## One Identifier, One Trace

```mermaid
graph LR
  S["Salesforce identifiers<br/>session · transaction · REQUEST_ID"] --> C["idconv deterministic derivation"]
  C --> T["trace_id"]
  T --> IS["Iris span"]
  T --> AS["Argus span"]
  style T fill:#37474f,stroke:#212121,color:#fff
  style C fill:#2e7d32,stroke:#1b5e20,color:#fff
```

The same identifier resolves to the same `trace_id` on both channels. Session unification, request anchors, trace stitching, dedup, async nesting, and segmentation all sit on top of this identity — they refine *how* signals from Iris and Argus join a trace, not *whether* they can.

***

## Related

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>🌈 Iris</strong></td><td>The Triton/Pharos application-telemetry bridge</td><td><a href="https://docs.pharos.ai/open-telemetry/hermes/iris.md">Iris</a></td></tr><tr><td><strong>👁️ Argus</strong></td><td>The Salesforce Event Monitoring bridge</td><td><a href="https://docs.pharos.ai/open-telemetry/hermes/argus.md">Argus</a></td></tr><tr><td><strong>🧩 ORG_CONFIGS Reference</strong></td><td>session_unification, trace_stitching, session caps, and the state stores behind them</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/org-configs.md">ORG_CONFIGS Reference</a></td></tr></tbody></table>
