> For the complete documentation index, see [llms.txt](https://docs.pharos.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pharos.ai/open-telemetry/configuration/azure.md).

# Azure Deployment

Runbook for deploying Hermes to your Azure subscription with Terraform — prerequisites, the four stages, Key Vault secrets, validation, and upgrades.

Hermes on Azure runs as a Container App inside your own subscription. Pharos delivers the Terraform modules; you run `terraform apply` for four stages in order. This page assumes [Salesforce setup](https://docs.pharos.ai/open-telemetry/configuration/salesforce-setup.md) is already done — the External Client App, RSA keypair, and `OTel_Configuration__mdt` record all need to exist before the Terraform in this page has anything to authenticate as.

***

## Components

| Component                          | Purpose                                                                                                                                           |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Azure Container Apps**           | Runs the Hermes container — Iris, Argus RTEM, Argus ELF, or the combined supervisor, depending on `hermes_runtimes`.                              |
| **Azure Table Storage**            | Backs Hermes's state stores: replay, ELF cursor, ELF lease, dedup, origin, trace-map, session-epoch, and stream-lease.                            |
| **Azure Key Vault**                | Holds the `org-configs` secret — the whole `ORG_CONFIGS` document. Nothing sensitive lives in environment variables or the image.                 |
| **Azure Container Registry (ACR)** | Your private copy of the Hermes image, imported with `az acr import` — no local build or push.                                                    |
| **Log Analytics**                  | Container logs, metrics, three provisioned workbooks, and the queries behind alerts and autoscaling.                                              |
| **User-assigned managed identity** | Attached to the Container App; grants `AcrPull`, `Key Vault Secrets User`, and `Storage Table Data Contributor` without any hardcoded credential. |

***

## Prerequisites

**Local tooling:** Terraform 1.5+, Azure CLI 2.60+, Docker 24+, `jq`.

**Access:** an Azure subscription with rights to create Resource Groups, ACR, Storage Accounts, Key Vault, managed identities, Container Apps, Log Analytics, and Azure Monitor resources. Authenticate and register the resource providers once per subscription:

```bash
az login
az account set --subscription "<subscription-id-or-name>"
for ns in Microsoft.App Microsoft.ContainerRegistry Microsoft.KeyVault \
          Microsoft.OperationalInsights Microsoft.Storage Microsoft.ContainerService; do
  az provider register --namespace $ns --wait
done
```

**Salesforce:** [Salesforce setup](https://docs.pharos.ai/open-telemetry/configuration/salesforce-setup.md) complete — consumer key, integration username, and the RSA private key whose certificate is uploaded to the app (`openssl genrsa -out hermes.private.key 2048`, then a self-signed certificate as shown on that page).

**Backend:** the credential for your export destination. In Collector mode the token goes to the Collector sidecar, not to Hermes; see [Backend Configuration](https://docs.pharos.ai/open-telemetry/configuration/backends.md).

***

## The Four Stages

On Azure the **apply order differs from the directory numbers**: the Foundation stage owns the Resource Group, so it is applied first.

| Order | Module         | Creates                                                                                                                                                        | Frequency                                  |
| ----- | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ |
| 1     | `2-foundation` | Resource Group, Storage Account + replay table, Key Vault, the empty `org-configs` secret                                                                      | Once; state survives runtime destroy/apply |
| 2     | `1-registry`   | ACR                                                                                                                                                            | Once                                       |
| 3     | `3-runtime`    | Managed identity and role assignments, Log Analytics workspace, Container Apps Environment, the seven per-runtime state tables, the Container App, 3 workbooks | Up/down as needed                          |
| 4     | `4-alerts`     | Azure Monitor alert rules and Action Group                                                                                                                     | After Stage 3 is validated                 |

Stages find each other by name through `environment` (prefix `hermes<environment>`, so `hermesprod`) and `resource_group_name` — set both identically everywhere. Storage Account names have no hyphen (`hermesprodreplay`); that is the Azure naming rule.

```mermaid
flowchart LR
  subgraph dirs["Module directories"]
    d1["1-registry"]
    d2["2-foundation"]
    d3["3-runtime"]
    d4["4-alerts"]
  end
  subgraph order["Apply order"]
    o1["① Foundation<br/>2-foundation"] --> s["populate org-configs<br/>(Key Vault)"] --> o2["② Registry<br/>1-registry"] --> o3["③ Runtime<br/>3-runtime"] --> o4["④ Alerts<br/>4-alerts"]
  end
  d2 -.-> o1
  d1 -.-> o2
  d3 -.-> o3
  d4 -.-> o4
  style o1 fill:#37474f,stroke:#212121,color:#fff
```

{% stepper %}
{% step %}

### Stage 1 — Foundation

Apply `2-foundation`. The `org-configs` secret is created with a placeholder value and `ignore_changes` on its value, so re-applying never overwrites what you put in it. Capture the outputs (`resource_group_name`, `key_vault_name`, `storage_account_name`), then **populate the secret** — not a Terraform action, and the Container App starts against the placeholder without it:

```bash
az keyvault secret set \
  --vault-name "$(terraform -chdir=2-foundation output -raw key_vault_name)" \
  --name org-configs \
  --value "$(jq -n \
    --arg api_key   "$HONEYCOMB_INGEST_KEY" \
    --arg org_id    "$SF_ORG_ID" \
    --arg client_id "$SF_CLIENT_ID" \
    --arg username  "$SF_USERNAME" \
    --rawfile private_key hermes.private.key \
    '{
       honeycomb: { api_key: $api_key },
       orgs: [ {
         org_id: $org_id,
         sf: { login_url: "https://login.salesforce.com", client_id: $client_id,
               private_key: $private_key, username: $username }
       } ]
     }')"
```

Swap the `honeycomb` block for `splunk`, `collector`, or `otlp` per [`ORG_CONFIGS` Reference](https://docs.pharos.ai/open-telemetry/configuration/org-configs.md).
{% endstep %}

{% step %}

### Stage 2 — Registry

Apply `1-registry` (one resource: the ACR; `create_acr = false` skips it if you host the image elsewhere). Capture `acr_name` and `acr_login_server`, then import the Pharos-published image from Docker Hub without pulling it locally:

```bash
az acr import --name "$ACR_NAME" \
  --source docker.io/pharosai/hermes:<vX.Y.Z> \
  --username <pharos-dockerhub-user> --password <pharos-dockerhub-token> \
  --image hermes:<vX.Y.Z>
```

See [Versioning & Upgrades](https://docs.pharos.ai/open-telemetry/configuration/versioning-and-upgrades.md) for choosing a tag.
{% endstep %}

{% step %}

### Stage 3 — Runtime

Write `3-runtime/terraform.tfvars`. `image` must be set — its default is a Microsoft sample container, not Hermes:

```hcl
location            = "eastus"
environment         = "prod"
resource_group_name = "hermes-rg"        # must match Stage 1
image               = "<ACR_LOGIN_SERVER>/hermes:<vX.Y.Z>"
hermes_runtimes     = "iris,argus-rtem,argus-elf-live"
container_cpu       = 0.5
container_memory    = 1.0
```

Apply. The stack reads the Key Vault, Storage Account, and ACR from the earlier stages by name. If the environment fails to create with a per-subscription environment quota or regional capacity error, pin it to another region with `container_apps_location = "<region>"` (the Log Analytics workspace stays in `location`) or reuse an existing environment via `existing_container_app_environment_id`. Outputs include the three workbook URLs, `container_app_name`, and `log_analytics_workspace_id`.
{% endstep %}

{% step %}

### Stage 4 — Alerts

Apply only after Stage 3 is validated — the stack reads the Container App, Storage Account, and workspace through data sources. Write `4-alerts/terraform.tfvars`:

```hcl
location            = "eastus"
environment         = "prod"
resource_group_name = "hermes-rg"
alert_email         = "ops@your-domain.com"   # empty skips the email receiver
elf_metrics_container_app_id = ""             # optional — see below
```

CPU and memory alerts read Container Apps' own `CpuPercentage` / `MemoryPercentage`, so there is no sizing value to keep in sync here. The ELF-lag alert needs a runtime publishing `argus_elf_lag_seconds` (`CLOUD_METRICS=azure_monitor`); set `elf_metrics_container_app_id` to that app's resource ID to create it, or leave it empty and the alert is skipped rather than created in a permanent no-data state. Thresholds are tunable through the optional `alarm_thresholds` object (`cpu_percent` 80, `memory_percent` 80, `revision_restart_max` 3, `table_latency_ms` 1000, `elf_lag_seconds` 600, `dedup_errors_per_min` 10). Confirm the Action Group email receiver from your inbox after apply.
{% endstep %}
{% endstepper %}

***

## Verify the Deployment

```bash
RESOURCE_GROUP=$(terraform -chdir=3-runtime output -raw resource_group_name)
APP_NAME=$(terraform -chdir=3-runtime output -raw container_app_name)

az containerapp revision list --resource-group "$RESOURCE_GROUP" --name "$APP_NAME" \
  --query "[].{name:name,active:properties.active,replicas:properties.replicas,health:properties.healthState}" -o table
az containerapp logs show --resource-group "$RESOURCE_GROUP" --name "$APP_NAME" --follow
```

Expected log lines when healthy:

```json
{"level":"INFO","msg":"hermes starting"}
{"level":"INFO","msg":"supervisor: starting runtime","runtime":"iris"}
{"level":"INFO","msg":"supervisor: starting runtime","runtime":"argus-rtem"}
{"level":"INFO","msg":"supervisor: starting runtime","runtime":"argus-elf-live"}
{"level":"INFO","msg":"health: server listening","port":8080}
{"level":"INFO","msg":"SF access token refreshed"}
```

Hermes runs as a background worker with **no external ingress** — the `:8080` health server is reached only by the Container Apps internal probe. Verify through the revision's `healthState` (`Healthy`) and the logs rather than a public URL; Log Analytics ingestion can lag a few minutes after a fresh environment comes up. Then trigger Salesforce activity — a Triton log for Iris, a login or API call for RTEM — and look for the first span in your backend within one to two minutes. Work through the [validation checklist](https://docs.pharos.ai/open-telemetry/configuration/operations.md#validation-checklist).

***

## Secrets Management

The whole `ORG_CONFIGS` document is one Key Vault secret, `org-configs`, read by the Container App through its managed identity — no credential is embedded anywhere in the deployment.

{% hint style="warning" %}
**A plain restart is not enough.** Azure Container Apps resolve Key Vault-referenced secrets when a **revision** is created. Restarting a replica reuses the already-resolved value, so after changing the secret you must create a new revision (below). If your Container App's secret is a literal rather than a Key Vault reference (`az containerapp show … --query properties.configuration.secrets` prints `keyVaultUrl` only for references), write the new value there too with `az containerapp secret set`.
{% endhint %}

***

## Applying Changes, Upgrading, and Rolling Back

* **`ORG_CONFIGS` change:** `az keyvault secret set --name org-configs --file org-configs.json`, then force a new revision — the secret value is not part of the revision hash, so re-submitting the same image is a no-op:

  ```bash
  az containerapp update --resource-group "$RESOURCE_GROUP" --name "$APP_NAME" \
    --revision-suffix "cfg$(date -u +%Y%m%d%H%M%S)"
  ```

  The old revision keeps draining its Pub/Sub subscription for about 90 seconds; wait for it to report zero replicas before judging the change.
* **New image:** import the new tag into ACR, set `image` in `3-runtime/terraform.tfvars`, `terraform apply` (or `az containerapp update --image …`); a new revision starts automatically. Deploy the image **before** rewriting `ORG_CONFIGS` when a release changes the schema — see [Versioning & Upgrades](https://docs.pharos.ai/open-telemetry/configuration/versioning-and-upgrades.md).
* **Rollback:** `az containerapp revision list` to find the previous revision, then `az containerapp revision activate --revision <previous>`. No state rollback is needed; Table Storage holds resumption state the previous image reads unchanged.

***

## Monitoring

Stage 3 provisions three Azure Monitor workbooks — **Operational Health**, **State Backend (Table Storage)**, and **Cost Snapshot** — plus an optional Pipeline Throughput workbook you can import by hand. Stage 4 provisions the alert rules, grouped by concern:

| Alert                                         | Severity | Fires when                                                              |
| --------------------------------------------- | -------- | ----------------------------------------------------------------------- |
| H-1 replica count below desired               | 1        | Fewer replicas than `min_replicas`                                      |
| H-2 / H-3 CPU or memory sustained above 80 %  | 2        | Container Apps `CpuPercentage` / `MemoryPercentage` over threshold      |
| H-4 replica restart count elevated            | 1        | More than 3 restarts in the window                                      |
| H-5 pipeline reports itself unhealthy         | 1        | Unhealthy verdicts logged for \~2 minutes or more                       |
| S-1 Table Storage throttling (HTTP 429)       | 2        | Any throttled request                                                   |
| S-2 Table Storage E2E latency above 1 s       | 2        | Mean latency over threshold                                             |
| S-3 dedup/origin table capacity above ceiling | 3        | Growing trend, no immediate impact                                      |
| P-1 dedup backend errors above 10/min         | 2        | Fall-open errors counted in logs                                        |
| P-2 ELF lag above 10 minutes                  | 2        | `argus_elf_lag_seconds` > 600 s (only when the publisher is configured) |
| P-3 startup failure or missing health log     | 1        | No health log line after start                                          |

Severity 1 is "pipeline stopped, respond now"; 2 is degraded within the hour; 3 is next business day. Hermes-level alerting — `hermes_org_connected`, `hermes_org_auth_failures_total` — lives in your OTel backend; see [Self-Instrumentation Metrics](https://docs.pharos.ai/open-telemetry/configuration/self-metrics.md) and [Dashboards & Alerts](https://docs.pharos.ai/open-telemetry/configuration/dashboards-and-alerts.md).

***

## Cost

Azure Container Apps on the Consumption plan, `min_replicas = 1`, `LOG_LEVEL=info`; the low end of each range is idle billing, the high end an always-active replica:

| Scenario                                                    | Container Apps | Table Storage + Key Vault | Log Analytics | ACR Basic (optional) | Total                                  |
| ----------------------------------------------------------- | -------------- | ------------------------- | ------------- | -------------------- | -------------------------------------- |
| 5 orgs, 100 events/s (0.25 vCPU / 0.5 Gi)                   | $6–20          | < $0.10                   | \~$0.30       | \~$5                 | **$6–21** without ACR, **$11–26** with |
| 20 orgs, 1 000 events/s (0.5 vCPU / 1 Gi)                   | $12–40         | < $0.50                   | \~$1.50       | \~$5                 | **$18–47**                             |
| 50 orgs, 5 000 events/s (ELF scaled, avg 2 × 1 vCPU / 2 Gi) | $48–158        | \~$2                      | \~$5          | \~$5                 | **$60–170**                            |

Container Apps compute is about 1.9× the equivalent AWS Fargate shape at list price. Set an Azure Budget on the resource group, keep Log Analytics retention at 30 days, and keep `LOG_LEVEL=info` — log ingestion is the line that grows fastest. The full cost model is in [Sizing & Cost](https://docs.pharos.ai/open-telemetry/hermes/sizing-and-cost.md).

***

## Teardown

`terraform -chdir=3-runtime destroy` removes the runtime compute while preserving ACR, Key Vault, and Table Storage; a later `apply` resumes where the cursors left off. For full decommission, destroy `4-alerts`, then `3-runtime`, then `1-registry`, then `2-foundation`. If you later re-create a Log Analytics workspace with the same name in the same region, purge the soft-deleted one first.

***

## Troubleshooting

| Symptom                                                | Check                                                                                                                                                                                                                                                                         |
| ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Revision keeps restarting                              | Read the console logs for the first `ERROR`. Usually the `org-configs` secret still holds the placeholder, is malformed, or an org was skipped at startup — see [Operations](https://docs.pharos.ai/open-telemetry/configuration/operations.md#an-org-is-missing-from-stats). |
| Container cannot pull image                            | The `image` value does not exist in ACR (check `az acr repository show-tags`), or the `AcrPull` role assignment has not propagated yet — wait a minute and create a new revision.                                                                                             |
| Key Vault permission denied after a Terraform re-apply | The role assignment was re-created and RBAC propagation lags; create a new revision after a few minutes rather than restarting.                                                                                                                                               |
| `/health` 503                                          | Fewer orgs connected than `HEALTH_MULTI_ORG_MODE` requires; query the console logs for the org names.                                                                                                                                                                         |
| Table Storage throttling alerts                        | A backfill or ELF burst; pause backfill or raise ELF parallelism gradually.                                                                                                                                                                                                   |
| Dedup table growing unbounded                          | Confirm the table's TTL cleanup is scheduled and the dedup mode for the noisy family is what you intend.                                                                                                                                                                      |
| Org disappeared from `/stats` but Key Vault has it     | The Container App is still on the old revision, or its secret is a literal that was not updated — see Secrets Management above.                                                                                                                                               |
| Log Analytics costs exploding                          | `LOG_LEVEL=debug` left on, or retention above 30 days.                                                                                                                                                                                                                        |

***

## Related

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>☁️ Salesforce Setup</strong></td><td>External Client App, JWT auth, and OTel_Configuration__mdt</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/salesforce-setup.md">Salesforce Setup</a></td></tr><tr><td><strong>🧩 ORG_CONFIGS Reference</strong></td><td>The document you place in Key Vault</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/org-configs.md">ORG_CONFIGS Reference</a></td></tr><tr><td><strong>🏷️ Versioning &#x26; Upgrades</strong></td><td>Choosing a tag, upgrade order, rollback</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/versioning-and-upgrades.md">Versioning &amp; Upgrades</a></td></tr><tr><td><strong>📐 Sizing &#x26; Cost</strong></td><td>Compute shapes and the cost model behind the table above</td><td><a href="https://docs.pharos.ai/open-telemetry/hermes/sizing-and-cost.md">Sizing &amp; Cost</a></td></tr></tbody></table>
