> For the complete documentation index, see [llms.txt](https://docs.pharos.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pharos.ai/open-telemetry/configuration/aws.md).

# AWS Deployment

Runbook for deploying Hermes to your AWS account with Terraform — prerequisites, the four stages, validation, upgrades, and troubleshooting.

Hermes on AWS runs as an ECS Fargate service inside your own account. Pharos delivers the Terraform modules; you run `terraform apply` for four stages in order. This page assumes [Salesforce setup](https://docs.pharos.ai/open-telemetry/configuration/salesforce-setup.md) is already done — the External Client App, RSA keypair, and `OTel_Configuration__mdt` record all need to exist before the Terraform in this page has anything to authenticate as.

***

## Components

| Component             | Purpose                                                                                                                                                                            |
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **ECS Fargate**       | Runs the Hermes container — Iris, Argus RTEM, Argus ELF, or the combined supervisor, depending on `hermes_runtimes`.                                                               |
| **DynamoDB**          | Backs Hermes's state stores — eight tables: replay, ELF cursor, ELF lease, dedup, origin, trace-map, session-epoch, and stream-lease.                                              |
| **Secrets Manager**   | Holds the `org-configs` secret — the whole `ORG_CONFIGS` document, including Salesforce keys and any backend token. Nothing sensitive lives in environment variables or the image. |
| **ECR**               | Your private copy of the Hermes image, imported from Docker Hub. You never build the image yourself.                                                                               |
| **VPC + NAT gateway** | Private subnets for the task; egress to Salesforce and your backend through the NAT gateway. Hermes has no public ingress.                                                         |
| **CloudWatch**        | Container logs, Container Insights metrics, three provisioned dashboards, and the alarms that page you.                                                                            |

***

## Prerequisites

**Local tooling:** Terraform 1.5+, AWS CLI v2, Docker 24+, `jq`.

**Access:** AWS credentials with rights to create ECS, ECR, IAM, VPC, Secrets Manager, DynamoDB, and CloudWatch resources (`export AWS_PROFILE=<your-profile>`). The task role Terraform creates is scoped to `secretsmanager:GetSecretValue` on the one secret, CloudWatch Logs on the module's log group, and DynamoDB item operations on only the tables the selected runtimes use — no S3, no EC2, no admin.

**Salesforce:** [Salesforce setup](https://docs.pharos.ai/open-telemetry/configuration/salesforce-setup.md) complete — you need the External Client App consumer key, the integration username, and the RSA private key whose certificate is uploaded to the app. Generate the keypair once if you have not:

```bash
openssl genrsa -out hermes.private.key 2048
openssl req -new -x509 -key hermes.private.key -out hermes.certificate.crt -days 730 \
  -subj "/C=US/ST=CA/O=YourOrg/CN=hermes"
```

**Backend:** the credential for your export destination — for example a Honeycomb Ingest key, or a Splunk ingest token. In Collector mode the token goes to the Collector, not to Hermes; see [Backend Configuration](https://docs.pharos.ai/open-telemetry/configuration/backends.md).

***

## The Four Stages

| Stage | Module         | Creates                                                                             | Frequency                            |
| ----- | -------------- | ----------------------------------------------------------------------------------- | ------------------------------------ |
| 1     | `1-registry`   | ECR repository                                                                      | Once                                 |
| 2     | `2-foundation` | DynamoDB tables, the `org-configs` secret                                           | Once; survives runtime destroy/apply |
| 3     | `3-runtime`    | VPC, NAT, ECS cluster, IAM roles, task definition, service, 3 CloudWatch dashboards | Up/down as needed                    |
| 4     | `4-alerts`     | CloudWatch alarms and SNS topic                                                     | After Stage 3 is validated           |

Each stage is applied with `terraform -chdir=<module> init && terraform -chdir=<module> apply`. Stages find each other by name through the `environment` variable (prefix `hermes-<environment>`), not by remote state — set it identically everywhere.

```mermaid
flowchart LR
  r["1-registry<br/>ECR + image"] --> f["2-foundation<br/>DynamoDB + secret"] --> s["populate org-configs<br/>(Secrets Manager)"] --> t["3-runtime<br/>VPC · ECS · dashboards"] --> a["4-alerts<br/>alarms · SNS"]
  e["environment = prod<br/>→ names prefixed hermes-prod-*"] -.-> f
  e -.-> t
  e -.-> a
  style s fill:#37474f,stroke:#212121,color:#fff
```

{% stepper %}
{% step %}

### Stage 1 — Registry

Apply `1-registry` and capture the repository URL from the `ecr_repository_url` output. Then import the Pharos-published image into it — pull `pharosai/hermes:<vX.Y.Z>` from Docker Hub with your Pharos-issued read credentials, tag it for your ECR URL, and push (or use your registry's import tooling). Note the resulting image URI; production pins the **digest** (`<ECR_URL>@sha256:<digest>`) rather than the tag. See [Versioning & Upgrades](https://docs.pharos.ai/open-telemetry/configuration/versioning-and-upgrades.md) for choosing a tag.
{% endstep %}

{% step %}

### Stage 2 — Foundation

Apply `2-foundation`. It creates the DynamoDB tables and an empty Secrets Manager secret. Then **populate the secret** — this is not a Terraform action and is easy to skip; the container starts against an empty secret without it:

```bash
aws secretsmanager put-secret-value \
  --secret-id "$(terraform -chdir=2-foundation output -raw org_configs_secret_id)" \
  --region "$AWS_REGION" \
  --secret-string "$(jq -n \
    --arg api_key   "$HONEYCOMB_INGEST_KEY" \
    --arg org_id    "$SF_ORG_ID" \
    --arg client_id "$SF_CLIENT_ID" \
    --arg username  "$SF_USERNAME" \
    --rawfile private_key hermes.private.key \
    '{
       honeycomb: { api_key: $api_key },
       orgs: [ {
         org_id: $org_id,
         sf: { login_url: "https://login.salesforce.com", client_id: $client_id,
               private_key: $private_key, username: $username }
       } ]
     }')"
```

Swap the `honeycomb` block for `splunk`, `collector`, or `otlp` per [`ORG_CONFIGS` Reference](https://docs.pharos.ai/open-telemetry/configuration/org-configs.md). The private key is the PEM text itself — `--rawfile` handles the newlines.
{% endstep %}

{% step %}

### Stage 3 — Runtime

Write `3-runtime/terraform.tfvars` (the file is gitignored and does not exist until you create it). `image_uri` is the only variable without a default:

```hcl
aws_region           = "us-west-1"
environment          = "prod"
image_uri            = "<ECR_URL>@sha256:<digest>"   # or "<ECR_URL>:<tag>" outside production
hermes_runtimes      = "iris,argus-rtem,argus-elf-live"
hermes_desired_count = 1
task_cpu             = 512
task_memory          = 1024
```

Apply. The stack reads the table ARNs and secret ARN from Stage 2 by name. Outputs include the three CloudWatch dashboard URLs (Operational Health, State Backend, Cost Snapshot), the cluster and service names, and the log group.
{% endstep %}

{% step %}

### Stage 4 — Alerts

Apply only after Stage 3 is validated — the stack reads the ECS cluster through a data source and `plan` fails if it does not exist. Write `4-alerts/terraform.tfvars`:

```hcl
aws_region  = "us-west-1"
environment = "prod"
alert_email = "ops@your-domain.com"   # empty skips the email subscription
task_cpu    = 512
task_memory = 1024
```

{% hint style="warning" %}
**`task_cpu` and `task_memory` must equal the Stage 3 values.** They are the divisor for the absolute CPU and memory alarm thresholds (`task_cpu × cpu_percent / 100`). Resize the task in Stage 3 without re-applying Stage 4 and the alarms keep the old thresholds silently — a 2048 MiB task still alarms at 819 MiB.
{% endhint %}

Thresholds are tunable through the optional `alarm_thresholds` object (`cpu_percent` 80, `memory_percent` 80, `task_restart_count` 3, `elf_lag_seconds` 600). Confirm the SNS email subscription from your inbox after apply.
{% endstep %}
{% endstepper %}

***

## Verify the Deployment

```bash
CLUSTER=$(terraform -chdir=3-runtime output -raw ecs_cluster_name)
SERVICE=$(terraform -chdir=3-runtime output -raw ecs_service_name)
LOG_GROUP=$(terraform -chdir=3-runtime output -raw hermes_log_group)

aws ecs wait services-stable --cluster "$CLUSTER" --services "$SERVICE" --region "$AWS_REGION"
aws logs tail "$LOG_GROUP" --follow --region "$AWS_REGION"
```

Expected log lines when healthy:

```json
{"level":"INFO","msg":"hermes starting"}
{"level":"INFO","msg":"supervisor: starting runtime","runtime":"iris"}
{"level":"INFO","msg":"supervisor: starting runtime","runtime":"argus-rtem"}
{"level":"INFO","msg":"health: server listening","port":8080}
{"level":"INFO","msg":"SF access token refreshed"}
{"level":"INFO","msg":"subscriber: stream open","topic":"/event/pharos__Log_Event__e"}
```

Hermes has no public ingress, so check `/health` through ECS Exec (`aws ecs execute-command … --command "wget -qO- http://localhost:8080/health"`; requires `enable_execute_command` on the task) or rely on the log stream. Then trigger Salesforce activity — a Triton log for Iris, a login or API call for RTEM — and look for the first span in your backend within one to two minutes. Work through the [validation checklist](https://docs.pharos.ai/open-telemetry/configuration/operations.md#validation-checklist).

***

## Applying Changes, Upgrading, and Rolling Back

* **`ORG_CONFIGS` change** (new org, rotated key, redaction flag): `aws secretsmanager put-secret-value` with the new document, then `aws ecs update-service --force-new-deployment`. The secret is read at startup.
* **New image:** the task definition pins the image and the service ignores task-definition drift, so a bare force-new-deployment does not move to a new image. Pin the new digest in `3-runtime/terraform.tfvars` → `terraform apply` → `aws ecs update-service --task-definition <family>:<N>` → `aws ecs wait services-stable`. Deploy the image **before** rewriting `ORG_CONFIGS` when a release changes the schema — see [Versioning & Upgrades](https://docs.pharos.ai/open-telemetry/configuration/versioning-and-upgrades.md).
* **Rollback:** `aws ecs update-service --task-definition <previous-family>:<N> --force-new-deployment`. No state rollback is needed; the DynamoDB tables are resumption state the previous image reads unchanged.
* **Resize:** change `task_cpu` / `task_memory` in Stage 3, apply, then re-apply Stage 4 with the same values.

***

## Monitoring

Stage 3 provisions three CloudWatch dashboards — **Operational Health** (task count, CPU, memory, restarts, health-check failures), **State Backend** (DynamoDB consumption, throttles, TTL deletions per table), and **Cost Snapshot** — plus an optional Pipeline Throughput dashboard you can import by hand. Stage 4 provisions 19 alarms and one log metric filter, named `hermes-<environment>-<thing>`:

| Alarm                                                                                                                 | Severity | Fires when                                                                                              |
| --------------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------- |
| `task-count-zero`                                                                                                     | P1       | Running tasks < 1 for 2 consecutive minutes (missing data also fires)                                   |
| `task-restarts`                                                                                                       | P1       | 3+ container restarts within 10 minutes — needs a container restart policy to be meaningful             |
| `cpu-high` / `memory-high`                                                                                            | P2       | Average above 80 % of the task size for 10 minutes                                                      |
| `dynamo-throttle-read-*` / `-write-*` (5 of the 8 tables — trace-map, session-epoch and stream-lease are not alarmed) | P1       | Any throttle event in a minute                                                                          |
| `dynamo-ttl-stalled-*` (lease, dedup, origin)                                                                         | P2       | Zero TTL deletions in 24 hours (also fires on an idle table)                                            |
| `elf-lag-high`                                                                                                        | P2       | `argus_elf_lag_seconds` above 600 s for 5 minutes — needs `CLOUD_METRICS=cloudwatch` on the ELF runtime |
| `health-fail`                                                                                                         | P1       | An unhealthy verdict logged in each of 2 consecutive minutes                                            |

Hermes-level alerting — `hermes_org_connected`, `hermes_org_auth_failures_total` — lives in your OTel backend, not CloudWatch; see [Self-Instrumentation Metrics](https://docs.pharos.ai/open-telemetry/configuration/self-metrics.md) and [Dashboards & Alerts](https://docs.pharos.ai/open-telemetry/configuration/dashboards-and-alerts.md).

***

## Cost

Typical monthly AWS spend at June 2026 `us-west-1` list prices:

| Component                             | Small (5 orgs, 1 task 0.5 vCPU/1 GiB) | Medium (20 orgs, 1 task 1 vCPU/2 GiB) | Large (50 orgs, 2 tasks 1 vCPU/2 GiB) |
| ------------------------------------- | ------------------------------------- | ------------------------------------- | ------------------------------------- |
| ECS Fargate                           | $20–21                                | $41–42                                | $82–84                                |
| NAT gateway (hours + data)            | $35–36                                | $35–36                                | $35–37                                |
| DynamoDB (on-demand)                  | < $0.10                               | < $1                                  | $1–5                                  |
| Secrets Manager, CloudWatch Logs, ECR | < $1                                  | < $1                                  | $1–3                                  |
| **Total**                             | **$56–58**                            | **$77–80**                            | **$120–125**                          |

Fargate cost scales with task size, not org count; the NAT gateway is the largest fixed line in the private-subnet stack. Tag resources with a project tag and set an AWS Budget on it; use Cost Explorer grouped by service to see drift. The full cost model — compute formulas, load units, network cost at scale — is in [Sizing & Cost](https://docs.pharos.ai/open-telemetry/hermes/sizing-and-cost.md).

***

## Teardown

`terraform -chdir=3-runtime destroy` removes all compute (ECS, VPC, NAT, IAM) while preserving DynamoDB state and the ECR image; residual cost is well under a dollar a month, and a later `apply` resumes exactly where the cursors left off. For full decommission, destroy `4-alerts`, then `3-runtime`, then `2-foundation`, then `1-registry`.

***

## Troubleshooting

| Symptom                                                            | Check                                                                                                                                                                                                                                                                                 |
| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Task keeps restarting every \~2 minutes                            | The container failed its health check: read the log group for the first `ERROR`. Usually an empty or malformed `org-configs` secret, or an org skipped at startup — see [Operations](https://docs.pharos.ai/open-telemetry/configuration/operations.md#an-org-is-missing-from-stats). |
| `CannotPullContainerError`                                         | The image URI in `terraform.tfvars` does not exist in ECR, or the execution role lacks pull permission. Verify the tag/digest with `aws ecr describe-images`.                                                                                                                         |
| `ResourceInitializationError … logs:CreateLogStream`               | The execution role cannot write the log group; re-apply Stage 3.                                                                                                                                                                                                                      |
| Salesforce auth fails with `app_not_found` / `tenantid is invalid` | Wrong `sf.login_url` for the org type (`test.salesforce.com` for sandboxes), or the consumer key belongs to a different org.                                                                                                                                                          |
| Subscriber opens but no events arrive                              | `Export_Mode__c` is `Off`, the running user lacks Platform Event access, or the org is emitting nothing yet.                                                                                                                                                                          |
| `/health` returns 503                                              | Fewer orgs connected than `HEALTH_MULTI_ORG_MODE` requires, or no event within the health event threshold; check `/stats` per org.                                                                                                                                                    |
| DynamoDB throttling alarms                                         | On-demand tables should not throttle at Hermes volumes; if they do, a backfill or a burst of ELF files is running — pause backfill or raise ELF parallelism gradually.                                                                                                                |
| TTL not deleting rows                                              | Confirm TTL is enabled on the lease, dedup, and origin tables with the attribute the module specifies; TTL deletes can lag up to 48 hours.                                                                                                                                            |
| Costs jumped                                                       | Check NAT data processing (an ELF backfill moves gigabytes) and CloudWatch Logs ingestion (`LOG_LEVEL=debug` left on).                                                                                                                                                                |

***

## Related

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>☁️ Salesforce Setup</strong></td><td>External Client App, JWT auth, and OTel_Configuration__mdt</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/salesforce-setup.md">Salesforce Setup</a></td></tr><tr><td><strong>🧩 ORG_CONFIGS Reference</strong></td><td>The document you place in Secrets Manager</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/org-configs.md">ORG_CONFIGS Reference</a></td></tr><tr><td><strong>🏷️ Versioning &#x26; Upgrades</strong></td><td>Choosing a tag, upgrade order, rollback</td><td><a href="https://docs.pharos.ai/open-telemetry/configuration/versioning-and-upgrades.md">Versioning &amp; Upgrades</a></td></tr><tr><td><strong>📐 Sizing &#x26; Cost</strong></td><td>Compute shapes and the cost model behind the table above</td><td><a href="https://docs.pharos.ai/open-telemetry/hermes/sizing-and-cost.md">Sizing &amp; Cost</a></td></tr></tbody></table>
