> For the complete documentation index, see [llms.txt](https://docs.pharos.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/alerting/notification-channels/pagerduty.md).

# PagerDuty

Configure PagerDuty integration for incident management and on-call escalation.

PagerDuty transforms critical Pharos alerts into actionable incidents with on-call escalation, ensuring that urgent production issues receive immediate human attention—even at 2 AM. However, PagerDuty should be reserved exclusively for true emergencies to avoid on-call burnout.

## Why PagerDuty for Observability

PagerDuty serves a distinct purpose from Slack/Teams notifications:

**On-Call Escalation:** Automated escalation through on-call rotations ensures someone responds, even outside business hours.

**Multi-Channel Alerting:** Phone calls, SMS, push notifications, and email ensure on-call engineers are reached regardless of their current state.

**Acknowledgment Tracking:** Visibility into who acknowledged the incident and when, preventing "someone else will handle it" syndrome.

**Auto-Escalation:** If not acknowledged within defined timeframe, automatically escalates to next level (team lead, manager, VP).

**Incident Lifecycle:** Track incidents from creation → acknowledgment → investigation → resolution, providing clear accountability.

**Integration Ecosystem:** Connects to Slack, Teams, Jira, ServiceNow, Zoom (war rooms), and other incident management tools.

{% hint style="danger" %}
**Reserve PagerDuty for True Emergencies**

PagerDuty should **only** be used for incidents requiring immediate response, typically:

* User-facing failures affecting customers
* Revenue-impacting issues (payment processing, order management)
* Security incidents requiring immediate containment
* Complete system outages

If it can wait until morning, **don't page someone**.
{% endhint %}

## Integration Setup

PagerDuty is the one channel that needs **two** credentials, and confusing them is the usual reason a connection fails:

* A **REST API token**, which lets Pharos talk to your PagerDuty account and enumerate its services
* A **Service Integration Key** from an Events API v2 integration on the specific service you want incidents raised against

The API token authenticates the account-level connection; the integration key routes incidents to one service. Pharos needs the first to succeed before it can offer you the second.

Beyond the credentials, PagerDuty itself has to be set up to do anything useful with an incident: the target service needs an escalation policy with an on-call schedule attached. Without one, incidents are created and then sit there unrouted — which looks like a Pharos failure but isn't.

For the credential setup and where each value comes from in PagerDuty, see [PagerDuty](https://docs.pharos.ai/configuration-guide/integrations/pagerduty.md).

## How PagerDuty Incidents Work

When a Pharos notification rule creates a PagerDuty incident:

1. **Incident Created** in PagerDuty with details from Pharos
2. **On-Call Engineer Paged** via push notification, SMS, phone call
3. **Acknowledgment Required** within defined timeframe (e.g., 5 minutes)
4. **Escalation if Not Acknowledged** to next level (team lead, manager)
5. **Incident Updated** with notes and status changes
6. **Incident Resolved** when issue is fixed

**PagerDuty Escalation Policy Example:**

```
Level 1: On-Call Developer
  - Page via push notification
  - If not acknowledged in 5 minutes → Escalate to Level 2

Level 2: On-Call Team Lead
  - Page via push + SMS
  - If not acknowledged in 10 minutes → Escalate to Level 3

Level 3: Engineering Manager
  - Page via push + SMS + phone call
  - Immediate notification (no timeout)
```

This ensures critical issues are addressed, even if the primary on-call person is unavailable.

## PagerDuty Notification Configuration

PagerDuty notifications are configured in Pharos notification rules with PagerDuty-specific actions.

**Basic Configuration:**

```
Rule Type: Notification (Is_Update__c = false)
Filter: Priority = "Highest" AND Organization = "Production"
Action: Create PagerDuty Incident
  - Service: Production Critical
  - Urgency: High
  - Include details: Issue summary, link, impact metrics
```

**Incident Details:** Pharos automatically populates PagerDuty incidents with:

* **Title:** Issue summary
* **Description:** Error details, category, functional area
* **Urgency:** High or Low (configurable per rule)
* **Details:** Custom fields like affected users, occurrence count
* **Links:** Direct link back to Pharos issue

**Merge Fields:** Use merge fields to customize incident titles:

```
Title: [Production] {!Issue.Category__c} - {!Issue.Functional_Area__c}
       
Example: [Production] Apex - Payment Processing
```

## When to Use PagerDuty

PagerDuty should be reserved for incidents that meet ALL of these criteria:

**1. Time-Sensitive:** Requires immediate response (can't wait until morning)

**2. User-Impacting:** Affects end users or customers directly

**3. Revenue-Impacting:** Has financial consequences if not addressed

**4. Requires Human Intervention:** Can't be automatically resolved or retried

### Valid Use Case Examples

Each of the following scenarios meets all four criteria above and genuinely warrants a page.

#### Use Case 1: Payment Processing Failures

**Scenario:** Integration with payment gateway fails, blocking customer orders.

**PagerDuty Configuration:**

```
Rule (Issue Created):
  Filter:
    Functional Area contains "Payment"
    AND Category = "Apex"
    AND Organization = "Production"
  Action: PagerDuty Incident (High Urgency)

Escalation Policy:
  Level 1: On-call integration engineer (5 min)
  Level 2: Integration team lead (10 min)
  Level 3: VP Engineering (immediate)
```

**Why This Warrants Paging:**

* Every failed payment = lost revenue
* Customer experience severely impacted
* Requires immediate investigation and vendor escalation
* Can't wait until business hours

#### Use Case 2: Core Functionality Outage

**Scenario:** Primary application flow fails affecting all users.

**PagerDuty Configuration:**

```
Rule (Issue Created):
  Filter:
    Priority = "Highest"
    AND Impacted Users Count > 50
    AND Organization = "Production"
  Action: PagerDuty Incident (High Urgency)
```

**Why This Warrants Paging:**

* Widespread user impact
* Core business process broken
* Company reputation at risk
* Immediate response required

#### Use Case 3: Security Incident

**Scenario:** Security exception indicating potential breach or unauthorized access.

**PagerDuty Configuration:**

```
Rule (Issue Created):
  Filter:
    Log Level = "ERROR"
    AND Organization = "Production"
  Action: PagerDuty Incident (High Urgency)
```

**Why This Warrants Paging:**

* Potential data breach
* Compliance requirements (immediate notification)
* Time-sensitive containment required
* Can't wait for triage

#### Use Case 4: High-Volume Escalation

**Scenario:** Existing issue suddenly spikes in frequency, indicating widespread problem.

**PagerDuty Configuration:**

```
Rule (Issue Updated):
  Filter:
    Log Count > 100 (within 1 hour)
    AND Organization = "Production"
  Action: PagerDuty Incident (High Urgency)
```

**Why This Warrants Paging:**

* Rapid escalation suggests systemic issue
* May affect large portion of user base
* Early intervention can prevent wider impact

## When NOT to Use PagerDuty

Knowing when not to page matters as much as knowing when to. The scenarios below rarely justify interrupting someone, and each has a better-suited channel.

### Non-Urgent Scenarios

**Batch Job Failures:**

```
❌ Don't Page:
Nightly batch job fails (can be reviewed next morning)

✓ Use Instead:
Slack notification or email digest
```

**Non-Critical Errors:**

```
❌ Don't Page:
Medium-priority flow error affecting 2 users

✓ Use Instead:
Slack notification to functional team
```

**Development/Sandbox Issues:**

```
❌ Don't Page:
Any error in non-production environments

✓ Use Instead:
Slack notification or no notification
```

**Known Issues Being Fixed:**

```
❌ Don't Page:
Issue already identified, fix in progress

✓ Use Instead:
Temporarily disable PagerDuty rule for this issue
```

### The Cost of Over-Paging

Paging too often carries real costs for the people on the other end of it.

{% hint style="warning" %}
**Avoid PagerDuty Fatigue**

Over-paging causes:

* **Burnout:** On-call engineers become exhausted and resentful
* **Desensitization:** Engineers start ignoring pages, assuming they're not urgent
* **Turnover:** Excessive on-call burden drives engineers to leave
* **Missed Critical Issues:** When every issue pages, truly critical ones get lost

**Rule of Thumb:** If your on-call engineer gets paged more than 2-3 times per week, you're over-paging. Tighten your PagerDuty filters.
{% endhint %}

## PagerDuty Best Practices

These practices keep PagerDuty effective without exhausting your on-call team.

### Practice 1: Start Conservative, Expand Carefully

**Week 1-2: No PagerDuty**

```
Use only Slack/Teams notifications
Observe which issues would warrant paging
Build baseline understanding of error patterns
```

**Week 3-4: PagerDuty for Critical Only**

```
Rule: Priority = "Highest" AND Impacted Users Count > 50
Expected Volume: <1 page per day
Monitor false positive rate
```

**Month 2+: Refine Thresholds**

```
Add specific functional area rules (payment processing, etc.)
Adjust thresholds based on actual data
Continue monitoring page volume
```

### Practice 2: Multi-Channel Notifications

Always pair PagerDuty with Slack/Teams:

```
Rule Configuration:
  Filter: Critical conditions
  Actions:
    1. PagerDuty Incident (pages on-call)
    2. Slack #prod-critical (team awareness)

Why: 
  - On-call gets paged
  - Rest of team sees issue in Slack
  - Team can collaborate on response
  - Visibility even if PagerDuty fails
```

### Practice 3: Clear Incident Titles

Make PagerDuty incident titles immediately actionable:

**Good:**

```
[Production] Payment Gateway Timeout - 47 Transactions Failed
[Critical] Apex Exception in OpportunityTriggerHandler - 120 Users Affected
```

**Poor:**

```
New Issue Created
Apex Exception
Production Error
```

**Why:** On-call engineer woken at 3 AM needs to immediately understand severity and scope.

### Practice 4: Include Direct Links

Always include direct link to Pharos in incident details:

```
Incident Description:
  Error: Apex NullPointerException in OpportunityTriggerHandler
  Functional Area: Sales
  Users Impacted: 23
  Occurrences: 8 in last hour
  
  View in Pharos: https://yourinstance.salesforce.com/apex/pharos__issueDetail?id=a1b2c3
```

**Why:** Engineer can immediately jump to full details without searching.

### Practice 5: Define Escalation Policies

Don't rely on a single on-call person:

**Recommended Escalation:**

```
Level 1: Primary On-Call (5 min timeout)
Level 2: Secondary On-Call (10 min timeout)
Level 3: Team Lead (15 min timeout)
Level 4: Engineering Manager (immediate)
```

**Why:** Ensures response even if primary is unavailable (traveling, poor signal, etc.)

### Practice 6: Regular Review and Refinement

**Weekly:**

* Review all PagerDuty incidents from past week
* Identify false positives (pages that weren't truly urgent)
* Adjust thresholds to reduce false positives

**Monthly:**

* Analyze page volume trends
* Review on-call engineer feedback
* Refine rules based on incident outcomes

**Quarterly:**

* Review escalation policies
* Update on-call rotations
* Assess overall paging strategy effectiveness

## Combining PagerDuty with Update Rules

One powerful pattern is using PagerDuty for escalation when issues grow:

**Strategy: Initial Awareness + Escalation**

```
Rule 1 (Issue Created):
  Filter: Priority = "High" OR Functional Area = "Payment"
  Action: Slack #prod-errors (team awareness, no page)

Rule 2 (Issue Updated):
  Filter: Log Count > 50 OR Impacted Users Count > 20
  Action: PagerDuty + Slack #prod-critical (escalate to on-call)

How It Works:
  1. High-priority issue created → Slack notification only
  2. Team sees notification, begins investigation during business hours
  3. If issue crosses threshold (50 occurrences or 20 users) → Page on-call
  4. Escalation indicates problem is more severe than initially assessed
```

**Benefits:**

* Not every high-priority issue pages (reduces fatigue)
* Team has opportunity to address during business hours
* Only truly escalating problems page
* On-call reserved for issues that crossed severity threshold

## PagerDuty Metrics to Monitor

Track these metrics to ensure healthy PagerDuty usage:

**Page Volume:**

* **Ideal:** <3 pages per week per on-call engineer
* **Acceptable:** 3-7 pages per week
* **Problem:** >7 pages per week (refine filters)

**False Positive Rate:**

* **Ideal:** <10% of pages are non-urgent
* **Acceptable:** 10-20% false positives
* **Problem:** >20% false positives (tighten criteria)

**Acknowledgment Time:**

* **Target:** <15 minutes for all pages
* **Red Flag:** Pages sitting unacknowledged for hours

**Escalation Rate:**

* **Healthy:** <20% of incidents escalate past Level 1
* **Problem:** >50% escalate (primary on-call may be unavailable too often)

## Real-World Example

**Organization:** E-commerce company, $50M annual revenue, 40-person engineering team

**Initial Setup (Over-Paging):**

```
Rule: All Priority = "High" issues → PagerDuty
Volume: 12-15 pages per week
Feedback: "I'm exhausted. Half these pages aren't urgent."
Result: On-call engineer quit after 3 months
```

**Revised Setup (Conservative):**

```
Rule 1 (PagerDuty):
  Filter:
    (Functional Area = "Payment" OR Functional Area = "Checkout")
    AND Category = "Apex"
    AND Organization = "Production"
  Action: PagerDuty High Urgency + Slack #prod-critical

Rule 2 (PagerDuty Escalation):
  Filter:
    Log Count > 100 in 1 hour
    AND Organization = "Production"
  Action: PagerDuty High Urgency + Slack #escalations

Rule 3 (Slack Only):
  Filter: Priority = "High"
  Action: Slack #prod-high-priority

Volume: 2-4 pages per week
Feedback: "Much better. Pages are actually urgent now."
Result: On-call engineers feel respected, response times improved
```

## Summary

PagerDuty is the most powerful—and most dangerous—notification channel:

**Use PagerDuty For:**

* User-facing critical failures
* Revenue-impacting issues
* Security incidents
* Complete system outages
* Issues requiring immediate response (can't wait until morning)

**Don't Use PagerDuty For:**

* Batch job failures
* Non-production environments
* Medium/low-priority issues
* Issues that can wait until business hours
* High-volume error patterns (causes fatigue)

**Best Practices:**

* Start with no PagerDuty, add conservatively
* Always pair with Slack/Teams notification
* Target <3 pages per week per engineer
* Monitor false positive rate (<10%)
* Define clear escalation policies
* Review and refine monthly

**Key Takeaway:** PagerDuty is for **emergencies only**. When in doubt, use Slack/Teams instead. It's better to under-page and miss an occasional issue than to over-page and burn out your on-call team.

For other notification channels, see [Slack](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/alerting/notification-channels/slack.md), [Microsoft Teams](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/alerting/notification-channels/teams.md), [Email](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/alerting/notification-channels/email.md), and [Webhooks](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/alerting/notification-channels/webhooks.md).
