> For the complete documentation index, see [llms.txt](https://docs.pharos.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/alerting/alerting-use-cases.md).

# Alerting Use Cases

Explore different use cases for leveraging notifications and recommended alert types for various situations.

Different teams have different alerting needs based on their workflows, team structure, and operational maturity. This guide explores common use cases and recommends notification strategies for each scenario.

## Awareness Layer vs. Triage Process

Pharos alerting serves two complementary purposes in your error management workflow:

**Active Monitoring (Notifications):** Real-time awareness that something just happened, delivered immediately to your communication channels. This ensures problems don't sit unnoticed for days or weeks.

**Passive Monitoring (Triage Process):** Scheduled reviews where team members proactively examine issues in Pharos, assign priorities, and action items. Typically done weekly or bi-weekly.

{% hint style="success" %}
**Best Practice: Use Both**

Notifications provide immediate awareness for urgent issues, while regular triage sessions ensure nothing falls through the cracks. They work together to create a comprehensive awareness layer.
{% endhint %}

The notification layer bridges the gap between error occurrence and team awareness. Without it, teams operate reactively—only learning about problems when users escalate them or during scheduled triage sessions.

## Common Alerting Scenarios

The right notification strategy depends on your team's size, structure, and environment mix. These scenarios illustrate common starting points.

### Scenario 1: Small Development Team (5-10 people)

**Context:** Single Salesforce org in production, small team wearing multiple hats, limited specialization.

**Challenges:**

* Everyone needs visibility into all errors
* Can't afford to miss critical issues
* Don't want to be overwhelmed with noise

**Recommended Setup:**

```
Single Slack Channel: #production-errors

Rule 1 (Create):
  Filter: (none - catch all)
  Action: Post to #production-errors
  
Result: Every new issue → one notification → team discusses in thread
```

**Why This Works:**

* Simple to maintain
* Complete visibility for everyone
* Team can discuss issues directly in Slack threads
* Low setup overhead

**When to Evolve:** As team grows beyond 10 people or error volume exceeds \~20 issues per week, consider splitting by severity or functional area.

### Scenario 2: Large Enterprise Team (50+ people)

**Context:** Multiple functional areas, specialized teams, high error volume, 18+ distinct business processes.

**Challenges:**

* Sales team doesn't need service cloud alerts
* Too many channels creates confusion
* Need clear ownership and accountability
* Risk of critical issues getting lost in noise

**Recommended Setup:**

```
Functional Area Channels:

#prod-sales-errors → Sales automation team
#prod-service-errors → Service cloud team  
#prod-marketing-errors → Marketing automation team
#prod-integration-errors → Middleware team
#prod-platform-errors → Platform/infrastructure team

PLUS Cross-Functional:

#prod-critical → All team leads (only critical/high-impact)
```

**Rule Strategy:**

```
Rule 1 (Create): 
  Priority = "Highest" OR Impacted Users Count > 50 
  → #prod-critical + PagerDuty

Rule 2 (Create):
  Functional Area contains "Sales"
  → #prod-sales-errors

Rule 3 (Create):
  Functional Area contains "Service"  
  → #prod-service-errors

[... additional functional area rules ...]

Rule N (Create):
  (catch-all for unmapped functional areas)
  → #prod-platform-errors
```

**Why This Works:**

* Notifications reach the right team immediately
* Reduces noise for each team (only see relevant errors)
* Critical channel ensures high-priority issues have cross-functional visibility
* Catch-all prevents issues from being missed

{% hint style="warning" %}
**Decision Point: Channel Granularity**

During implementation, teams often debate: "Should we have 18 separate channels (one per functional area) or group them?"

**Real-world guidance:** Start with broader groups (4-8 channels based on team structure), then split later if specific teams request their own channel. Creating channels is easy; getting teams to monitor them is hard.
{% endhint %}

### Scenario 3: Multiple Environments (Production + UAT + Sandbox)

**Context:** Active development in UAT, production monitoring required, sandbox used for testing.

**Challenges:**

* Don't want sandbox noise in production channels
* UAT errors need visibility during releases
* Production must remain top priority

**Recommended Setup:**

```
Production:
  #prod-critical → Critical issues
  #prod-errors → All other production issues

UAT:
  #uat-errors → UAT-specific channel
  OR
  Same functional channels with "[UAT]" prefix in message

Sandbox:
  No real-time alerts (use log-level during active dev, then disable)
  OR
  Daily email digest for awareness
```

**Rule Strategy:**

```
Production Rules:
  Rule 1: Organization = "Production" AND Priority = "Highest" → #prod-critical
  Rule 2: Organization = "Production" → #prod-errors

UAT Rules:
  Rule 3: Organization = "UAT" → #uat-errors

Sandbox:
  (No active notification rules or email digest only)
```

**Why This Works:**

* Clear separation prevents prod/UAT confusion
* Production gets primary focus
* UAT visibility during deployment windows
* Sandbox doesn't create noise (use triage process instead)

### Scenario 4: Integration-Heavy Organization

**Context:** Multiple external system integrations (ERP, marketing automation, payment processors, data warehouses).

**Challenges:**

* Integration failures often require vendor escalation
* Need to distinguish between Salesforce issues and external system issues
* Some failures require immediate action (payment processing), others can wait

**Recommended Setup:**

```
#prod-integration-critical → Payment processors, customer-facing APIs
#prod-integration-errors → All other integration issues
#prod-internal-errors → Salesforce-only issues (no external deps)
```

**Rule Strategy:**

```
Rule 1 (Create):
  Functional Area = "Integration"
  AND External System contains "Payment"
  → #prod-integration-critical + PagerDuty

Rule 2 (Create):
  Functional Area = "Integration"
  → #prod-integration-errors
  
Rule 3 (Update):
  Functional Area = "Integration"
  AND Log Count > 10
  → #prod-integration-critical (escalation)

Rule 4 (Create):
  (catch-all)
  → #prod-internal-errors
```

**Additional Capability: Webhooks for Vendor Escalation**

```
Rule (Create):
  Functional Area = "Integration"
  AND External System = "PaymentGateway"
  → Webhook to vendor monitoring API
```

**Why This Works:**

* Clear ownership (integration team vs. application team)
* Critical integrations get immediate attention
* Escalation for recurring integration issues
* Webhook integration automates vendor notification

### Scenario 5: Issue Triaging Workflow

**Context:** Team performs regular triage sessions (weekly or bi-weekly) to review and prioritize issues.

**Challenges:**

* Not every issue needs immediate notification
* Want notifications only for issues that cross specific thresholds
* Don't want duplicate awareness (notification + triage session)

**Recommended Setup:**

```
Threshold-Based Notifications:

#prod-high-priority → Only issues with significant impact
```

**Rule Strategy:**

```
Rule 1 (Create):
  Impacted Users Count > 20 OR Log Count > 50 on first occurrence
  → #prod-high-priority

Rule 2 (Update):
  Log Count > 100 (escalation threshold)
  → #prod-escalations

Everything else: No notification, rely on triage process
```

**Triage Process:**

* Weekly review of all new/untriaged issues
* Team assigns priorities in Pharos
* Creates Jira tickets for actionable items
* Notifications only for urgent/high-impact scenarios

**Why This Works:**

* Reduces notification fatigue
* Empowers team to proactively manage issues during triage
* Notifications reserved for truly urgent scenarios
* Balances active and passive monitoring

## Notification Timing Strategies

Beyond who receives an alert, when it arrives matters. Match the timing to the urgency of the issue.

### Real-Time Notifications (Immediate)

Send these the moment an issue appears, for problems that can't wait.

**When to Use:**

* Production critical errors
* User-facing failures
* Integration issues with external systems
* Security exceptions

**Channel Recommendations:**

* Slack/Teams for team awareness
* PagerDuty for on-call escalation

**Example:**

```
Rule (Create):
  Priority = "Highest" AND Organization = "Production"
  → Immediate Slack notification
```

### Digest Notifications (Scheduled)

Batch non-urgent issues into scheduled summaries instead of interrupting the team.

**When to Use:**

* Batch job failures (can be reviewed next morning)
* Non-critical errors in non-production
* Low-priority issues for awareness only

**Channel Recommendations:**

* Email (daily or weekly digest)
* Slack summary message (once per day)

**Example:**

```
Rule (Scheduled Daily 8 AM):
  Category = "Apex" AND Created Yesterday
  → Email digest to data-team@company.com
```

### Escalation Notifications (Threshold-Based)

Trigger these only when an existing issue crosses a threshold that signals it's getting worse.

**When to Use:**

* Issue volume exceeds normal levels
* User impact grows beyond initial assessment
* Issue frequency increases significantly

**Channel Recommendations:**

* Slack/Teams escalation channel
* PagerDuty for severe escalations

**Example:**

```
Rule (Update):
  Log Count > 100 OR Impacted Users Count > 50
  → Escalate to #prod-escalations
```

## Functional Area Routing Patterns

Once you route beyond a single channel, choose a pattern for mapping issues to the teams that own them.

### Pattern 1: Direct Functional Area Mapping

Map Pharos functional areas directly to team ownership:

```
Functional Area in Pharos → Team Channel
  "Opportunity" → #sales-team
  "Case" → #service-team
  "Campaign" → #marketing-team
  "Integration" → #middleware-team
```

**Pros:** Simple, clear ownership **Cons:** Requires consistent functional area tagging

### Pattern 2: Business Process Grouping

Group related functional areas into broader business processes:

```
Sales Process (#sales-automation):
  - Opportunity
  - Quote
  - Contract
  - Account

Service Process (#service-automation):
  - Case
  - Knowledge
  - Entitlement
```

**Pros:** Fewer channels, aligned with business structure **Cons:** Broader notifications (more noise per team)

### Pattern 3: Technical Ownership

Route by technical capability rather than business area:

```
#dev-apex-errors → Apex developers
#admin-flow-errors → Admin team
#integration-errors → Integration specialists
#platform-errors → Platform team
```

**Pros:** Clear technical skill alignment **Cons:** May not align with business ownership

### Pattern 4: Hybrid Approach

Combine business and technical routing:

```
Critical Issues:
  #prod-critical → Cross-functional visibility

Functional Areas:
  #sales-automation → Sales business process
  #service-automation → Service business process

Technical Specialty:
  #integration-errors → Integration team
  #platform-errors → Platform/infrastructure
```

**Pros:** Balances business and technical needs **Cons:** More complex rule configuration

## Common Anti-Patterns to Avoid

A handful of configurations reliably undermine alerting. Watch for these.

### Anti-Pattern 1: Notification for Every Error

```
❌ Bad Setup:
  Log-level rule → Every log occurrence → Slack notification
  
Result: 500 occurrences of same error = 500 notifications
```

**Solution:** Use issue-level notifications to group identical errors.

### Anti-Pattern 2: No Filtering (Catch-All Everything)

```
❌ Bad Setup:
  Single rule with no filters → All issues to one channel
  
Result: High-volume notifications, critical issues lost in noise
```

**Solution:** Add severity, priority, or functional area filters.

### Anti-Pattern 3: Too Many Channels

```
❌ Bad Setup:
  18 functional areas = 18 separate Slack channels
  
Result: Team fatigue from monitoring too many channels
```

**Solution:** Start with 4-6 grouped channels, split only if teams request it.

### Anti-Pattern 4: Mixing Development and Production

```
❌ Bad Setup:
  Sandbox and Production errors in same #errors channel
  
Result: Production urgency diluted by sandbox noise
```

**Solution:** Separate channels or disable notifications for non-production.

### Anti-Pattern 5: PagerDuty for Everything

```
❌ Bad Setup:
  Every issue creates a PagerDuty incident
  
Result: On-call burnout, desensitization to real emergencies
```

**Solution:** Reserve PagerDuty for true emergencies (user-facing, revenue-impacting).

## Escalation Ladders

Advanced teams implement multi-tier notification strategies that escalate based on issue severity:

### Tier 1: Awareness (Low Priority)

```
Trigger: Issue created
Filter: No specific severity
Action: Post to #prod-monitoring (no mentions)
Purpose: Visibility, reviewed during triage
```

### Tier 2: Action Required (Medium Priority)

```
Trigger: Issue created OR Issue updated (Log Count > 25)
Filter: Impacted Users Count > 5 OR Functional Area = "Critical Process"
Action: Post to #prod-errors (with @channel mention)
Purpose: Team awareness, action expected within 24 hours
```

### Tier 3: Urgent (High Priority)

```
Trigger: Issue created OR Issue updated (Rapid occurrence growth)
Filter: Impacted Users Count > 20 OR Log Count > 50 in 1 hour
Action: Post to #prod-urgent + PagerDuty (low urgency)
Purpose: Immediate investigation required
```

### Tier 4: Critical (Emergency)

```
Trigger: Issue created
Filter: Priority = "Highest" OR Revenue-impacting functional area
Action: PagerDuty (high urgency) + Slack #prod-critical (@channel)
Purpose: Immediate response, escalation if not acknowledged
```

**Why Ladders Work:**

* Proportional response to issue severity
* Clear expectations for each tier
* Prevents over-alerting and under-alerting

## Summary

Effective alerting strategies balance immediate awareness with sustainable team practices:

**Key Principles:**

1. **Start simple:** Single catch-all channel, add complexity as needed
2. **Filter by impact:** User-facing and high-volume issues deserve immediate attention
3. **Respect team boundaries:** Route notifications to appropriate functional owners
4. **Avoid fatigue:** Reserve PagerDuty and urgent channels for true emergencies
5. **Iterate:** Review notification effectiveness monthly, adjust thresholds

**Recommended Starting Point:**

```
Week 1-2: Single Slack channel, all production issues
Week 3-4: Add severity or functional area filtering
Month 2: Introduce PagerDuty for critical only
Ongoing: Refine thresholds based on team feedback
```

For implementation details, see [Notification Channels](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/alerting/notification-channels.md) and [Issue-level Alerts](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/alerting/issue-level-alerts.md).
