> For the complete documentation index, see [llms.txt](https://docs.pharos.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/alerting/order-of-execution.md).

# Order of Execution

Understand rule ordering to avoid excessive noise and implement ladder system notifications.

When multiple notification rules are configured, Pharos evaluates them in a specific order to ensure only one notification is sent per event. Understanding this execution order is critical for avoiding notification spam and implementing effective escalation strategies.

## Why Order Matters

Without controlled execution order, a single issue could trigger multiple notifications:

**Example Without Order Control:**

```
Rule 1: Category = "Flow" → Slack #flow-errors
Rule 2: Functional Area = "Sales" → Slack #sales-team
Rule 3: All Issues → Slack #prod-errors

Result: One sales flow error triggers 3 notifications
```

This creates duplicate alerts for the same problem, leading to notification fatigue and confusion about which channel should handle the issue.

## The Stop-on-First-Match Principle

Pharos notification rules follow a **stop-on-first-match** execution model:

1. Rules are evaluated **top to bottom** in the order shown in the UI
2. When a rule's filter criteria is satisfied, that notification is sent
3. Evaluation **stops immediately**—remaining rules are not checked
4. Only **one notification** is sent per log or issue creation

This is similar to Salesforce lead assignment rules: the first matching rule wins, and subsequent rules are never evaluated for that record.

{% hint style="info" %}
**Single Notification Per Event**

Each issue creation or log creation triggers at most one notification. This is deliberate to prevent notification spam and ensure teams know which channel owns each issue.
{% endhint %}

## Rule Ordering in the UI

The notification rules interface displays rules in a table with up/down arrows for reordering.

**Visual Representation:**

```mermaid
graph TD
    subgraph Order["Evaluation Order - Top to Bottom"]
        R1[Rule 1: Critical Issues → PagerDuty]
        R2[Rule 2: Sales Errors → #sales-team]
        R3[Rule 3: Service Errors → #service-team]
        R4[Rule 4: Catch-All → #platform-team]
    end
    R1 --> R2 --> R3 --> R4
```

The rule at the top (Rule 1) is evaluated first. If its criteria match, Rules 2-4 never run for that issue.

## Best Practice: Specific to General

The golden rule of notification ordering: **most specific rules at the top, catch-all rules at the bottom**.

### Example: Correct Ordering

```
Rule 1 (Top):
  Priority = "Highest" → PagerDuty + Slack #urgent
  
Rule 2:
  Functional Area contains "Sales" AND Category = "Flow"
  → Slack #sales-automation
  
Rule 3:
  Functional Area contains "Sales"
  → Slack #sales-team
  
Rule 4:
  Category = "Flow"
  → Slack #flow-errors
  
Rule 5 (Bottom):
  (No filters - catch all)
  → Slack #prod-errors
```

**How This Works:**

* Critical issues go to PagerDuty (regardless of functional area)
* Sales flow errors go to #sales-automation (specific combination)
* Other sales errors go to #sales-team (broader category)
* Non-sales flow errors go to #flow-errors
* Everything else goes to #prod-errors

### Example: Incorrect Ordering

```
❌ Rule 1 (Top):
  (No filters - catch all)
  → Slack #prod-errors
  
Rule 2:
  Priority = "Highest"
  → PagerDuty
```

**Problem:** Rule 1 matches everything, so Rule 2 never executes. Critical issues never reach PagerDuty because they're caught by the catch-all rule first.

## Real-World Example from Training

During a customer training session, the facilitator demonstrated this principle with two rules:

**Initial Setup (Incorrect):**

```
Rule 1: (No filters) → Send notification with generic template
Rule 2: Category = "Flow" → Send notification with flow-specific template

Result: All issues matched Rule 1, flow-specific template never used
```

**After Reordering (Correct):**

```
Rule 1: Category = "Flow" → Flow-specific template
Rule 2: (No filters) → Generic template

Result: Flow errors use specific template, everything else uses generic
```

The facilitator explained: *"So this flow notification is all of a sudden at the top, and this is general practice as well. When you have a rule that is very specific, like this one that has some specific criteria, it should be closer to the top, whereas stuff that's less specific, like this has no filters, should be towards the bottom."*

{% hint style="warning" %}
**Catch-All Rules Always Last**

If you have a catch-all rule (no filter criteria), it must be the last rule. Otherwise, it will match every issue and prevent all subsequent rules from ever executing.
{% endhint %}

## Ladder System Notifications

Advanced teams implement **notification ladders**—multi-tier escalation strategies that route notifications based on severity and impact.

### What Is a Notification Ladder?

A notification ladder uses rule ordering to create escalation tiers:

**Tier 1 (Top):** Emergencies → Immediate response required **Tier 2:** High priority → Action expected within hours **Tier 3:** Medium priority → Review within 24 hours **Tier 4 (Bottom):** Catch-all → Triage during scheduled reviews

### Example Ladder Implementation

```
Rule 1: Emergency Escalation
  Filter: 
    Priority = "Highest"
    OR Impacted Users Count > 50
    OR (Functional Area = "Payment" AND Category = "Apex")
  Action: PagerDuty (high urgency) + Slack #prod-critical
  
Rule 2: High-Volume Escalation  
  Filter:
    Log Count > 100 (for issue-update rules)
    OR Frequency < 1 (happening multiple times per day)
  Action: Slack #prod-escalations
  
Rule 3: Functional Area - Sales
  Filter: Functional Area contains "Sales"
  Action: Slack #sales-team
  
Rule 4: Functional Area - Service
  Filter: Functional Area contains "Service"
  Action: Slack #service-team
  
Rule 5: Integration Errors
  Filter: Functional Area = "Integration"
  Action: Slack #integration-team
  
Rule 6: Catch-All
  Filter: (none)
  Action: Slack #prod-errors
```

**How the Ladder Works:**

Example 1: Critical payment integration failure

* Matches Rule 1 (emergency)
* Triggers PagerDuty + Slack #prod-critical
* Stops (Rules 2-6 never evaluated)

Example 2: Sales flow error (not critical)

* Doesn't match Rule 1 (not critical)
* Doesn't match Rule 2 (no volume threshold)
* Matches Rule 3 (sales functional area)
* Triggers Slack #sales-team
* Stops (Rules 4-6 never evaluated)

Example 3: Unknown error type

* Doesn't match Rules 1-5
* Matches Rule 6 (catch-all)
* Triggers Slack #prod-errors

### Ladder Benefits

**Proportional Response:** More severe issues get more urgent notifications.

**Clear Ownership:** Each tier has defined ownership and response expectations.

**Noise Reduction:** General monitoring channels (#prod-errors) don't get critical issue spam; urgent channels (#prod-critical) don't get routine errors.

**Escalation Path:** Issues can trigger different rules at different lifecycle stages (create vs. update).

## Combining Create and Update Rules

Ladders become even more powerful when combining create-based and update-based rules in the same ordered list.

### Example: Two-Stage Ladder

```
Notification Rules with Is_Update__c = false (Immediate Notification):

Rule 1 (Create):
  Priority = "Highest"
  → PagerDuty + Slack #urgent

Rule 2 (Create):
  Functional Area contains "Sales"
  → Slack #sales-team

Rule 3 (Create):
  (Catch-all)
  → Slack #prod-errors

Notification Rules with Is_Update__c = true (Escalation):

Rule 4 (Update):
  Log Count > 100
  → Slack #prod-escalations + PagerDuty

Rule 5 (Update):
  Impacted Users Count > 20
  → Slack #prod-escalations
```

**How This Works:**

**Stage 1 - Issue Created:**

* New sales error created
* Matches Rule 2 (create)
* Notification sent to #sales-team
* Create rules done

**Stage 2 - Issue Updated (New Occurrences):**

* Issue grows to 110 occurrences
* Matches Rule 4 (update)
* Escalation notification sent to #prod-escalations
* Update rules done (fire only once per rule per issue)

This creates a two-stage awareness ladder:

1. Initial notification to functional team
2. Escalation notification when problem grows

## Order of Execution for Update Rules

Update rules follow the same stop-on-first-match principle, but with an important distinction: **each update rule can only fire once per issue**.

**Example Scenario:**

```
Rule 1 (Update): Log Count > 50 → Slack #escalations
Rule 2 (Update): Log Count > 100 → PagerDuty

Issue lifecycle:
- Created: 1 occurrence (no update rules fire)
- Updated to 55 occurrences: Rule 1 fires → #escalations
- Updated to 110 occurrences: Neither rule fires (Rule 1 already fired, Rule 2 never evaluated)
```

**Problem:** Rule 2 never gets a chance to fire because Rule 1 already fired.

**Solution:** Reorder to put most critical rule first:

```
Rule 1 (Update): Log Count > 100 → PagerDuty
Rule 2 (Update): Log Count > 50 AND Log Count <= 100 → Slack #escalations
```

Now:

* Updated to 55 occurrences: Rule 1 doesn't match, Rule 2 matches → #escalations
* Updated to 110 occurrences: Rule 1 matches → PagerDuty

{% hint style="warning" %}
**Update Rule Overlap**

When using multiple update rules with overlapping conditions (like different thresholds), order them from most to least critical, and use range filters (>= 50 AND < 100) to prevent conflicts.
{% endhint %}

## Testing Rule Order

Before deploying notification rules to production, test the execution order:

**Testing Process:**

1. Set up rules in sandbox environment
2. Trigger test errors that match different rules
3. Verify correct rule fires (not catch-all when specific rule should match)
4. Reorder if needed
5. Test again
6. Deploy to production

**Common Test Cases:**

* Does critical issue trigger PagerDuty? (not caught by catch-all)
* Does functional area route to correct team?
* Does catch-all only fire for unmapped issues?
* Do update escalations fire at correct thresholds?

## Managing Rule Order Over Time

As your notification strategy evolves, rule order requires maintenance:

### When to Reorder

**New specific rule added:**

* Insert near top, above catch-all and general rules
* Test to ensure it fires correctly

**Rule becoming too broad:**

* Add more specific filters
* OR split into multiple rules and reorder

**Catch-all rule catching too much:**

* Add specific rules above it for common patterns
* Tighten catch-all filters (though this defeats "catch-all" purpose)

### Monthly Review Questions

1. Are catch-all rules at the bottom?
2. Are critical/emergency rules at the top?
3. Are specific rules above general rules?
4. Do any rules never fire? (might be blocked by earlier rule)
5. Do any rules fire too often? (might need to be more specific/reordered)

## Rule Order Anti-Patterns

### Anti-Pattern 1: Catch-All First

```
❌ Incorrect:
Rule 1: (No filters) → #prod-errors
Rule 2: Priority = "Highest" → PagerDuty

Result: Critical issues go to #prod-errors, never reach PagerDuty
```

### Anti-Pattern 2: Overlapping Update Rules (Wrong Order)

```
❌ Incorrect:
Rule 1 (Update): Log Count > 50 → Slack
Rule 2 (Update): Log Count > 100 → PagerDuty

Result: Issue hits 110 occurrences, only Rule 1 fires (already fired at 51)
```

### Anti-Pattern 3: General Before Specific

```
❌ Incorrect:
Rule 1: Category = "Flow" → #flow-errors
Rule 2: Category = "Flow" AND Functional Area = "Sales" → #sales-team

Result: Sales flow errors go to #flow-errors (Rule 1 matches first)
```

**Correct Order:** Rule 2 before Rule 1 (most specific first)

## Summary

Effective rule ordering prevents notification spam and ensures the right teams receive the right alerts:

**Key Principles:**

1. **Stop-on-first-match:** Only one rule fires per event
2. **Specific to general:** Most specific rules at top, catch-all at bottom
3. **Test thoroughly:** Verify rule order in sandbox before production
4. **Review regularly:** Rule order requires maintenance as strategy evolves
5. **Ladder approach:** Multi-tier escalation using ordered rules

**Recommended Ordering Pattern:**

```
1. Emergency/Critical (specific conditions) → PagerDuty
2. Functional area specific rules (ordered by priority)
3. Category-specific rules (Apex, Flow, Bulk API)
4. Update escalation rules (thresholds)
5. Catch-all (no filters) → General monitoring channel
```

For rule types and filter options, see [Issue-level Alerts](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/alerting/issue-level-alerts.md) and [Log-level Alerts](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/alerting/log-level-alerts.md).
