> For the complete documentation index, see [llms.txt](https://docs.pharos.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/alerting/notification-channels/webhooks.md).

# Webhooks

Configure webhook notifications for integration with third-party observability tools.

Webhooks enable Pharos to send error data to any HTTP endpoint, creating powerful integration possibilities with third-party observability tools, custom dashboards, data warehouses, and automation workflows. Webhooks are the most flexible notification channel, limited only by your development resources.

## What Are Webhooks?

A webhook is an HTTP POST request sent to a URL you control, containing a JSON payload with issue or log data from Pharos.

**Simple Explanation:** When an error occurs in Pharos, instead of (or in addition to) sending a Slack message, Pharos can send the error data to your custom API endpoint. Your system receives the data and processes it however you want.

**Technical Mechanism:**

```
1. Error occurs in Salesforce
2. Pharos creates log/issue
3. Notification rule evaluates
4. Pharos sends HTTP POST to your webhook URL
5. Your endpoint receives JSON payload
6. Your system processes data (store, alert, transform, etc.)
```

Webhooks transform Pharos from a standalone observability tool into an **integration hub** that can feed data into your broader ecosystem.

## Why Webhooks for Observability

Webhooks serve use cases that pre-built integrations (Slack, Teams, PagerDuty) cannot:

**Custom Dashboards:** Stream error metrics to internal monitoring dashboards built on custom tech stacks.

**Data Warehousing:** Send all error data to your data warehouse (Snowflake, BigQuery, Redshift) for historical analysis and ML model training.

**ServiceNow/Jira Integration:** Create tickets in systems Pharos doesn't natively integrate with.

**Custom Alerting Logic:** Implement sophisticated alerting rules that exceed Pharos's built-in capabilities.

**Multi-System Workflows:** Trigger automation across multiple systems (create ticket, update status board, send SMS, etc.).

**Third-Party Observability:** Integrate with observability platforms like Datadog, New Relic, Splunk, or custom monitoring tools.

## Webhook Configuration

Webhooks are configured in Pharos notification rules with endpoint URL and payload customization.

**Basic Configuration:**

```
Rule Type: Notification (Is_Update__c = false)
Filter: Organization = "Production"
Action: Webhook
  - URL: https://your-api.company.com/pharos-events
  - Method: POST
  - Headers: 
      Content-Type: application/json
      Authorization: Bearer YOUR_API_KEY
  - Payload: JSON template with merge fields
```

**Authentication:**

Rather than hand-writing an `Authorization` header, pick an **Authentication Type** — None, Basic, Bearer, API Key, or Custom — and supply the corresponding value. Pharos assembles the header for you, which keeps the credential in one field instead of buried in free-form header text where it's easy to mistype or leak into a copied example. Custom headers remain available for anything the receiving system needs beyond authentication.

Avoid putting credentials in the URL as a query parameter. URLs are logged far more liberally than headers are, by proxies and by the receiving application, so a key placed there tends to end up somewhere you didn't intend.

**Payload Structure:**

Pharos sends a JSON payload containing issue or log data. You define the structure using merge fields:

**Default Payload Template:**

```json
{
  "event_type": "issue_created",
  "timestamp": "{!NOW()}",
  "issue": {
    "id": "{!Issue.Id}",
    "title": "{!Issue.Summary__c}",
    "category": "{!Issue.Category__c}",
    "functional_area": "{!Issue.Functional_Area__c}",
    "organization": "{!Issue.Organization_Name__c}",
    "priority": "{!Issue.Priority__c}",
    "log_count": {!Issue.Log_Count__c},
    "impacted_users": {!Issue.Impacted_Users_Count__c},
    "first_occurred": "{!Issue.First_Occurred_On__c}",
    "last_occurred": "{!Issue.Last_Occurred_On__c}",
    "pharos_link": "{!Issue_Link}"
  }
}
```

**Custom Payload:** You can fully customize the JSON structure to match your receiving system's API requirements.

For the action form itself — the URL, method, body template, authentication fields, and headers — see [Notification Actions](https://docs.pharos.ai/configuration-guide/notifications/notification-actions.md#webhook).

## Use Case Examples

The examples below show how teams use webhooks to bridge Pharos with systems it doesn't natively support.

### Use Case 1: Internal Monitoring Dashboard

**Scenario:** Engineering team built custom React dashboard for real-time error monitoring across all systems (Salesforce, AWS, web apps).

**Webhook Configuration:**

```
Rule: Issue Created OR Issue Updated
URL: https://dashboard.company.com/api/ingest
Payload:
{
  "source": "salesforce_pharos",
  "timestamp": "{!NOW()}",
  "severity": "{!Issue.Priority__c}",
  "title": "{!Issue.Summary__c}",
  "category": "{!Issue.Category__c}",
  "user_impact": {!Issue.Impacted_Users_Count__c},
  "metadata": {
    "organization": "{!Issue.Organization_Name__c}",
    "functional_area": "{!Issue.Functional_Area__c}"
  }
}
```

**Receiving System:** Custom Node.js API stores data in PostgreSQL, triggers WebSocket updates to dashboard UI for real-time display.

**Why Webhooks:**

* Custom dashboard tech stack (can't use pre-built integration)
* Real-time updates required
* Need to aggregate errors from multiple sources

### Use Case 2: ServiceNow Incident Creation

**Scenario:** Enterprise ITSM team requires all production issues to create ServiceNow incidents automatically.

**Webhook Configuration:**

```
Rule: Issue Created AND Organization = "Production" AND Priority = "Highest"
URL: https://yourinstance.service-now.com/api/now/table/incident
Headers:
  Content-Type: application/json
  Authorization: Basic base64(serviceNowUser:password)
Payload:
{
  "short_description": "Salesforce Issue: {!Issue.Summary__c}",
  "description": "Category: {!Issue.Category__c}\nFunctional Area: {!Issue.Functional_Area__c}\n\nError: {!Issue.Description__c}\n\nPharos Link: {!Issue_Link}",
  "urgency": "1",
  "impact": "1",
  "assignment_group": "Salesforce Engineering",
  "category": "Software",
  "subcategory": "Application"
}
```

**Result:** Critical Pharos issues automatically create ServiceNow incidents with all relevant details, eliminating manual ticket creation.

**Why Webhooks:**

* Pharos doesn't natively integrate with ServiceNow
* ServiceNow has robust REST API
* Webhook payload maps directly to ServiceNow incident fields

### Use Case 3: Data Warehouse Integration

**Scenario:** Data engineering team wants all error data streamed to Snowflake for historical trend analysis and ML model training.

**Webhook Configuration:**

```
Rule: Issue Created OR Issue Updated (all environments)
URL: https://data-pipeline.company.com/api/salesforce/errors
Payload:
{
  "event_type": "salesforce_error",
  "timestamp": "{!NOW()}",
  "organization": "{!Issue.Organization_Name__c}",
  "category": "{!Issue.Category__c}",
  "functional_area": "{!Issue.Functional_Area__c}",
  "occurrence_count": {!Issue.Log_Count__c},
  "unique_users_affected": {!Issue.Impacted_Users_Count__c},
  "frequency_days": {!Issue.Average_Frequency__c},
  "first_seen": "{!Issue.First_Occurred_On__c}",
  "last_seen": "{!Issue.Last_Occurred_On__c}",
  "issue_key": "{!Issue.Key__c}"
}
```

**Receiving System:** Python Flask API receives webhook, validates payload, writes to Kafka topic, which is consumed by Snowflake connector.

**Analysis Use Cases:**

* Historical error trends (categories over time)
* Correlation analysis (errors vs. deployments)
* ML models predicting error likelihood based on patterns
* Executive dashboards showing error reduction over quarters

**Why Webhooks:**

* Need all error data, not just notifications
* Data pipeline requires structured JSON
* Real-time streaming required (not batch export)

### Use Case 4: Datadog Integration

**Scenario:** Infrastructure team uses Datadog for all observability. Want Salesforce errors integrated into Datadog dashboards alongside AWS, Kubernetes, and application metrics.

**Webhook Configuration:**

```
Rule: Issue Created AND Organization = "Production"
URL: https://api.datadoghq.com/api/v1/events
Headers:
  Content-Type: application/json
  DD-API-KEY: your_datadog_api_key
Payload:
{
  "title": "{!Issue.Category__c}: {!Issue.Summary__c}",
  "text": "Salesforce error detected\n\nCategory: {!Issue.Category__c}\nFunctional Area: {!Issue.Functional_Area__c}\nUsers Impacted: {!Issue.Impacted_Users_Count__c}\n\n[View in Pharos]({!Issue_Link})",
  "alert_type": "error",
  "source_type_name": "salesforce",
  "tags": [
    "organization:{!Issue.Organization_Name__c}",
    "category:{!Issue.Category__c}",
    "functional_area:{!Issue.Functional_Area__c}",
    "source:pharos"
  ]
}
```

**Result:** Salesforce errors appear as Datadog events, correlated with infrastructure metrics and APM data in unified dashboards.

**Why Webhooks:**

* Datadog has events API (no native Pharos integration)
* Enables correlation analysis (errors vs. infrastructure)
* Unified observability across entire tech stack

### Use Case 5: Slack Webhook (Alternative to OAuth)

**Scenario:** Security policy prohibits OAuth apps. Want Slack notifications using incoming webhooks instead.

**Webhook Configuration:**

```
Rule: Issue Created
URL: https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXX
Payload:
{
  "text": "*New Issue in Production*\n\n*Summary:* {!Issue.Summary__c}\n*Category:* {!Issue.Category__c}\n*Users Impacted:* {!Issue.Impacted_Users_Count__c}\n\n<{!Issue_Link}|View Issue>"
}
```

**Result:** Slack notifications without OAuth, using Slack's incoming webhooks feature.

**Limitations vs. Native Slack Integration:**

* Can't choose channel dynamically (webhook URL is tied to one channel)
* Less rich formatting
* No thread support

**Why Webhooks:**

* Security policy blocks OAuth
* Simple alternative for basic Slack notifications

### Use Case 6: Multi-System Automation

**Scenario:** Critical payment errors require coordinated response across multiple systems.

**Webhook Configuration:**

```
Rule: Functional Area = "Payment" AND Category = "Apex" AND Priority = "Highest"
URL: https://automation.company.com/api/orchestration/payment-failure
Payload:
{
  "workflow": "payment_critical_response",
  "issue_id": "{!Issue.Id}",
  "issue_summary": "{!Issue.Summary__c}",
  "user_impact": {!Issue.Impacted_Users_Count__c},
  "pharos_link": "{!Issue_Link}"
}
```

**Automation Workflow (Receiving System):**

1. Create Jira P0 ticket
2. Post to Slack #payments-critical with @channel
3. Create PagerDuty incident
4. Update status page (StatusPage.io)
5. Send SMS to payment processor account manager
6. Create Zoom meeting room link for war room

**Result:** One webhook triggers coordinated response across 6 systems automatically.

**Why Webhooks:**

* Orchestration logic too complex for Pharos rules
* Need flexibility to add/remove steps
* Centralized automation service handles multi-system coordination

## Webhook Reliability and Retry Logic

Pharos includes retry logic for failed webhook deliveries:

**Retry Behavior:**

* **Initial Attempt:** Webhook fires immediately
* **First Retry:** If 5xx error or timeout, retry after 30 seconds
* **Second Retry:** If still failing, retry after 5 minutes
* **Third Retry:** If still failing, retry after 30 minutes
* **Failure:** After 3 retries, mark as failed and stop

**Success Criteria:** Webhook is considered successful if receiving endpoint returns HTTP 2xx status code.

**Timeout:** Webhook requests timeout after 10 seconds. If your endpoint takes longer, Pharos considers it failed and retries.

**Implication:** Your webhook endpoint should:

* Respond quickly (<5 seconds ideally)
* Return 2xx on success
* Return 5xx if processing failed (triggers retry)
* Return 4xx for bad request (no retry, permanent failure)

**Best Practice:** For long-running processing, use asynchronous pattern:

```
Your Endpoint Logic:
1. Receive webhook from Pharos
2. Validate payload
3. Write to queue (Kafka, SQS, etc.)
4. Immediately return 200 OK
5. Background worker processes queue asynchronously
```

This ensures Pharos webhook succeeds quickly, while heavy processing happens asynchronously.

## Security Considerations

Because webhooks send error data to an endpoint you expose, secure it on several fronts.

### Authentication

Always require authentication for webhook endpoints:

**Good:**

```
Endpoint validates API key in Authorization header
Reject requests without valid key (401 Unauthorized)
```

**Bad:**

```
Public endpoint with no authentication
Anyone can send fake payloads
```

### HTTPS Required

**Always use HTTPS**, not HTTP:

```
✓ https://api.company.com/webhooks
❌ http://api.company.com/webhooks
```

Unencrypted HTTP exposes error data in transit.

### Payload Validation

Always validate incoming webhook payloads:

**Validation Checklist:**

* Verify API key/token
* Validate JSON structure
* Check required fields present
* Sanitize input before processing
* Rate limit to prevent abuse

**Example (Python):**

```python
@app.route('/pharos-webhook', methods=['POST'])
def pharos_webhook():
    # Validate API key
    api_key = request.headers.get('Authorization')
    if api_key != f'Bearer {EXPECTED_API_KEY}':
        return {'error': 'Unauthorized'}, 401
    
    # Validate JSON
    try:
        payload = request.get_json()
    except:
        return {'error': 'Invalid JSON'}, 400
    
    # Validate required fields
    if 'issue' not in payload or 'event_type' not in payload:
        return {'error': 'Missing required fields'}, 400
    
    # Process payload
    process_issue(payload)
    
    return {'status': 'success'}, 200
```

### IP Whitelisting

For additional security, whitelist Pharos IPs at your firewall:

**Pharos Outbound IPs:** (Contact support for current IP list—varies by Salesforce instance)

**Firewall Rule:**

```
Allow: HTTP POST from [Pharos IP ranges] to your-api.company.com:443
Deny: All other traffic
```

## Webhook Best Practices

These practices keep your webhook integration reliable as it grows.

### Practice 1: Start Simple, Expand Gradually

**Week 1:**

```
Simple webhook posting to test endpoint
Log payload, don't process
Verify data structure
```

**Week 2:**

```
Add basic processing
Store in database
Validate stability
```

**Week 3+:**

```
Build out full integration
Add error handling
Monitor performance
```

### Practice 2: Idempotency

Design your endpoint to handle duplicate deliveries:

**Problem:** Network issues may cause Pharos to retry, resulting in duplicate webhook deliveries.

**Solution:** Use `issue_id` or `event_id` to detect duplicates:

```python
def process_issue(payload):
    issue_id = payload['issue']['id']
    
    # Check if already processed
    if db.exists('processed_issues', issue_id):
        return  # Already processed, skip
    
    # Process new issue
    store_issue(payload)
    db.insert('processed_issues', issue_id)
```

### Practice 3: Async Processing

Don't block webhook response on heavy processing:

**Poor Pattern:**

```python
def pharos_webhook():
    payload = request.get_json()
    
    # BLOCKING operations (slow)
    query_database()
    call_external_apis()
    generate_reports()
    
    return {'status': 'success'}, 200  # Pharos waits for all this
```

**Better Pattern:**

```python
def pharos_webhook():
    payload = request.get_json()
    
    # Quick validation and queueing
    queue.publish('pharos_events', payload)
    
    return {'status': 'success'}, 200  # Immediate response

# Separate worker processes queue
@worker.task
def process_pharos_event(payload):
    query_database()
    call_external_apis()
    generate_reports()
```

### Practice 4: Monitoring and Alerting

Monitor your webhook endpoint:

**Metrics to Track:**

* Request rate (webhooks per minute)
* Error rate (% of 5xx responses)
* Response latency (p50, p95, p99)
* Queue depth (if using async processing)

**Alerts:**

* Error rate > 5% for 5 minutes
* Response latency p95 > 5 seconds
* Queue depth > 1000 messages

**Why:** If your endpoint is down, Pharos will retry but eventually fail. You need to know immediately to restore service.

### Practice 5: Testing

Test webhooks before deploying to production:

**Test Scenarios:**

1. **Valid payload:** Returns 200
2. **Invalid JSON:** Returns 400
3. **Missing auth:** Returns 401
4. **Server error:** Returns 500 (Pharos retries)
5. **Timeout:** Endpoint takes >10 seconds (Pharos retries)
6. **Duplicate delivery:** Idempotency works correctly

Use sandbox Pharos environment to trigger test webhooks without affecting production.

## Limitations and Considerations

Webhooks are flexible, but a few constraints shape how you design your endpoint.

### Payload Size Limits

Pharos webhook payloads are capped at \~100KB.

**Implication:** Don't include full stack traces or extensive log details in webhook payload. Include essentials and link to Pharos for full details.

**Good:**

```json
{
  "issue_summary": "Apex Exception in OpportunityTriggerHandler",
  "pharos_link": "{!Issue_Link}"
}
```

**Poor:**

```json
{
  "full_stack_trace": "[10,000 characters of stack trace]",
  "all_log_details": "[50,000 characters of log data]"
}
```

### Rate Limiting

If your endpoint has rate limits, ensure they're higher than your expected webhook volume:

**Example Calculation:**

* 100 issues created per day
* \= \~4 webhooks per hour
* Rate limit must support ≥10 requests/hour (with buffer)

If you hit rate limits, webhook failures increase.

### Synchronous Execution

Webhooks fire synchronously during issue/log creation process.

**Implication:** Extremely slow webhook endpoints (>10 second timeout) may delay Pharos processing. Use async pattern to avoid this.

## Summary

Webhooks are the most flexible notification channel, enabling unlimited integration possibilities:

**Use Webhooks For:**

* Custom internal dashboards
* Data warehouse streaming
* ServiceNow/Jira ticket creation
* Third-party observability tools (Datadog, Splunk, New Relic)
* Multi-system automation workflows
* Systems Pharos doesn't natively integrate with

**Webhook Capabilities:**

* Fully customizable JSON payload
* Any HTTP endpoint
* Multiple authentication methods
* Retry logic for reliability
* Sub-second delivery time

**Best Practices:**

* HTTPS only, always authenticate
* Design for idempotency (handle duplicates)
* Async processing (immediate 200 response)
* Monitor endpoint health
* Start simple, expand gradually

**Limitations:**

* Requires development resources (API endpoint)
* Payload size limit (\~100KB)
* Synchronous execution (can't be extremely slow)
* Must handle retries and errors

**Key Takeaway:** Webhooks transform Pharos into an integration hub, enabling sophisticated error data workflows beyond standard notifications. If pre-built integrations (Slack, Teams, PagerDuty) don't meet your needs, webhooks provide the flexibility to build exactly what you need.

For pre-built notification channels, see [Slack](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/alerting/notification-channels/slack.md), [Microsoft Teams](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/alerting/notification-channels/teams.md), [PagerDuty](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/alerting/notification-channels/pagerduty.md), and [Email](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/alerting/notification-channels/email.md).
