> For the complete documentation index, see [llms.txt](https://docs.pharos.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/tracing.md).

# Tracing

Understand how Pharos solves common pain points in obtaining Salesforce debug logs.

Tracing in Pharos refers to the automated collection and analysis of Salesforce debug logs. While Pharos logs provide rich context, sometimes you need the **raw Salesforce debug log** for deep troubleshooting—but obtaining these logs manually is notoriously difficult. Pharos automates this process.

## Overview

Salesforce debug logs are one of the most powerful troubleshooting tools available, yet obtaining them manually is slow and unreliable. The sections below break down why debug logs are so hard to work with by hand, and how Pharos automates the entire workflow.

{% hint style="danger" %}
**The Debug Log Struggle**

Getting Salesforce debug logs traditionally requires manual setup, perfect timing, and luck. Logs expire quickly, are hard to find, and require tedious parsing. Pharos automates this entire painful process.
{% endhint %}

### The Debug Log Problem

Salesforce debug logs are invaluable for troubleshooting, but they come with significant pain points:

**Challenge 1: Manual Setup Required**

```
Traditional Process:
  1. Navigate to Setup → Debug Logs
  2. Click "New" to create trace flag
  3. Select user to trace (must know who will hit the error)
  4. Set log levels for each category (Apex, Flow, Visualforce, etc.)
  5. Set expiration time (max 24 hours)
  6. Wait for user to reproduce the error
  7. Find the specific log among potentially hundreds of entries
  8. Download and parse the log file
  
Time Investment: 15-30 minutes per debug session
```

**Challenge 2: Logs Expire Quickly**

* Debug logs automatically expire after a few hours
* If you don't download in time, they're gone forever
* No historical record for pattern analysis

**Challenge 3: Hard to Find Relevant Logs**

* Production org generates thousands of logs per day
* Finding the specific log for a specific error is like finding a needle in a haystack
* No automatic correlation between error and debug log

**Challenge 4: Difficult to Analyze**

* Debug logs can be massive (megabytes of text)
* Hard to read and parse manually
* No built-in search or filtering within logs

**Challenge 5: Limited Storage**

* Salesforce limits total debug log storage (1,000 MB per org)
* Old logs automatically deleted when storage fills up
* Can't keep historical logs for trend analysis

### Pharos's Solution: Automated Tracing

{% hint style="success" %}
**From Hours to Minutes**

What used to take 30 minutes of manual setup and monitoring now takes 2 minutes to configure and runs automatically. Debug logs are automatically captured, parsed, and linked to the relevant Issue.
{% endhint %}

Pharos **automates** the entire debug log workflow:

**Automated Process:**

```
1. Pharos detects an error (via Issue creation)
2. You configure a trace schedule: "For this Issue, fetch debug logs"
3. Pharos automatically:
   - Creates trace flags for affected users
   - Monitors for occurrences of the specific error
   - Fetches matching debug logs
   - Parses and indexes the logs
   - Notifies you when relevant logs are found
4. You review parsed logs directly in Pharos UI (no download needed)

Time Investment: 2 minutes to set up, fully automated after that
```

**Key Benefits:**

* **No manual trace flag management:** Pharos handles it
* **Automatic correlation:** Debug logs linked directly to Issues
* **Persistent storage:** Logs retained beyond Salesforce's 24-hour limit
* **Searchable:** Full-text search within debug logs
* **Notification:** Get alerted when relevant logs are captured

## How Tracing Works

### Issue-Based Tracing

Tracing is configured at the **Issue level**, not the user level:

**Scenario:**

```
Issue: "Apex Exception in OpportunityTriggerHandler" (47 users affected)

Without Pharos Tracing:
  - Set up 47 trace flags manually (one per user)
  - Monitor each user's debug logs individually
  - Download and search 47 sets of logs
  
With Pharos Tracing:
  - Create one trace schedule on the Issue
  - Pharos automatically traces all affected users
  - Pharos finds logs containing the specific error message
  - Pharos alerts you when relevant logs are captured
```

### Trace Schedule Configuration

A trace schedule is defined by several key parameters that control who is traced, what Pharos looks for, and for how long.

**Key Parameters:**

**1. Target Users:**

* All users affected by this Issue
* Specific users (filter by profile, role, or individual)
* Next N users who encounter the error

**2. Error Pattern:**

```
What to look for in debug logs:
  - Exact error message: "Attempt to de-reference a null object"
  - Stack trace line: "OpportunityTriggerHandler.afterUpdate line 156"
  - Custom pattern: Regular expression matching specific scenarios
```

**3. Duration:**

```
How long to actively trace:
  - 24 hours (short-term debugging)
  - 7 days (intermittent issues)
  - Until manually stopped (ongoing monitoring)
```

**4. Log Levels:**

```
Granularity of debug logs (from least to most detail):
  - NONE
  - ERROR
  - WARN
  - INFO (minimal)
  - DEBUG (standard)
  - FINE (detailed)
  - FINER
  - FINEST (maximum detail, larger logs)
```

**5. Notification:**

```
When to alert:
  - Immediately when matching log found
  - Daily digest of captured logs
  - No notification (check manually)
```

### Trace Execution Workflow

Once a schedule is active, Pharos runs tracing through five stages, from creating trace flags to final cleanup.

**Step 1: Schedule Creation**

```
User Action: Create trace schedule on Issue
Pharos Action: 
  - Identifies users affected by this Issue
  - Creates trace flags for those users via Tooling API
  - Sets log levels and expiration
```

**Step 2: Active Monitoring**

```
Pharos Continuously:
  - Polls for new debug logs from traced users
  - Downloads logs via Tooling API
  - Parses logs looking for the error pattern
  - Indexes logs for search
```

**Step 3: Match Detection**

```
When Pharos finds a matching log:
  - Links log to Issue
  - Sends notification (Slack, Email, PagerDuty)
  - Highlights relevant portions of the log
  - Makes log available in Pharos UI
```

**Step 4: Analysis**

```
Developer:
  - Opens Issue in Pharos
  - Navigates to "Tracing" tab
  - Reviews captured debug logs
  - Identifies root cause from detailed execution trace
  - Implements fix
```

**Step 5: Cleanup**

```
When schedule expires or is manually stopped:
  - Trace flags removed
  - Logs retained in Pharos (configurable retention)
  - Schedule marked as complete
```

## Use Cases

### Use Case 1: Intermittent Errors

**Scenario:** Error occurs sporadically, can't reproduce on demand

**Without Tracing:**

```
Problem: Can't leave trace flags running indefinitely (expire after 24 hours)
Solution: Keep manually recreating trace flags until error happens
Result: Days or weeks of manual monitoring
```

**With Pharos Tracing:**

```
Solution: 
  1. Set up trace schedule for 7 days
  2. Pharos automatically renews trace flags
  3. Wait for error to occur naturally
  4. Pharos captures debug log when it happens
  5. Notification sent immediately
  
Result: Hands-off monitoring, guaranteed capture
```

### Use Case 2: Multi-User Issues

**Scenario:** Error affecting 47 users across different profiles

**Without Tracing:**

```
Problem: Need to set up 47 trace flags manually
Complexity: Each user may hit the error at different times
Result: Impractical to trace everyone
```

**With Pharos Tracing:**

```
Solution:
  1. Create single trace schedule on Issue
  2. Select "All affected users" option
  3. Pharos manages all 47 trace flags automatically
  4. Captures logs from whoever hits error next
  
Result: Comprehensive coverage without manual effort
```

### Use Case 3: Production Debugging

**Scenario:** Production error, need debug log, but can't reproduce in sandbox

**Without Tracing:**

```
Problem: Can't deploy debug code to production
Challenge: Production generates thousands of logs per day
Risk: Relevant log may expire before you find it
```

**With Pharos Tracing:**

```
Solution:
  1. Set up trace schedule in production
  2. Pharos filters for specific error pattern
  3. Captures only relevant logs (ignores noise)
  4. Notifies immediately when found
  
Result: Targeted debugging without log overload
```

### Use Case 4: Performance Analysis

**Scenario:** Need to understand why specific transaction is slow

**With Pharos Tracing:**

```
Setup:
  1. Create trace schedule for affected users
  2. Set log levels to FINEST (maximum detail)
  3. Specify error pattern (or capture all executions)
  
Analysis:
  - Review governor limit consumption
  - Identify slow SOQL queries
  - Find inefficient loops
  - Measure method execution times
  
Result: Data-driven performance optimization
```

## Tracing vs. Pharos Logs

**Question:** Why do I need Salesforce debug logs if Pharos already provides enriched logs?

**Answer:** Pharos logs and debug logs serve different purposes:

| Pharos Logs                                                | Salesforce Debug Logs                             |
| ---------------------------------------------------------- | ------------------------------------------------- |
| **What:** Structured, enriched error logs                  | **What:** Raw execution trace                     |
| **When:** Every error automatically captured               | **When:** Only when trace flag active             |
| **Content:** Error message, stack trace, context           | **Content:** Line-by-line code execution          |
| **Size:** Typically 1-10 KB per log                        | **Size:** Often 1-10 MB per log                   |
| **Purpose:** High-level troubleshooting, impact assessment | **Purpose:** Deep debugging, performance analysis |
| **Storage:** Unlimited (configurable retention)            | **Storage:** Limited (1,000 MB org-wide)          |

**Combined Power:**

```
Pharos Log tells you:
  - What error occurred
  - Who was affected
  - What records were involved
  - When it started happening
  
Debug Log tells you:
  - Exact sequence of method calls
  - Variable values at each step
  - SOQL queries executed
  - Why the error occurred (line-by-line trace)
  
Together: Complete picture from high-level impact to low-level cause
```

## Tracing Limitations

### 1. Storage Overhead

**Impact:** Each debug log can be 1-10 MB

**Mitigation:**

* Set aggressive retention policies (delete after 7 days)
* Use DEBUG or INFO log levels (not FINEST) unless necessary
* Limit trace schedules to specific users, not "all users"

### 2. Salesforce API Limits

**Impact:** Fetching debug logs consumes API calls

**Typical Consumption:**

* 1 API call to create trace flag
* 1 API call per log to fetch
* 1 API call to delete trace flag

**Mitigation:**

* Pharos uses bulk operations where possible
* Schedules API calls during off-peak hours
* Configurable polling frequency (every 5 min, 15 min, hourly)

### 3. Trace Flag Limit

**Salesforce Limit:** Subject to Salesforce's trace-flag limits per org

**Impact:** Can't trace an unlimited number of users simultaneously

**Pharos Handling:**

* Rotates trace flags (traces subset of users, then switches)
* Prioritizes users most likely to hit the error (based on frequency)
* Notifies if trace flag limit reached

### 4. Performance Impact (Minimal)

**Impact:** Trace flags increase log generation, which consumes slight CPU/storage

**Mitigation:**

* Only trace when actively debugging (not 24/7)
* Use INFO or DEBUG levels for production (not FINEST)
* Limit to specific users/Issues

## Best Practices

### 1. Use Tracing for Stubborn Issues Only

**When to Trace:**

* Error root cause unclear from Pharos enriched logs
* Intermittent issues that are hard to reproduce
* Performance debugging requiring detailed execution trace

**When NOT to Trace:**

* Error message is self-explanatory ("Required field missing")
* Pharos logs already provide sufficient context
* Issue resolved before trace needed

### 2. Start with Narrow Scope, Expand If Needed

**Phase 1: Single User**

```
Trace Schedule:
  Users: 1 (specific user who reports error most frequently)
  Duration: 24 hours
  Log Level: DEBUG
```

**Phase 2: Profile-Wide** (if single user trace doesn't catch it)

```
Trace Schedule:
  Users: All "Sales Rep" profile
  Duration: 3 days
  Log Level: DEBUG
```

**Phase 3: All Affected** (if still not captured)

```
Trace Schedule:
  Users: All affected by Issue
  Duration: 7 days
  Log Level: DEBUG
```

### 3. Use Appropriate Log Levels

**Log Level Guide:**

| Level      | Detail              | Log Size | When to Use                            |
| ---------- | ------------------- | -------- | -------------------------------------- |
| **NONE**   | None                | Minimal  | Disable logging for a category         |
| **ERROR**  | Errors only         | Minimal  | Production, errors only                |
| **WARN**   | Warnings and errors | Minimal  | Production, low-noise monitoring       |
| **INFO**   | Minimal             | \~100 KB | Production, well-understood code paths |
| **DEBUG**  | Standard            | \~500 KB | Most debugging scenarios               |
| **FINE**   | Detailed            | \~1 MB   | Variable inspection needed             |
| **FINER**  | More detailed       | \~2-3 MB | Deeper execution detail than FINE      |
| **FINEST** | Maximum             | \~5 MB   | Performance analysis, complex logic    |

**Recommendation:** Start with DEBUG. Escalate to FINE/FINEST only if needed.

### 4. Set Aggressive Retention Policies

**Configuration:**

```
Retention Policy for Trace Logs:
  - Development: 7 days
  - Staging: 14 days
  - Production: 30 days
  
Reasoning: Debug logs are massive. After issue is resolved, 
           they're no longer needed. Purge to save storage.
```

### 5. Stop Traces When Issue is Resolved

**Important:** Don't leave trace schedules running indefinitely.

**Workflow:**

```
Issue Resolved:
  1. Mark Issue as "Resolved" in Pharos
  2. Stop active trace schedules
  3. Review captured logs one final time
  4. Document root cause
  5. Trace flags automatically removed
```

## Integration with Pharos Platform

Trace logs are integrated throughout Pharos:

**Issue Tracker:**

* "Tracing" tab on Issue detail page
* Create new trace schedule directly from Issue
* View all captured trace logs linked to Issue

**Notifications:**

* Alert when trace log captured
* Include link to log in Pharos UI
* Optionally attach log file to notification

**AI Analysis (Org Genie):**

* AI can analyze debug logs for root cause
* Correlate debug log findings with Issue data
* Suggest code fixes based on execution trace

## In This Section

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>🎯 Issue Tracing</strong></td><td>Detailed documentation on Issue-based tracing—trace schedules, captured debug logs, and analysis tools.</td><td><a href="https://docs.pharos.ai/platform-overview/pharos-platform-architecture/tracing/issue-tracing.md">Issue Tracing</a></td></tr><tr><td><strong>⚡ Triton Tracing</strong></td><td>How Triton custom instrumentation enhances tracing capabilities with transaction-level visibility.</td><td><a href="https://docs.pharos.ai/platform-overview/pharos-platform-architecture/tracing/triton-tracing.md">Triton Tracing</a></td></tr></tbody></table>

{% hint style="success" %}
**Getting Started with Tracing**

1. Identify an Issue where Pharos logs don't provide enough detail
2. Open the Issue in Pharos
3. Navigate to "Tracing" tab
4. Click "New Trace Schedule"
5. Configure users, duration, and log levels
6. Save and wait for notification

Tracing will automatically capture debug logs when the error occurs next.
{% endhint %}

## Summary

Pharos Tracing automates the painful process of obtaining and analyzing Salesforce debug logs:

* **Automatic trace flag management:** No manual setup
* **Issue-based correlation:** Logs automatically linked to errors
* **Persistent storage:** Logs retained beyond Salesforce's limits
* **Intelligent filtering:** Only captures logs matching your error pattern
* **Proactive notifications:** Alerts when relevant logs found

Combined with Pharos's enriched logs, tracing provides complete visibility from high-level impact to line-by-line execution—enabling faster root cause identification and resolution.
