> For the complete documentation index, see [llms.txt](https://docs.pharos.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/logs/log-storage/data-retention.md).

# Data Retention

Configure retention rules to manage log storage via archival and removal strategies.

Retention rules help you manage log storage by automatically removing or archiving old logs. This guide explains how to configure retention policies to balance historical data retention with storage capacity constraints.

## Overview

Data retention in Pharos allows you to automatically manage log lifecycle through a flexible rule-based system. Rather than manually deleting old logs or watching storage climb indefinitely, retention rules apply age-based policies that execute automatically—removing old logs to free space, archiving them to Big Objects for long-term preservation, or selectively applying different policies based on log characteristics.

**What retention rules enable:**

* **Automatic cleanup** - Set it once, runs daily without manual intervention
* **Strategic preservation** - Archive before removal to keep historical data
* **Selective policies** - Different retention for critical vs. routine logs
* **Storage optimization** - Balance capacity with historical visibility

{% hint style="success" %}
**Golden Rule**: Keep as much historical data as your storage allows. Only implement retention rules when storage becomes constrained or when you have specific business reasons to limit data retention (e.g., compliance requirements).
{% endhint %}

## When to Use Retention Rules

### Recommended Scenarios

Implement retention rules when your situation genuinely requires them:

**Storage Constraints** - If Data Storage exceeds 80% capacity, retention rules help bring usage back to manageable levels without manually deleting logs.

**Rapid Growth** - When log volumes are growing faster than expected and threatening to overwhelm your storage allocation, automated cleanup becomes essential to maintain headroom.

**Compliance Mandates** - Some regulations require data deletion after specific periods. In these cases, retention rules aren't just helpful—they're legally necessary to maintain compliance.

**Cost-Benefit Analysis** - If storage costs clearly outweigh the value of keeping old data indefinitely (e.g., paying thousands per year for logs you'll never reference), retention provides a way to balance cost with utility.

**Performance Degradation** - When query performance suffers due to large data volumes, reducing the active dataset through retention can restore responsiveness.

### Not Recommended

Avoid retention rules in these situations:

* **Abundant storage** (below 70% capacity) - No pressure, no need to delete
* **Historical analysis needs** - Year-over-year trending requires long-term data
* **Compliance retention requirements** - Regulations mandate keeping data
* **Uncertainty** - If unsure of appropriate retention period, wait and monitor

{% hint style="warning" %}
**Don't Rush**: Many organizations implement aggressive retention rules prematurely, then regret losing historical data when investigating long-term trends or root causes. Monitor for 60-90 days before deciding on retention periods.
{% endhint %}

### Benefits of Longer Retention

Keeping data for extended periods enables powerful analytical capabilities that justify storage costs:

**Year-over-year comparisons** demonstrate improvement: "We had 100 issues last Q4, only 10 this Q4" becomes a concrete metric for system stability gains.

**Trend analysis** reveals patterns that only emerge over months or years—seasonal variations, the impact of major releases, or the long-term effects of architectural changes.

**Root cause investigation** often requires looking back 6-12 months to understand when problems first started appearing and correlating them with specific deployments or configuration changes.

**Organizational learning** depends on understanding how error patterns evolve over time and which interventions prove most effective at reducing them.

## Retention Rule Components

### Age Threshold

The primary criterion for retention is log age, measured from the log creation date. Choose a threshold that balances your storage constraints with analytical needs:

| Threshold    | Retention Level   | Best For                                 | Trade-off                       |
| ------------ | ----------------- | ---------------------------------------- | ------------------------------- |
| **30 days**  | Very aggressive   | High-volume orgs with severe constraints | Minimal history, emergency only |
| **60 days**  | Aggressive        | Recent issue tracking                    | Limited trend analysis          |
| **6 months** | Moderate          | Balance history with storage             | Good for most scenarios         |
| **1 year**   | Conservative      | Trend analysis and comparisons           | Requires adequate storage       |
| **2+ years** | Minimal retention | Abundant storage orgs                    | Maximum analytical value        |

You can specify any custom interval in days (e.g., 45 days, 90 days, 180 days, etc.) to match your specific needs.

### Filter Criteria

Retention rules support sophisticated filter criteria that let you selectively apply retention based on log characteristics. This enables strategic policies where critical logs remain accessible longer while routine logs are removed more aggressively.

**Common filter strategies:**

**By Category** - Different retention for different log types

* Keep Apex logs for 1 year (complex debugging value)
* Remove Bulk API logs after 60 days (less diagnostic value)
* Retain Flow logs for 6 months (medium complexity)

**By Functional Area** - Preserve critical business areas longer

* Sales automation: 2 year retention
* Administrative tasks: 6 month retention
* Reporting processes: 3 month retention

**By Category (severity values)** - Keep the more serious records longer

* Error: 1 year minimum
* Warning: 6 months
* Debug: 3 months or less

**By Status** - Different rules for completed issues

* New/active issues: Keep all related logs
* Completed issues (>6 months): Apply standard retention

**By Custom Fields** - Any field you've added to the log object

**Example multi-tier strategy:**

```
Rule 1: Remove Bulk API logs after 60 days
Rule 2: Remove Flow logs after 1 year  
Rule 3: Keep Apex logs indefinitely (no rule)
```

### Actions

Retention rules can perform multiple actions on matching logs. Choose the actions that balance your storage needs with data preservation requirements:

**Two action types:**

1. **Removal** - Permanently delete the log record from Data Storage
2. **Local Archival** - Copy to Big Object storage before removal (preserves data)

You can combine actions—commonly archiving before removal to preserve historical data while freeing up expensive Data Storage.

## Action Types

### Removal

Removal permanently deletes log records from custom objects, immediately freeing Data Storage. This is the most aggressive action and cannot be undone, so consider carefully before implementing.

**When removal makes sense:**

Storage is critically constrained (>90% capacity) and aggressive action is necessary to prevent hitting limits. Old data has minimal business value—routine logs from 2+ years ago that were never referenced. Compliance explicitly requires deletion after a certain period. Logs have already been archived to Big Objects or external systems, so data is preserved elsewhere.

**What happens:**

Logs are permanently deleted from your org. Issues that still have remaining logs will show reduced log counts. If all logs under an issue are removed, the issue itself is automatically deleted. Historical detail for those specific occurrences is lost, and detailed reporting and analysis for that timeframe is no longer possible.

{% hint style="warning" %}
**Permanent Action**: Removal cannot be undone. If there's any chance you'll need the data later—for compliance, auditing, or trend analysis—archive before removing.
{% endhint %}

### Local Archival

Local archival provides a middle ground between keeping everything and deleting it all. This action copies log data to Big Object storage, preserving it in a different storage tier before removing the original log from custom objects.

**When archival makes sense:**

You want to preserve historical data without consuming expensive Data Storage. Big Object storage is available (every org gets 1M records free, separate allocation). You need storage optimization but anticipate occasional need to reference old logs. Long-term compliance or audit requirements exist but regular access isn't needed.

**Understanding the trade-offs:**

Big Objects have limited query capabilities compared to custom objects. You cannot create standard reports or list views on them. Data must be accessed via SOQL queries (no UI browsing). This makes archived data preserved but less accessible—suitable for occasional reference rather than regular analysis.

{% hint style="info" %}
**Sweet Spot**: Archival is ideal when historical data has value but isn't frequently accessed. You preserve compliance and troubleshooting capability without the storage costs of keeping logs in active custom objects.
{% endhint %}

#### Understanding Big Objects

Big Objects are a special storage type in Salesforce specifically designed for archival and historical data. Think of them as a separate storage tier optimized for preserving large volumes of data that you need to keep but don't access frequently.

{% hint style="warning" %}
**Evolving Capability:** Big Object-backed local archival is a newer part of the retention feature set. Confirm current availability and behavior with your Pharos contact before relying on it as part of a critical retention strategy.
{% endhint %}

**Key characteristics:**

| Aspect         | Details                                                       |
| -------------- | ------------------------------------------------------------- |
| **Cost**       | 1 million records free per org, additional capacity available |
| **Allocation** | Completely separate from Data Storage limits                  |
| **Purpose**    | Archival and historical data preservation                     |
| **Access**     | SOQL queries (with limitations), API access, data export      |

**What you CAN'T do with Big Objects:**

* Create list views or tabs for browsing
* Build standard Salesforce reports directly
* Delete individual records (bulk deletion only)
* Use complex indexing strategies

**What you CAN do:**

* Query via SOQL with some limitations
* Access programmatically via APIs
* Export data for external analysis
* Preserve long-term historical records cost-effectively

{% hint style="info" %}
**Tiered Strategy**: Organizations with mature observability practices commonly use two tiers—active logs in Data Storage (0-6 months) and archived logs in Big Objects (6+ months). If you need to preserve log data outside Salesforce for multi-year compliance, export it with standard Salesforce data tools rather than through a retention rule.
{% endhint %}

## How Retention Rules Execute

For step-by-step setup — the Admin navigation, the rule form field by field, and screenshots — see [Log Retention](https://docs.pharos.ai/configuration-guide/log-retention.md) in the Configuration Guide.

Retention rules execute automatically on a daily schedule without requiring manual intervention. The system evaluates all logs against all active rules to identify which ones meet the age and filter criteria for processing.

Unlike notification rules which stop at the first match, retention rules evaluate all matching rules. Multiple rules can apply to the same logs. Archival actions always execute first (if configured), followed by removal actions. There's no priority ordering between rules—all applicable rules run, and where two rules would remove the same logs, the first successful removal leaves nothing for the others to act on.

{% hint style="info" %}
**Automatic & Daily**: Once configured and activated, retention rules run automatically every day. You don't need to manually trigger them or remember to clean up logs.
{% endhint %}

## Example Retention Strategies

### Strategy 1: Simple Age-Based Removal

**Scenario:** Storage is at 85% capacity and you need to free up space quickly.

The simplest approach is a single rule that removes all logs beyond a certain age, regardless of category or severity. This is easy to configure and understand, though it's a blunt instrument that doesn't distinguish between critical and routine logs.

**Configuration:**

```
Rule: Remove all logs older than 1 year
- Age: 1 year
- Filter Criteria: (none)
- Actions: Removal
```

**Effect:** Deletes all logs created more than a year ago, providing immediate storage relief but losing all historical detail beyond 12 months.

### Strategy 2: Selective Retention by Category

**Scenario:** You want to keep critical logs longer than routine logs, recognizing that Apex logs provide more debugging value than Bulk API logs from routine bulk data loads.

This multi-rule approach applies different retention periods based on log category. It's more sophisticated but provides better balance between storage optimization and preserving valuable debugging information.

**Configuration:**

```
Rule 1: Remove Bulk API logs after 60 days
- Age: 60 days
- Filter: Category equals "Bulk API"
- Actions: Removal

Rule 2: Remove Flow logs after 6 months
- Age: 6 months
- Filter: Category equals "Flow"
- Actions: Removal

Rule 3: (No rule for Apex - keeps indefinitely)
```

**Effect:** Bulk API logs (less debugging value) deleted after 60 days. Flow logs retained for 6 months. Apex logs (high debugging value) kept indefinitely or until general storage cleanup is needed.

### Strategy 3: Archive Then Remove

**Scenario:** You need to free storage space but want to preserve historical data for occasional reference or compliance purposes.

Archiving before removal is the "best of both worlds" approach—you free up expensive Data Storage while preserving logs in cheaper Big Object storage. This is ideal when you don't frequently access old logs but might occasionally need them.

**Configuration:**

```
Rule: Archive and remove logs older than 1 year
- Age: 1 year
- Filter Criteria: (none)
- Actions: 
  - Local Archival (to Big Objects)
  - Removal (from custom objects)
```

**Effect:** Logs older than 1 year are copied to Big Objects, then deleted from custom objects. You free up Data Storage while maintaining long-term historical records accessible via SOQL queries.

### Strategy 4: Functional Area Priority

**Scenario:** You want to keep logs from revenue-impacting areas longer than those from administrative or support functions.

This strategy recognizes that not all logs have equal business value. Order Management and Payment Processing errors directly impact revenue, warranting longer retention than administrative logs.

**Configuration:**

```
Rule 1: Remove non-critical logs after 60 days
- Age: 60 days
- Filter: Functional Area NOT equals "Order Management" 
         AND Functional Area NOT equals "Payment Processing"
- Actions: Removal

Rule 2: Remove critical area logs after 2 years
- Age: 2 years  
- Filter: Functional Area equals "Order Management" 
         OR Functional Area equals "Payment Processing"
- Actions: Local Archival, Removal
```

**Effect:** Administrative and support logs deleted after 60 days. Revenue-impacting logs archived and then removed after 2 years, providing long-term visibility for critical business functions while aggressively managing less important logs.

### Strategy 5: One-Time Aggressive Cleanup

**Scenario:** Storage is at 95% capacity and you need immediate emergency relief.

Sometimes you face a storage crisis that requires aggressive short-term action. A one-time cleanup rule removes logs more aggressively than your long-term strategy would allow, providing breathing room to implement a sustainable approach.

**Configuration:**

```
Rule: Emergency cleanup - remove logs older than 30 days
- Age: 30 days
- Filter Criteria: (none)
- Actions: Removal
```

**Implementation process:**

1. Create and activate the rule
2. Allow it to run overnight (rules execute daily)
3. Check storage usage the next day
4. **Deactivate or delete the rule** once cleanup achieves target
5. Implement a sustainable long-term retention strategy

{% hint style="warning" %}
**Critical Step**: One-time aggressive rules MUST be deactivated after achieving the desired result. If left active, they'll continue deleting logs at 30 days indefinitely—far more aggressive than most orgs need long-term.
{% endhint %}

## What Rule Changes Do and Don't Affect

Retention actions are one-way: edits and deletions change what happens *next*, never what already happened. Previously removed logs stay removed, and previously archived logs are unaffected. Issues that lost all their logs to an earlier run remain deleted.

This is why the safe sequence is to deactivate a rule before reworking it — and why some fields lock once a rule has run. For the editing rules and constraints, see [Log Retention](https://docs.pharos.ai/configuration-guide/log-retention.md#editing-active-rules).

To judge whether your rules are doing what you intended, watch log counts before and after a run, storage-usage trends in Setup → Storage Usage, and whether archived records are landing in Big Objects.

## Best Practices

### Start Conservative

Begin with longer retention periods and adjust based on actual experience rather than guessing at aggressive thresholds upfront:

* Start with 1-2 year retention for initial rules
* Monitor storage impact for 30-60 days before making changes
* Adjust retention periods based on actual storage consumption patterns
* Gradually reduce retention only if storage pressure continues

### Plan Before Implementing

Ask these key questions before configuring retention rules:

* What is our current storage usage and growth rate?
* What retention period balances historical analysis with capacity?
* Do different log categories warrant different retention periods?
* Should we archive before removing to preserve long-term history?
* What compliance or audit requirements apply?

### Test First

Test rules carefully before activating aggressive retention:

1. Create rule with very old age threshold initially (e.g., 3 years)
2. Verify rule executes and processes logs as expected
3. Check that filtering works correctly
4. Confirm archival is working (if configured)
5. Then adjust to actual desired threshold

### Monitor Impact

Track these metrics after implementing retention rules:

* Storage usage before and after retention runs
* Number of logs removed per execution
* Log count trends over time
* Any unexpected deletions or pattern changes

### Document Your Strategy

Maintain documentation for future administrators:

* Why specific retention periods were chosen
* Business justification for each rule
* Explanation of filter criteria
* Review schedule and ownership
* Decision history (why rules were changed)

## Advanced Configurations

### Complex Filter Logic

For sophisticated retention criteria, use filter logic to combine multiple conditions:

**Example:** Remove logs from specific areas OR a specific technology

```
Filter 1: Functional Area equals "Integration"
Filter 2: Functional Area equals "Batch Jobs"  
Filter 3: Category equals "Flow"

Filter Logic: (1 OR 2) OR 3
```

This removes logs matching any of these conditions—either Integration or Batch Jobs functional areas, or any Flow category logs. To purge by severity, filter on the log level rather than Category.

### Multiple Action Combination

Combine archival and removal to preserve history while freeing storage:

```
Action 1: Local Archival (copy to Big Objects)
Action 2: Removal (delete from custom objects)
```

**Execution guarantee:** Archival always executes first, then removal. This order is automatic regardless of how you add the actions, ensuring data isn't lost before archival completes.

### Selective Big Object Archival

Apply different archival strategies based on log importance:

```
Rule 1: Archive critical logs before removal
- Age: 1 year
- Filter: Functional Area equals "Payment Processing"
- Actions: Local Archival, Removal

Rule 2: Remove non-critical logs without archival
- Age: 6 months
- Filter: Functional Area NOT equals "Payment Processing"
- Actions: Removal only
```

This preserves critical payment logs in Big Objects while deleting less important logs without archival.

## Retention and Compliance

### Compliance Considerations

Data retention requirements vary significantly across industries and regulations. Some mandate **minimum** retention periods (keep data for X years), while others impose **maximum** retention periods (delete after Y months for privacy). Configure retention rules to comply with applicable regulations.

**Common regulatory frameworks:**

**SOX (Sarbanes-Oxley)** - May require multi-year retention of financial system logs\
**HIPAA** - Healthcare data retention and disposal requirements\
**GDPR** - Right to deletion may require shorter retention periods\
**Industry-specific** - Follow standards for your sector (PCI-DSS, etc.)

{% hint style="warning" %}
**Legal Requirements Trump Storage Optimization**: If compliance requires 7-year retention, you can't implement 1-year retention even if storage is constrained. Purchase additional storage or archive externally instead.
{% endhint %}

### Audit Trail

Understanding what's preserved vs. lost helps make informed retention decisions:

**What remains after log deletion:**

* Issue records (high-level view of what problems occurred)
* Issue metadata (occurrence counts, dates, impact metrics)
* Audit trail (triage, resolution actions taken)
* Archived logs (if archival configured)

**What is lost:**

* Detailed log content and context
* Individual occurrence details
* Stack traces and error messages
* Full debugging information

## Troubleshooting

Retention problems are almost always one of a few things: the rule or the global toggle is inactive, no logs are old enough yet, filters are narrower than intended, or archival moved records to Big Objects (which frees Data Storage but not Big Object storage). See [Log Retention](https://docs.pharos.ai/configuration-guide/log-retention.md#troubleshooting) for the diagnostic table.

## Related Topics

* [Storage Usage](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/logs/log-storage/storage-usage.md) - Monitor and understand storage consumption
* [Native Data Model](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/logs/log-storage/native-data-model.md) - Learn about Pharos custom objects
* [Log Storage](https://docs.pharos.ai/platform-overview/pharos-platform-architecture/logs/log-storage.md) - Overview of log storage in Pharos
