Drop Filters - Reduce Data Costs

What are Drop Filters?

Drop filters let you exclude specific logs, metrics, or traces before they're stored. This means you don't pay for data you don't need, while still keeping the data that matters.

40%

Avg. reduction from DEBUG logs

25%

Avg. reduction from health checks

15%

Avg. reduction from unused metrics

How It Works

// Data Flow

Your App → OTLP Endpoint → Drop Filters → Storage → Dashboard
↓
Dropped data
(not stored, not billed)

Pre-ingestion filtering: Data is filtered before it hits storage
No SDK changes needed: Filters are applied server-side
Instant savings: Billing reflects filtered data immediately
Reversible: Delete a filter to start collecting that data again

Creating Drop Filters

Via Dashboard

Go to Settings → Drop Filters
Click Create Filter
Select the data type (Logs, Metrics, or Traces)
Enter your filter criteria
Click Save

Via API

curl -X POST https://qorrelate.io/v1/organizations/{org_id}/drop-filters \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Drop DEBUG logs",
    "type": "logs",
    "query": "severity:DEBUG",
    "enabled": true
  }'

Log Filters

Log filters use Lucene query syntax to match logs. Any log matching the query will be dropped.

Syntax Examples

Drop all DEBUG logs

severity:DEBUG

Matches any log with severity level DEBUG

Drop health check logs

body:"/health" OR body:"/healthz" OR body:"/ready"

Matches logs containing common health check paths

Drop logs from specific service

service.name:internal-scheduler AND severity:INFO

Drop INFO logs from a noisy internal service

Drop logs matching pattern

body:"Connection pool" AND body:"recycled"

Drop repetitive connection pool messages

Metric Filters

Metric filters use pattern matching on metric names and labels.

Examples

Drop test metrics

{
  "metric_name": "test_*"
}

Drop all metrics starting with "test_"

Drop metrics from staging

{
  "metric_name": "*",
  "label_filters": {
    "environment": "staging"
  }
}

Drop all metrics with environment=staging label

Drop high-cardinality internal metrics

{
  "metric_name": "go_*"
}

Drop Go runtime metrics (often not needed)

Trace Filters

Trace filters can drop entire traces or individual spans based on attributes.

Drop health check traces

{
  "span_name": "GET /health*"
}

Drop traces under 10ms (noise)

{
  "max_duration_ms": 10
}

Common Patterns

Here are the most effective drop filters used by Qorrelate customers:

Pattern	Filter	Typical Savings
DEBUG logs in production	`severity:DEBUG`	30-50%
Kubernetes health checks	`body:"/healthz"`	10-25%
Load balancer logs	`service.name:nginx AND status:200`	15-30%
Go runtime metrics	`metric_name: go_*`	5-15%
Static asset requests	`body:".js" OR body:".css" OR body:".png"`	10-20%

Monitoring Your Savings

Track how much data your filters are dropping:

Via Dashboard

Go to Settings → Drop Filters to see per-filter statistics:

Events dropped (last 24h, 7d, 30d)
Estimated cost savings
Percentage of total data dropped

Via API

curl https://qorrelate.io/v1/organizations/{org_id}/drop-filters/stats \
  -H "Authorization: Bearer YOUR_API_KEY"

# Response:
{
  "filters": [
    {
      "id": "filter_123",
      "name": "Drop DEBUG logs",
      "events_dropped_24h": 1523456,
      "bytes_saved_24h": 892341234,
      "estimated_savings_usd": 12.50
    }
  ],
  "total_events_dropped_24h": 2341567,
  "total_savings_usd": 18.75
}

API Reference

List Drop Filters

GET /v1/organizations/{org_id}/drop-filters

Create Drop Filter

POST /v1/organizations/{org_id}/drop-filters

{
  "name": "Filter name",
  "type": "logs",          // logs, metrics, or traces
  "query": "severity:DEBUG",
  "enabled": true
}

Update Drop Filter

PUT /v1/organizations/{org_id}/drop-filters/{filter_id}

{
  "enabled": false  // Disable without deleting
}

Delete Drop Filter

DELETE /v1/organizations/{org_id}/drop-filters/{filter_id}

Important: Dropped Data Cannot Be Recovered

Once data is dropped by a filter, it's gone forever. Start with specific filters and expand gradually. Use the "test mode" to preview what would be dropped before enabling.