OpenTelemetry Profiling Just Hit Alpha. We Already Ingest It. Here’s How.

This is a big deal. And not just because profiling is useful—but because of what it completes.

For the first time, there’s a vendor-neutral, open standard that covers the entire observability stack: why a request was slow (traces), what the system looked like when it happened (metrics), what it logged (logs), and where the CPU and memory actually went (profiles). No proprietary agents. No lock-in. Just OTLP.

We’ve been waiting for this moment at Qorrelate. So we didn’t wait—we started building profiling ingestion and visualization months ago, tracking the spec as it evolved. As of this week, Qorrelate ingests OTel profiling data in production, stores it in ClickHouse, and correlates it with traces end-to-end.

This post covers what OTel Profiling means for the industry, how we built our profiling backend, and why trace-to-profile correlation changes everything.

What OTel Profiling Actually Is

If you’ve used tools like pprof, async-profiler, or perf, you already know continuous profiling. The idea is simple: periodically sample what your application is doing—which functions are on the CPU, where memory is being allocated—and aggregate those samples into flamegraphs that show you exactly where time and resources are spent.

What’s new is the standardization. Until now, profiling was a fragmented world—different formats (pprof, JFR, collapsed stacks), different collection agents, different backends. The OTel Profiling signal changes that by defining:

• A standard data model for profiling data within the OTLP protocol
• A standard eBPF-based collection agent (donated by Elastic) that works across C/C++, Go, Rust, Python, Java, Node.js, .NET, PHP, Ruby, and Perl—with zero code changes
• Native correlation with traces via Link messages in the profiling protobuf

That last point is the one most people are sleeping on.

Why the Fourth Signal Matters More Than People Think

The observability industry has spent a decade building around three signals. But there’s always been a gap: traces tell you that something was slow, metrics tell you when it got slow, logs tell you what happened around the time it got slow—but none of them tell you why the code was slow at the CPU level.

Was it a hot loop? Excessive garbage collection? Lock contention? A slow regex? You’d leave your observability tool and go fire up a profiler separately, try to reproduce the issue, and hope you caught it. That workflow is broken.

With OTel Profiling, the profiler runs continuously alongside your existing OTel instrumentation. When a trace span executes, the eBPF profiler detects the active span context on the thread and automatically links the profiling samples to that span. No application code changes. No manual correlation.

The result: you click on a slow span in your trace waterfall, and you see the flamegraph showing exactly where that span spent its CPU time. That’s the observability loop that’s been missing.

How We Built Profiling Ingestion on ClickHouse

Most observability backends weren’t designed for profiling data. The challenge is that profiling generates a lot of data—continuous profiling at 100Hz across a fleet of services produces millions of samples per hour—but most of that data is highly repetitive. The same call stacks appear thousands of times. A hot handleRequest → dbQuery → net.Write path that fires 10,000 times per hour doesn’t need to store the full stack trace 10,000 times.

We took an approach inspired by the Coroot pattern: two tables, one join.

The Schema

profile_samples (MergeTree)—the measurements:

organization_id     String
service_name        LowCardinality(String)
profile_type        LowCardinality(String)    -- 'cpu', 'alloc_objects', etc.
start_time          DateTime64(9)
stack_hash          UInt64                     -- FK to profile_stacks
value               Int64                      -- sample value (CPU ns, bytes, count)
trace_id            String                     -- OTel Profiles trace correlation
span_id             String
labels              Map(String, String)

profile_stacks (ReplacingMergeTree)—deduplicated stack traces:

organization_id     String
service_name        LowCardinality(String)
stack_hash          UInt64                     -- deterministic hash of frame array
last_seen           DateTime64(9)
stack               Array(String)              -- ['main.go:10:main', 'handler.go:25:handleRequest', ...]

Why Two Tables?

Stack deduplication. In continuous profiling, the same call stacks appear thousands of times. That hot handleRequest → dbQuery → net.Write path stores the full stack once in profile_stacks, and 10,000 tiny rows (hash + value + timestamp) in profile_samples. Most profiling backends store the full stack on every sample—we see 10–50x storage savings compared to that approach.

Aggregation becomes a single CTE + JOIN. Building a flamegraph across any time window:

WITH samples AS (
    SELECT stack_hash, sum(value) AS value
    FROM profile_samples
    PREWHERE organization_id = {org}
        AND start_time >= {start} AND start_time <= {end}
    WHERE service_name = {svc} AND profile_type = {pt}
    GROUP BY stack_hash
),
stacks AS (
    SELECT stack_hash AS hash, any(stack) AS stack
    FROM profile_stacks
    PREWHERE organization_id = {org}
    WHERE service_name = {svc}
    GROUP BY stack_hash
)
SELECT samples.value, stacks.stack
FROM samples
INNER JOIN stacks ON samples.stack_hash = stacks.hash

ClickHouse aggregates millions of sample rows down to a few thousand unique stacks, then joins once. The PREWHERE on organization_id prunes partitions before the scan even starts.

The profile_stacks table uses ReplacingMergeTree(last_seen) so background merges automatically keep only the most recent version of each stack hash—no garbage accumulation, no manual cleanup.

Differential Profiling

Same pattern, different time windows. Run the CTE for a baseline period and a comparison period, merge the results, and compute delta_pct = ((comparison - baseline) / baseline) * 100 per frame. Red means regression. Blue means improvement. You can answer “did last Tuesday’s deploy make checkout slower?” in seconds.

Timeline Queries

Skip the stacks table entirely—just GROUP BY toStartOfInterval(start_time, INTERVAL N SECOND) on profile_samples alone. This powers the profiling timeline view that shows CPU/memory utilization over time without ever touching the stack data.

Trace ↔ Profile Correlation: The Full Loop

This is the part we’re most excited about. We have three correlation paths, all live today:

1. OTLP Native Link Messages

The OTel Profiles protobuf includes a Link table in the ProfilesDictionary. Each Sample carries a linkIndex pointing into that table, where each link has a traceId and spanId. At ingestion time, we resolve the link index and store the trace/span IDs directly on the sample row:

link_index = sample.get("linkIndex", 0)
s_trace_id, s_span_id = "", ""
if 0 < link_index < len(dictionary["links"]):
    link = dictionary["links"][link_index]
    s_trace_id = link["trace_id"]
    s_span_id = link["span_id"]

Every sample collected during a traced span execution carries the exact trace_id:span_id pair. The eBPF profiler populates these links automatically when it detects an active OTel span context on the profiled thread—no application code changes needed.

2. Trace → Profile (Span Detail Drawer)

Click a span in the trace waterfall. The “Profiles” tab loads via GET /v1/profiles/trace/{traceId}. The backend query is straightforward—WHERE trace_id = {tid} on profile_samples, grouped by profile_id. The UI renders profile cards with type-colored badges and a “View Profile” button that navigates directly to the flamegraph.

3. Profile → Trace (Profiling View)

The profile list table shows a clickable trace ID column. The flamegraph detail header shows a “View linked trace” link. The trace correlation panel at the bottom of the flamegraph shows all unique trace IDs found in the current profile’s samples, each clickable to navigate to the full trace view.

Why This Matters

Most profiling tools show you what is slow (hot function) but not why it was invoked. With trace correlation, you see the flamegraph in context: this dbQuery was hot because it was called from checkout → processPayment → chargeCard → dbQuery—and you can click through to the exact trace that triggered it, see the HTTP request that caused it, and pivot to the logs for that request.

Profile → Trace → Logs → back to Profile. That’s the full observability loop, powered entirely by open standards.

What This Means for the OTel Ecosystem

OTel Profiling entering alpha is a milestone, but it’s still early. The community needs:

• More backends to support the profiling signal. Right now, very few OTel-compatible backends can ingest profiling data. We’re hoping to contribute our learnings back to the community—particularly around ClickHouse storage patterns and the collector exporter.
• Production battle-testing. The eBPF profiler is solid, but operational patterns for running it at scale (sampling rates, resource budgets, fleet-wide rollouts via OpAMP) are still being discovered.
• Better symbolization. Resolving function names and source locations across different runtimes and build systems is still one of the hardest parts of profiling. The OTel community is actively working on standardizing symbolization APIs.
• Semantic conventions for profiling. What profile types should be standard? How should profiling labels map to resource attributes? These conventions are still being defined.

We’re actively contributing to several of these areas and plan to share more of our implementation details—including the ClickHouse schema, query patterns, and collector configuration—as open-source resources.

Getting Started

If you want to try OTel Profiling today:

1. Deploy the eBPF profiler—the opentelemetry-ebpf-profiler agent runs as a DaemonSet on Kubernetes or a systemd service on VMs. It requires zero application code changes.
2. Configure the OTel Collector with a profiling-capable exporter. The collector receives profiling data via OTLP and forwards it to your backend.
3. Point it at a backend that supports profiles. Qorrelate ingests profiling data out of the box. If you’re already sending traces, metrics, and logs to Qorrelate, adding profiles is a single collector config change.

The fourth signal is here. The question isn’t whether profiling becomes a standard part of the observability stack—it’s who builds the best experience around it. We’re betting that experience should be open, correlated, and cost-efficient.

---

Interested in how we built this, have questions about the ClickHouse schema, or want to contribute to OTel Profiling? Find us in #opentelemetry on the CNCF Slack or reach out at support@qorrelate.io.