Profiling

Continuous profiling is provided by Grafana Pyroscope, deployed automatically as part of the observability stack.

There are two capture mechanisms in the fleet, and which one applies depends on the workload:

• Cassandra uses runtime-controlled async-profiler, attached to the already-running JVM and driven by the cassandra profile command group. Nothing profiling-related is injected at JVM startup.
• Every other JVM workload — stress jobs, Presto/Trino, Spark/EMR, the Cassandra Sidecar — still uses the Pyroscope Java agent, loaded with -javaagent at startup.
• Everything else (ClickHouse, TiDB/TiKV/PD, and all host processes) is covered by the Grafana Alloy eBPF profiler.

Architecture

Profiling data is collected from these sources and sent to the Pyroscope server on the control node (port 4040):

• async-profiler, attached at runtime (Cassandra) — A systemd timer on every Cassandra node runs edl-profiling-reconcile, which attaches async-profiler to the live Cassandra JVM, writes rotating JFR chunks to /mnt/db1/cassandra/profiles, uploads completed chunks to Pyroscope's /ingest, and prunes the directory. What is being profiled changes without restarting Cassandra.
• Pyroscope Java agent (Stress jobs) — Runs as a -javaagent inside cassandra-easy-stress K8s Jobs. Collects CPU, allocation, and lock profiles. The agent JAR is mounted from the host via a hostPath volume.
• Pyroscope Java agent (Presto) — Runs as a -javaagent inside both the Presto coordinator and worker JVMs, injected via JAVA_TOOL_OPTIONS during presto start. Profiles appear under service_name=presto with component=coordinator or component=worker.
• Pyroscope Java agent (Spark/EMR) — Runs as a -javaagent on Spark driver and executor JVMs, installed via EMR bootstrap action to /opt/pyroscope/pyroscope.jar. Profiles appear under service_name=spark-<job-name>.
• Grafana Alloy eBPF profiler — Runs as a DaemonSet on all nodes via Grafana Alloy. Profiles all processes at the system level using eBPF, including kernel stack frames. Pod processes are attributed per pod/container/service_name; see the eBPF Agent section below.

Cassandra profiling at runtime

A cluster is profiling as soon as it comes up — setup-instances seeds each Cassandra node with profiling enabled and -e cpu --alloc 512k --lock 10ms. Nothing needs to be run to get CPU, allocation and lock-contention flame graphs. Only wall-clock sampling is exclusive of a CPU event, so all three of these run in one recording.

Why it is not a startup agent

The Pyroscope Java agent takes exactly one primary profiler.event, fixed for the JVM's lifetime. Changing what was being profiled therefore meant restarting Cassandra, which discards page cache and compaction state — precisely what an operator on a benchmarking rig is trying to observe. Attaching at runtime makes the event set a runtime property instead. It also takes profiling out of the startup path entirely, so a bad profiler configuration can no longer abort Cassandra's start.

The cost, worth knowing: attach-based profiling dies with the JVM, so a node-resident reconciler re-establishes it. After a Cassandra restart or a mode switch, expect up to one reconcile interval (60s by default) with no profiling. That is expected behaviour, not a bug.

Switching modes

There is one async-profiler session per JVM, so one mode at a time. Switch by stopping and starting:

# what is running right now, per node
easy-db-lab cassandra profile status

# switch the whole cluster to wall-clock profiling
easy-db-lab cassandra profile stop
easy-db-lab cassandra profile start -- -e wall -i 10ms

# back to CPU, with allocation sampling alongside it
easy-db-lab cassandra profile stop
easy-db-lab cassandra profile start -- -e cpu --alloc 512k

# only two nodes, rotating every 30 seconds
easy-db-lab cassandra profile start --hosts db0,db1 --loop 30s -- -e cpu

stop finalizes the in-flight JFR chunk before detaching, so a mode switch loses no data — the outgoing chunk still ships, and fetch and flamegraph can reach it once nothing is writing to it. stop also leaves your retention and byte ceiling exactly as start set them; it only turns profiling off. If a node's configuration document cannot be read — truncated by an interrupted write, say — stop has nothing to preserve and falls back to the defaults, and it says so with a Profiling.DesiredStateUnreadable event naming the node and the bounds it is applying, rather than resetting them silently.

Every profiling command applies to all Cassandra nodes by default; --hosts narrows it to a subset.

async-profiler's own parameters pass through

Anything after -- is handed to asprof byte-identically. easy-db-lab does not model, enumerate, or re-parse async-profiler's option surface, so options added by newer async-profiler releases work with no change here.

easy-db-lab cassandra profile start -- -e wall -i 10ms
easy-db-lab cassandra profile start -- -e cpu --lock 10ms --alloc 512k

Reserved parameters

Five parameters are rejected, at the CLI, before any node is contacted. Four of them easy-db-lab supplies itself, because the node-side JFR shipper depends on them; the fifth changes which JFR event types reach that shipper at all:

Rejection covers both of async-profiler's spelling systems (CLI flags and the comma-separated agent-option form) and reserved options smuggled inside another value after a comma, such as -e cpu,file=/tmp/elsewhere.jfr. That last case is the reason the check exists at all: without it, profiling would appear to work while chunks landed somewhere the shipper never looks and Pyroscope stayed silently empty.

The first four remove no capability. Every profiling parameter async-profiler offers remains reachable, and the one knob from that set people actually want — rotation — is a first-class option on the command itself:

easy-db-lab cassandra profile start --loop 30s -- -e cpu

Retention on the node

Shipped chunks are pruned by age and by total size: their data is already in Pyroscope. A chunk that has not shipped, or that Pyroscope rejected, is never deleted — it is data you cannot get back.

When the directory reaches --max-bytes and only unshipped or rejected chunks are left, the node stops recording instead of deleting anything, so an unreachable Pyroscope still cannot fill /mnt/db1 and take the database down. Recording resumes on its own once the directory is back under the bound — when shipping recovers and shipped chunks are pruned, or when you fetch and remove chunks. The stop is reported by cassandra profile status (STOPPED), a typed Profiling.RecordingStoppedAtBound event, the jfr_recording_stopped journal line, and the edl_jfr_recording_stopped_at_bound metric.

easy-db-lab cassandra profile start --retention 30 --max-bytes 1073741824 -- -e cpu

Profiles live in /mnt/db1/cassandra/profiles, deliberately not artifacts/ — that directory is world-writable and holds operator-dropped heap dumps and jstacks, which automatic pruning must never delete.

Fetching chunks and building flame graphs

# download the 5 most recent completed chunks from every node into ./profiles/<host>/
easy-db-lab cassandra profile fetch

easy-db-lab cassandra profile fetch --last 20 --hosts db0

# convert the 10 most recent chunks into a flame graph on the node and download it
easy-db-lab cassandra profile flamegraph --last 10

# split stacks by thread — see below
easy-db-lab cassandra profile flamegraph --last 10 -- --threads

Conversion runs on the node, because jfrconv is installed there and not on your machine, and because the rendered HTML is far smaller than the raw JFR.

jfrconv arguments pass through after -- the same way async-profiler's do. Only its input and output are reserved — positional arguments and -o/--output are rejected, because easy-db-lab supplies the input chunks (--last) and the output destination (--format).

The chunk currently being written is never offered by fetch or flamegraph: async-profiler finalizes constant pools when a chunk closes, so an unfinalized chunk is unreadable by any tool. Everything else on the node is offered, including chunks already shipped to Pyroscope — shipping marks a chunk with a -shipped.jfr infix rather than moving it aside, so --last 20 really does give you twenty chunks. Once you stop profiling, the final chunk of the run becomes available too, since nothing is writing to it any more.

Thread-pool attribution

jfrconv --threads on a fetched chunk is the only way to see Cassandra thread-pool attribution (ReadStage-3, CompactionExecutor:4, MutationStage-7).

Pyroscope's JFR ingest discards thread identity: the JFR carries sampledThread and Grafana's parser decodes it, but the pprof converter never reads it. Passing -t to async-profiler does not help either — in JFR output mode it deliberately puts thread identity in the constant pool rather than into the stack. So the continuous-profiling path structurally cannot show you which pool a stack came from, and the flamegraph path exists alongside it precisely for this:

easy-db-lab cassandra profile flamegraph --last 20 -- --threads

Status

easy-db-lab cassandra profile status

Reports per node: what you asked for (desired) and what is actually attached to the JVM (attached), reported separately, the process being profiled, how long the session has been running, your arguments verbatim, the full command line as actually invoked (including the -o jfr --loop ... -f ... the tool appended), chunks pending/shipped/rejected, bytes held on disk, and the most recent shipping and attach errors.

It also reports pruned: — how many shipped chunks the node reclaimed for age and for size — and on disk: as bytes held against the --max-bytes ceiling.

A node that stopped recording at its byte ceiling is marked STOPPED and reports attached: no (stopped at the size bound). It emits a typed Profiling.RecordingStoppedAtBound event, and no Profiling.AttachFailed: nothing is wrong with the JVM. Fix shipping, raise --max-bytes, or fetch and remove chunks; recording resumes on the next pass once the directory is back under the ceiling.

The report is a snapshot the node's reconciler left behind, not a live reading, so status also prints updated: — how long ago that node last completed a pass. If the reconciler has stopped running, the report is marked STALE above everything else in it and names the unit to check, and status emits a typed Profiling.StateStale event. Without that, a node whose timer is masked keeps reporting attached: yes with a session age that grows convincingly against the current clock while nothing is running.

A node whose configuration document cannot be read is marked CONFIG instead, never STALE, and emits Profiling.NodeConfigUnreadable. The two are opposite diagnoses. A pass that cannot read /etc/easy-db-lab/profiling.json refuses only the attach/detach decision — it keeps shipping and pruning under the bounds the last good pass recorded, and keeps reporting every metric — so the reconciler is running perfectly and the file it reads is what is wrong. Run cassandra profile start to rewrite it. A node with no document at all has simply never been configured; it is not marked CONFIG.

desired and attached are two lines rather than one because they answer different questions. A node showing desired: enabled with attached: no is a node whose profiler will not attach — usually PrivateTmp hiding the attach socket, a java.io.tmpdir mismatch, or perf_event_paranoid — and status emits a typed Profiling.AttachFailed event with the reconciler's captured error, rather than letting it look like a deliberate stop.

A node that has just started is marked WAITING instead, and reports attached: no (waiting for the database to become ready). The reconciler will not attach to a database whose native transport is not yet listening, because async-profiler attaches with jattach, jattach signals SIGQUIT, and a process that has not installed a handler for that signal yet is killed by it — which is how attaching to a starting node used to kill it. This state clears itself on the first pass after the database is up, so it emits a typed Profiling.AttachDeferred event, which is not an error, and no Profiling.AttachFailed. If it does not clear, see edl_jfr_attach_deferred_total below.

Shipping and attach problems are also visible without the CLI. The reconciler writes structured logfmt lines to journald, which Fluent Bit ships to Loki, and exports counters to the node's OTel collector, which forwards them to Mimir:

Each successful attach also logs event=jfr_session_started ... ready=<code>, naming which readiness answer allowed it. ready=listening is the healthy answer. A node — or a whole cluster — attaching on ready=uptime is profiling ten minutes late every time: the socket check is not finding the native transport, and the ten-minute backstop is doing work it was only meant to do for a transport that is deliberately off.

Every series carries the node's host_name and the cluster label, the same as the rest of the stack.

The cpu+wall hazard

Combining a CPU event and wall-clock sampling in one recording corrupts the CPU sample weights, silently, by up to three orders of magnitude.

The defect is upstream, in Pyroscope's JFR ingest. Pyroscope delegates to github.com/grafana/jfr-parser (pinned at v0.18.0), whose pprof/parser.go:56 declares the sample value buffer once, outside the event loop:

var values = [2]int64{1, 0}

The wall-clock branch assigns values[0] = WallClockSample.Samples and never restores it, and the execution-sample branch reuses the buffer without re-initialising it. After the first wall sample, every subsequent CPU sample carries that wall event's batch count as its weight. Wall sampling coalesces idle threads into batches of up to 1000, and Cassandra runs hundreds of mostly-idle pool threads — hence the magnitude.

Three things follow:

• The corruption is scoped to a single upload. Separate sessions are uploaded separately and are each clean. Switching modes with stop/start is the supported approach and is not affected.
• --nobatch is not a mitigation, and easy-db-lab rejects it. It does not merely disable coalescing: it changes the emitted event type, so wall samples arrive as jdk.ExecutionSample — indistinguishable from CPU samples — and Pyroscope merges them all into process_cpu. It causes exactly the failure it appears to prevent. async-profiler itself refused the flag until 4.5 accepted it, so the guard now lives in easy-db-lab.
• Getting CPU, wall, and allocation profiles requires more than one session, in sequence. cpu+alloc in one session is fine; wall belongs in its own.

Note also that CPU and wall sampling have very different costs. A CPU sample only fires on a running thread, so its rate is bounded by core count. A wall tick targets threads regardless of run state, so its rate is bounded by thread count — far higher on a Cassandra node. That is why wall is treated as its own mode rather than something left on by default.

Accessing profiles

Profiling dashboard

1. Open Grafana (port 3000)
2. Navigate to Dashboards and select the Profiling dashboard
3. Use the Service dropdown to select a service (e.g. cassandra, cassandra-easy-stress, clickhouse-server)
4. Use the Hostname dropdown to filter by specific nodes
5. Select a time range

The dashboard includes panels for:

• CPU Flame Graph — CPU time spent in each method
• Wall Clock Flame Graph — Wall-clock time, for finding I/O and blocking
• Memory Allocation Flame Graph — Heap allocation hotspots
• Lock Contention Flame Graph — Time spent waiting for monitors
• Mutex Contention Flame Graph — Mutex delay analysis

Grafana Explore

1. Open Grafana (port 3000) and navigate to Explore
2. Select the Pyroscope datasource
3. Choose a profile type (e.g. process_cpu, memory, mutex, wall)
4. Filter by labels: service_name, hostname, cluster

Cassandra profiles arrive under service_name=cassandra with hostname and cluster labels.

Pyroscope runs native multi-tenancy, and the Grafana Pyroscope datasource sends the cluster's tenant (init --tenant, default default) on every query, so Explore and the dashboards show the cluster's tenant. Every profile writer sends the same tenant in X-Scope-OrgID. A request made directly against Pyroscope's HTTP API (port 4040) must send it too: curl -H 'X-Scope-OrgID: <tenant>' ....

Pyroscope UI

Pyroscope 2's own UI is served on port 4040, and the profiling dashboard links to it as Pyroscope UI. On the first visit the UI finds that the server runs multi-tenancy (its first query is refused with 401 no org id) and shows an Enter a Tenant ID dialog. Enter the cluster's tenant: the value of init --tenant, default if none was given. The UI remembers it per browser (in localStorage, as pyroscope:tenantID) and sends it as X-Scope-OrgID on every query; the navbar can change it. The dashboard link cannot preselect the tenant, because the UI reads no URL parameter for it.

Profile types

Cassandra (runtime async-profiler)

Which types are populated depends on the session that is running, since one session runs at a time.

Thread identity is not available through this path; use flamegraph -- --threads (see above).

Java agent (stress jobs, Presto, Spark, sidecar)

eBPF agent (all processes)

The eBPF agent profiles all processes on every node, including ClickHouse and other kit databases (TiDB, TiKV, PD). Since these are written in C++/Go, only CPU profiles are available.

Processes running inside Kubernetes pods are attributed to their pod: they carry namespace, pod, container, and a service_name derived as <namespace>/<container> (for example tidb-cluster/tikv). Host processes that don't run in a pod are still profiled, just without pod labels.

Stress job profiling

Stress jobs are automatically profiled via the Pyroscope Java agent. No configuration is needed — when you start a stress job, the agent is mounted from the host node and configured to send profiles to the Pyroscope server. Profiles appear under service_name=cassandra-easy-stress with cluster and job_name labels.

Configuration

Cassandra runtime profiling

You should not need to edit these by hand; cassandra profile start/stop own the desired-state document, and stopping is recorded as an explicit disabled state rather than by deleting the file.

eBPF agent

The eBPF profiler runs as a privileged Grafana Alloy DaemonSet (pyroscope-ebpf) and profiles all processes on each node. Configuration is in the pyroscope-ebpf-config ConfigMap (Alloy River format). It uses discovery.kubernetes to discover the pods on each node, discovery.process (joined to those pods by container id) to discover host processes, and pyroscope.ebpf to collect CPU profiles. The DaemonSet runs under the pyroscope-ebpf ServiceAccount, whose ClusterRole grants read access to pods so samples can be attributed to a pod/container/service_name.

Pyroscope server

The Pyroscope server (2.3.1) runs on the control node with pure v2 storage in the account bucket under s3://<account-bucket>/pyroscope/, and native multi-tenancy. Nothing is deleted by age: the metastore's retention cleanup is off and there is no retention period. Its metastore index lives on the control node at /mnt/db1/pyroscope, so profiles stay queryable across a Pyroscope restart. Pyroscope accepts a push only after its segment is in S3, so down needs no flush for profiles and reports that. After down the profiles stay in S3, but v2 cannot find them without the cluster's index (see Where observability data is stored). Configuration is in the pyroscope-config ConfigMap.

Data flow

Cassandra is the one path where nothing is injected into the JVM at startup and nothing pushes from inside the process — the reconciler on each node reads finished chunks off disk and uploads them.

Cassandra JVM
    │ (async-profiler attached at runtime, writes JFR chunks)
    ▼
/mnt/db1/cassandra/profiles/*.jfr
    │ (edl-profiling-reconcile.timer, every 60s: ship completed chunks, prune)
    ├──────────────► Pyroscope Server (:4040) ──┐
    │                                            │
    └──(fetch / flamegraph)──► your workspace    │
                                                 │
Stress Jobs   ──(Java agent)────────────────────►│
Presto JVMs   ──(Java agent)────────────────────►│
Spark JVMs    ──(Java agent)────────────────────►│
All Processes ──(eBPF agent)────────────────────►│
                                                 │
                                                 ▼
                                            S3 storage
                                          Grafana (:3000)
                                     Pyroscope datasource
                                    + Profiling dashboard