easy-db-lab

easy-db-lab creates lab environments for database evaluations in AWS. It provisions infrastructure, deploys databases, and sets up a full observability stack so you can focus on testing, benchmarking, and learning.

Supported databases include Apache Cassandra, ClickHouse, TiDB, and OpenSearch, with Trino, Presto, and Apache Spark available for analytics workloads.

If you are looking for a tool to aid in stress testing Cassandra clusters, see the companion project cassandra-easy-stress.

If you're looking for tools to help manage Cassandra in production environments please see Reaper, cstar, and K8ssandra.

Quick Start

1. Install easy-db-lab
2. Set up your profile - Run easy-db-lab profile setup
3. Follow the tutorial

Features

Database Support

• Apache Cassandra: Released versions and branch builds that ship in the AMI, plus custom builds from a local source tree. Includes Cassandra Sidecar and integration with cassandra-easy-stress for benchmarking.
• ClickHouse: Sharded clusters with configurable replication, distributed tables, and S3-tiered storage.
• TiDB: MySQL-compatible HTAP clusters deployed via the TiDB Operator, with TiKV row storage for OLTP and TiFlash columnar storage for analytics.
• Trino: Distributed SQL query engine with federation support for querying across data sources.
• Presto: Distributed SQL query engine for in-memory analytics.
• OpenSearch: AWS OpenSearch domains for search and analytics.
• Apache Spark: EMR-based Spark clusters for analytics workloads.

AWS Integration

• EC2 Provisioning: Automated provisioning with configurable instance types
• EBS Storage: Optional EBS volumes for persistent storage
• S3 Backup: Automatic backup of configurations and state to S3
• IAM Integration: Managed IAM policies for secure operations

Kubernetes (K3s)

• Lightweight K3s: Automatic K3s cluster deployment across all nodes
• kubectl/k9s: Pre-configured access with SOCKS5 proxy support
• Private Registry: HTTPS Docker registry for custom images
• Jib Integration: Push custom containers directly from Gradle

Monitoring and Observability

• Mimir: Metrics storage, written to S3 so metrics outlive the cluster
• Loki: Log storage, written to S3 so logs outlive the cluster
• Tempo: Distributed trace storage with trace-to-metrics correlation
• Pyroscope: Continuous profiling of database and system processes
• Grafana: Pre-configured dashboards for Cassandra, ClickHouse, and system metrics
• OpenTelemetry: Distributed tracing and metrics collection
• eBPF Observability: Beyla for L7 network metrics and ebpf_exporter for low-level kernel metrics
• AxonOps: Optional integration with AxonOps for Cassandra monitoring and management

Developer Experience

• Shell Aliases: Convenient shortcuts for cluster management (c0, c-all, c-status, etc.)
• Server: Integration with Claude Code for AI-assisted operations
• Restore Support: Recover cluster state from VPC ID or S3 backup
• SOCKS5 Proxy: Secure access to private cluster resources

Stress Testing

• cassandra-easy-stress: Native integration with Apache stress testing tool
• Sysbench: OLTP benchmarks against any MySQL- or PostgreSQL-compatible kit, with live results in Grafana
• Kubernetes Jobs: Run stress tests as K8s jobs for scalability
• Artifact Collection: Automatic collection of metrics and diagnostics

Installation

Prerequisites

System Requirements

AWS Requirements

• AWS Account: A dedicated AWS account is recommended for lab environments
• AWS Credentials: Either static access key & secret, or a named profile — including an AWS SSO (IAM Identity Center) profile
• IAM Permissions: Permissions to create EC2, IAM, S3, and optionally EMR resources

Optional

• AxonOps Account: For free Cassandra monitoring. Create an account at axonops.com

Install from Release

Download a tarball from the releases page and add the bin directory to your $PATH:

export PATH="$PATH:/path/to/easy-db-lab/bin"

Build from Source

git clone https://github.com/rustyrazorblade/easy-db-lab.git
cd easy-db-lab
./gradlew assemble

The built distribution will be in build/distributions/.

Next Steps

Run the interactive setup to configure your profile:

easy-db-lab profile setup

See the Setup Guide for detailed instructions.

Setup

This guide walks you through the initial setup of easy-db-lab, including AWS credentials configuration, IAM policies, and AMI creation.

Overview

The profile setup command handles all initial configuration interactively. It will:

1. Collect your email and AWS credentials
2. Validate your AWS access
3. Create necessary AWS resources (key pair, IAM roles, Packer VPC)
4. Build or validate the required AMI

Prerequisites

Before running setup:

• AWS Account: An AWS account with appropriate permissions (see IAM Policies below)
• Java 21+: Required to run easy-db-lab
• Docker: Required only if building custom AMIs

Step 1: Run Setup Profile

Run the interactive setup:

easy-db-lab profile setup

The setup wizard will prompt you for:

profile setup validates your credentials against AWS immediately (and then provisions resources), so your credentials must be usable before you run it. For static keys this is automatic; for an SSO profile, run aws sso login first (see below).

What Gets Created

During setup, the following AWS resources are created:

• EC2 Key Pair: For SSH access to instances
• IAM Role: For instance permissions (EasyDBLabEC2Role), including a minimal inline Session Manager policy so every node can be reached over SSM Session Manager
• Packer VPC: Infrastructure for building AMIs
• AMI (if needed): Takes 10-15 minutes to build

Configuration Location

Your profile is saved to:

~/.easy-db-lab/profiles/default/settings.yaml

Using AWS SSO (IAM Identity Center)

If your AWS access is provisioned through AWS SSO (IAM Identity Center) rather than long-lived access keys, easy-db-lab authenticates through a named AWS profile backed by an SSO session. You do not copy temporary credentials out of the AWS access portal — the tool resolves them automatically from your SSO login.

1. Define an SSO profile in ~/.aws/config:

[sso-session my-sso]
sso_start_url = https://my-company.awsapps.com/start
sso_region = us-east-1
sso_registration_scopes = sso:account:access

[profile edl]
sso_session = my-sso
sso_account_id = 123456789012
sso_role_name = YourRoleName
region = us-west-2

2. Log in to start an SSO session (this opens a browser):

aws sso login --profile edl

3. Run profile setup and enter edl when prompted for the AWS profile name (leave the access key and secret blank):

easy-db-lab profile setup

Day-to-day usage

After the one-time setup, you do not sign in for every command. An aws sso login session lasts for hours (your organization sets the exact window), and easy-db-lab resolves and refreshes credentials from it automatically:

# Once per work session (e.g. each morning), or whenever the session expires:
aws sso login --profile edl

# Then run commands freely — no per-command authentication:
easy-db-lab up
easy-db-lab cassandra start
easy-db-lab down --auto-approve

When the session expires, the next command fails with an authentication error directing you to log in again. Re-run aws sso login --profile edl and continue.

Step 2: Getting IAM Policies

If you need to request permissions from your AWS administrator, use the show-iam-policies command to display the required policies with your account ID populated:

easy-db-lab show-iam-policies

This displays five policies:

Filter by Policy Name

To show a specific policy:

easy-db-lab show-iam-policies ec2    # Show EC2 policy only
easy-db-lab show-iam-policies iam    # Show IAM policy only
easy-db-lab show-iam-policies emr    # Show EMR policy only
easy-db-lab show-iam-policies compactor    # Show the compactor policy only

Recommended IAM Setup

For teams with multiple users, we recommend creating managed policies attached to an IAM group:

1. Create an IAM group (e.g., "EasyDBLabUsers")
2. Create the managed policies from the JSON output
3. Attach all policies to the group
4. Add users to the group

Step 3: Build Custom AMI (Optional)

If setup couldn't find a valid AMI for your architecture, or if you want to customize the base image, build one manually:

easy-db-lab build-image

Build Options

Examples

# Build AMD64 AMI (default)
easy-db-lab build-image

# Build ARM64 AMI for Graviton instances
easy-db-lab build-image --arch ARM64

# Build in specific region
easy-db-lab build-image --region eu-west-1

If your profile's SSH transport is ssm, AMI builds reach the build instance over SSM Session Manager too. The first such build also builds a local Packer image with the Session Manager plugin added (see AMI builds).

Environment Variables

Verify Installation

After setup completes, verify by running:

easy-db-lab

You should see the help output with available commands.

Next Steps

Once setup is complete, follow the Tutorial to create your first cluster.

Tutorial: Getting Started

This tutorial walks you through creating a database cluster from scratch, covering initialization, infrastructure provisioning, and database configuration. The examples below use Cassandra, but the same infrastructure supports ClickHouse, OpenSearch, and Spark.

Part 1: Initialize Your Cluster

The init command creates local configuration files for your cluster. It does not provision AWS resources yet.

easy-db-lab init my-cluster

This creates a 3-node Cassandra cluster by default.

Init Options

The database and application node groups use a namespaced --db.* / --app.* scheme. Every older flag still works as an alias with its established default; when both a namespaced option and its legacy alias are given for the same setting, the namespaced option wins, regardless of order.

Examples

Basic 3-node cluster:

easy-db-lab init my-cluster

5-node cluster with 2 stress nodes:

easy-db-lab init my-cluster --db 5 --stress 2

Production-like cluster with EBS storage:

easy-db-lab init prod-test --db 5 --ebs.type gp3 --ebs.size 500 --ebs.iops 3000

ARM64 cluster for Graviton instances: The architecture is derived automatically from the instance type — no flag needed.

easy-db-lab init my-cluster --db.instance-type r7g.2xlarge

Mixed-architecture cluster (arm64 database, x86_64 application):

easy-db-lab init my-cluster --db.instance-type r7g.2xlarge --app.instance-type c6id.2xlarge

Initialize and provision in one step:

easy-db-lab init my-cluster --up

Storage Requirements

Database instances need a data disk separate from the root volume. This can come from either:

• Instance store (local NVMe) — Instance types with a d suffix (e.g., i3.xlarge, m5d.xlarge, c5d.2xlarge) include local NVMe storage and require no extra configuration.
• EBS volumes — Attach an EBS volume using --ebs.type for any other instance type.

If the selected instance type has no instance store and --ebs.type is not specified, up will fail. For example:

easy-db-lab init my-cluster --db.instance-type c5.2xlarge --ebs.type gp3 --ebs.size 200

Part 2: Launch Infrastructure

Once initialized, provision the AWS infrastructure:

easy-db-lab up

This command creates:

• S3 Storage: Cluster data stored under a dedicated prefix in the account S3 bucket
• VPC: With subnets and security groups
• EC2 Instances: Cassandra nodes, stress nodes, and a control node (an m5d.xlarge; K3s data is stored on its NVMe drive to avoid filling the root volume)
• K3s Cluster: Lightweight Kubernetes across all nodes

What Happens During up

1. Configures account S3 bucket with cluster prefix
2. Creates VPC with public subnets in your availability zones
3. Provisions EC2 instances in parallel
4. Waits for SSH availability
5. Configures K3s cluster on all nodes
6. Writes SSH config and environment files

Up Options

Environment Setup

After up completes, source the environment file:

source env.sh

This configures your shell with:

• SSH shortcuts: ssh db0, ssh db1, ssh stress0, etc.
• Cluster aliases: c0, c-all, c-status
• SOCKS proxy configuration

See Shell Aliases for all available shortcuts.

Part 3: Configure Cassandra 5.0

With infrastructure running, configure and start Cassandra.

Step 1: Select Cassandra Version

easy-db-lab cassandra use 5.0

This command:

• Sets the active Cassandra version on all nodes
• Downloads configuration files to your local directory
• Applies any existing patch configuration

cassandra use accepts any installed version. Run easy-db-lab cassandra list to see them. This includes a build published by cassandra build. See Custom Builds, or run easy-db-lab help cassandra-builds.

Step 2: Customize Configuration (Optional)

Edit cassandra.patch.yaml to customize settings:

# Example: Change token count
vim cassandra.patch.yaml

Common customizations:

Step 3: Apply Configuration

easy-db-lab cassandra update-config

This uploads and applies the patch to all Cassandra nodes.

To apply and restart Cassandra in one command:

easy-db-lab cassandra update-config --restart

Step 4: Start Cassandra

easy-db-lab cassandra start

Step 5: Verify Cluster

Check cluster status:

ssh db0 nodetool status

Or use the shell alias (after sourcing env.sh):

c-status

You should see all nodes in UN (Up/Normal) state.

Part 4: Working with Your Cluster

SSH Access

After sourcing env.sh:

ssh db0          # First Cassandra node
ssh db1          # Second Cassandra node
ssh stress0      # First stress node (if provisioned)
ssh control0     # Control node

Cassandra Management

# Stop Cassandra on all nodes
easy-db-lab cassandra stop

# Start Cassandra on all nodes
easy-db-lab cassandra start

# Restart Cassandra on all nodes
easy-db-lab cassandra restart

Filter to Specific Hosts

Most commands support the --hosts filter:

# Apply config only to db0 and db1
easy-db-lab cassandra update-config --hosts db0,db1

# Restart only db2
easy-db-lab cassandra restart --hosts db2

Download Configuration Files

To download the current configuration from nodes:

easy-db-lab cassandra download-config

This saves configuration files to a local directory named after the version (e.g., 5.0/).

Part 5: Shut Down

When finished, destroy the cluster infrastructure:

easy-db-lab down

Quick Reference

Next Steps

• Kubernetes Access - Access K3s cluster with kubectl and k9s
• Shell Aliases - All available CLI shortcuts
• ClickHouse - Deploy ClickHouse for analytics
• Spark - Set up Apache Spark via EMR

Legacy Integrations

The following databases have native integrations that predate the Kits system. They are fully functional but will eventually be migrated to kits.

• Cassandra
• OpenSearch
• Spark

Configuring Cassandra

This page covers Cassandra version management and configuration. For a step-by-step walkthrough, see the Tutorial.

Supported Versions

easy-db-lab supports the following Cassandra versions:

A released version downloads once and is then served from your account's S3 download cache. A nightly (5.0-HEAD, 6.0-HEAD, trunk) and any cassandra install --url tarball always download from their origin, so each bake or install gets the current build.

Quick Start

# Select Cassandra 5.0
easy-db-lab cassandra use 5.0

# Generate configuration patch
easy-db-lab cassandra write-config

# Apply configuration and start
easy-db-lab cassandra update-config
easy-db-lab cassandra start

# Verify cluster
ssh db0 nodetool status

Version Management

Select a Version

easy-db-lab cassandra use <version>

Examples:

easy-db-lab cassandra use 5.0       # Latest stable
easy-db-lab cassandra use 4.1       # LTS version
easy-db-lab cassandra use trunk     # Development branch

This command:

1. Sets the active Cassandra version on all nodes
2. Downloads current configuration files locally
3. Applies any existing cassandra.patch.yaml

It fails if the version is not installed on a targeted node rather than leaving the node pointed at something that isn't there. Install it first — see Custom Builds.

Specify Java Version

easy-db-lab cassandra use 5.0 --java 11

This selects the JDK the already-installed build runs under. It does not reinstall or rebuild anything. The AMI has JDK 8, 11, 17, 21 and 25 installed. If the JDK switch fails on a node, the command fails.

List Available Versions

easy-db-lab ls

Versions installed on the node are listed first. A version declared with lazy: true that has not been installed is listed too, marked (declared, not installed).

Configuration

The Patch File

Cassandra configuration uses a patch file approach. The cassandra.patch.yaml file contains only the settings you want to customize, which are merged with the default cassandra.yaml.

Generate a new patch file:

easy-db-lab cassandra write-config

Options:

• -t, --tokens: Number of tokens (default: 4)

Example patch file:

cluster_name: "my-cluster"
num_tokens: 4
concurrent_reads: 64
concurrent_writes: 64
trickle_fsync: true

Apply Configuration

easy-db-lab cassandra update-config

Options:

• --restart, -r: Restart Cassandra after applying
• --hosts: Filter to specific hosts

Apply and restart in one command:

easy-db-lab cassandra update-config --restart

Download Configuration

Download current configuration files from nodes:

easy-db-lab cassandra download-config

Files are saved to a local directory named after the version (e.g., 5.0/).

Starting and Stopping

# Start on all nodes
easy-db-lab cassandra start

# Stop on all nodes
easy-db-lab cassandra stop

# Restart on all nodes
easy-db-lab cassandra restart

# Target specific hosts
easy-db-lab cassandra start --hosts db0,db1

Cassandra Sidecar

The Apache Cassandra Sidecar is automatically installed and started alongside Cassandra. The sidecar provides:

• REST API for Cassandra operations
• S3 import/restore capabilities
• Streaming data operations
• Metrics collection (Prometheus-compatible)

Sidecar Access

The sidecar runs on port 9043 on each Cassandra node:

# Check sidecar health
curl http://<cassandra-node-ip>:9043/api/v1/__health

Sidecar Management

The sidecar runs in K3s as the cassandra-sidecar DaemonSet, with one pod on each db node. Each pod uses the host network, so the sidecar answers on the node's own IP.

cassandra start deploys the sidecar to every db node, and cassandra stop removes it. A stop limited with --hosts leaves the sidecar in place while any other db node still runs the database; the stop that leaves no db node running removes it. cassandra restart restarts the database, then restarts the sidecar pods one at a time.

Use kubectl from the cluster workspace after source env.sh (see Kubernetes):

# See the sidecar pods and the node each one runs on
kubectl get pods -l app.kubernetes.io/name=cassandra-sidecar -o wide

# Read the sidecar logs
kubectl logs -l app.kubernetes.io/name=cassandra-sidecar --tail=100

# Restart the sidecar on every db node
kubectl rollout restart daemonset/cassandra-sidecar

Sidecar Configuration

The configuration is in the cassandra-sidecar-config ConfigMap. When a pod starts, it writes the configuration to /conf/sidecar.yaml in the pod, with the node's IP filled in. Key settings:

• Cassandra connection details
• Data directory paths
• Traffic shaping and throttling
• S3 integration settings

Custom Builds

To run a custom Cassandra build (your own fork, a feature branch, or a prebuilt tarball), you can build it on your own machine and publish it, install it onto a cluster that is already running, or bake it into the AMI so every future cluster has it.

Build your working branch and install it by name

If you are developing Cassandra itself, build the checkout you are already working in and publish it once, rather than making every node build the same branch:

cd ~/dev/cassandra
easy-db-lab cassandra build --java 17
# -> 5.1-CASSANDRA-19000-20260905-a1b2c3d-jdk17

easy-db-lab cassandra install 5.1-CASSANDRA-19000-20260905-a1b2c3d-jdk17
easy-db-lab cassandra use 5.1-CASSANDRA-19000-20260905-a1b2c3d-jdk17

The build is published to your profile's S3 bucket, so it outlives the cluster and installs onto whatever you bring up next. Uncommitted changes are fine — the manifest records the tree as dirty so you can tell later. cassandra list shows everything you have published that is not yet on the node.

See cassandra build for the naming scheme and what the manifest records.

Install onto a running cluster

easy-db-lab cassandra install <version>
easy-db-lab cassandra use <version>

This downloads (or clones and builds) the version on every Cassandra node and records it in each node's version list, so cassandra use works afterward. It never rebuilds the AMI, so nothing is preserved across a down/up cycle — install it again on the new cluster.

Where the install parameters come from:

• A declared entry (the default): whatever you declared in your profile's extras directory, described below. Nothing else to pass.

• CLI options, for a genuinely one-off test with no file to edit:

  easy-db-lab cassandra install my-build \
    --url https://example.com/apache-cassandra-my-build-bin.tar.gz \
    --java 11
  

  Or from a branch, built with ant on each node:

  easy-db-lab cassandra install my-build \
    --url https://github.com/myuser/cassandra.git \
    --branch my-feature-branch \
    --java 11 \
    --ant-flags "-Duse.jdk11=true"
  

  An option you pass overrides the declared entry's value for that field; anything you leave out falls back to the declared entry. --python defaults to 3.11.9.

Useful details:

• --hosts db0,db1 installs on a subset of nodes only.
• Re-installing a version a node already has is a no-op, as long as you ask for the same parameters it was built with. See Re-installing with different options.
• If a node fails, the command exits non-zero and names the node and the reason; nodes that succeeded keep their install.
• A branch build runs a full ant build on every targeted node, which is CPU-heavy and takes several minutes. It is safe while Cassandra is running — the install only writes to /usr/local/cassandra/<version> — but it is not free. Tarball installs are much cheaper.

Re-installing with different options

A version that is already on a node is never rebuilt. If you re-run install for it with options that contradict what the node recorded when it was built — a different --java, --python, or --ant-flags — the command changes nothing, names the node and the fields that disagree, and exits non-zero:

db0: my-build is already installed, built with the parameters this node declares —
java: declared 11, requested 17. Nothing was changed. To change the java version
in use, run: cassandra use my-build --java <version>

This is deliberate: the bits on disk were compiled against Java 11, so silently recording them as a Java 17 build would leave every later cassandra use picking the wrong JDK.

What to do instead depends on what you actually want:

• Run the existing build under a different JDK — no reinstall needed:

  easy-db-lab cassandra use my-build --java 17
  

• Genuinely rebuild with the new options — install it under a different version name (my-build-jdk17), or remove /usr/local/cassandra/<version> on the affected nodes and install again.

Bake into the AMI

You don't edit the repository's cassandra_versions.yaml. Instead, drop one or more YAML files into your profile's extras directory:

~/.easy-db-lab/profiles/<profile>/cassandra_versions/

(The default profile is default.) At build time these are merged with the built-in versions. Each version must be unique across the built-in list and your extras, or the build fails.

1. Add a version entry

Create e.g. ~/.easy-db-lab/profiles/default/cassandra_versions/my-build.yaml.

Build from a git branch (cloned and compiled with ant during the AMI build):

- version: "my-build"
  java: "11"
  python: "3.10.6"
  url: "https://github.com/myuser/cassandra.git"
  branch: "my-feature-branch"
  ant_flags: "-Duse.jdk11=true"   # optional, passed to ant

Or install a prebuilt tarball:

- version: "my-build"
  java: "11"
  python: "3.10.6"
  url: "https://example.com/apache-cassandra-my-build-bin.tar.gz"

An entry with no url/branch downloads the matching official Apache release.

2. Rebuild the Cassandra AMI

easy-db-lab build-cassandra

3. Select the custom build

easy-db-lab cassandra use my-build

Declaring a version without baking it

Add lazy: true to an entry to declare a version without spending AMI build time on it:

- version: "my-build"
  java: "11"
  python: "3.11.9"
  url: "https://github.com/myuser/cassandra.git"
  branch: "my-feature-branch"
  lazy: true

The AMI build skips the download/clone/build for that entry, but the entry still ships in the image. easy-db-lab cassandra list shows it as (declared, not installed), and easy-db-lab cassandra install my-build installs it on a running cluster without you re-specifying any of its fields.

Next Steps

• Tutorial - Complete walkthrough
• Shell Aliases - Convenient shortcuts for Cassandra management

OpenSearch

AWS OpenSearch can be provisioned as a managed domain for full-text search and log analytics.

Commands

Starting OpenSearch

easy-db-lab opensearch start

This creates an AWS-managed OpenSearch domain linked to your cluster's VPC. The domain takes several minutes to provision.

Checking Status

easy-db-lab opensearch status

Stopping OpenSearch

easy-db-lab opensearch stop

This deletes the OpenSearch domain. Data stored in the domain will be lost.

Spark

easy-db-lab supports provisioning Apache Spark clusters via AWS EMR for analytics workloads.

Enabling Spark

There are two ways to enable Spark:

Option 1: During Init (before up)

Enable Spark during cluster initialization with the --spark.enable flag. The EMR cluster will be created automatically when you run up:

easy-db-lab init --spark.enable
easy-db-lab up

Init Spark Configuration Options

Example with Custom Configuration

easy-db-lab init \
  --spark.enable \
  --spark.master.instance.type m5.2xlarge \
  --spark.worker.instance.type m5.4xlarge \
  --spark.worker.instance.count 5

Option 2: After up (standalone spark init)

Add Spark to an existing environment that is already running. This is useful when you forgot to pass --spark.enable during init, or when you decide to add Spark later:

easy-db-lab spark init

Prerequisites: easy-db-lab init and easy-db-lab up must have been run first.

Spark Init Configuration Options

Example with Custom Configuration

easy-db-lab spark init \
  --master.instance.type m5.2xlarge \
  --worker.instance.type m5.4xlarge \
  --worker.instance.count 5

Submitting Spark Jobs

Submit JAR-based Spark applications to your EMR cluster:

easy-db-lab spark submit \
  --jar /path/to/your-app.jar \
  --main-class com.example.YourMainClass \
  --conf spark.easydblab.keyspace=my_keyspace \
  --conf spark.easydblab.table=my_table \
  --wait

Submit Options

When --jar is a local path, it is automatically uploaded to the cluster's S3 bucket before submission. When it is an s3:// URI, it is used directly.

When a job submitted with --wait fails, the command waits up to 7 minutes for EMR to upload the step's final stderr. EMR uploads a running step's logs every few minutes and once more after the step ends, so an earlier copy can end before the exception. The command then shows the last lines of the final copy. If the final copy does not arrive in time, the command prints the S3 path where it will appear; fetch it later with easy-db-lab spark status --step-id <step-id> --logs.

Using a JAR Already on S3

If your JAR is already on S3 (e.g., from a CI pipeline or a previous upload), pass the S3 URI directly:

easy-db-lab spark submit \
  --jar s3://my-bucket/jars/your-app.jar \
  --main-class com.example.YourMainClass \
  --conf spark.easydblab.keyspace=my_keyspace \
  --wait

This skips the upload step entirely, which is useful for large JARs or when resubmitting the same job.

Cancelling a Job

Cancel a running or pending Spark job without terminating the cluster:

easy-db-lab spark stop

Without --step-id, this cancels the most recent job. To cancel a specific job:

easy-db-lab spark stop --step-id <step-id>

The cancellation uses EMR's TERMINATE_PROCESS strategy (SIGKILL). The API is asynchronous — use spark status to confirm the job has been cancelled.

Checking Job Status

View Recent Jobs

List recent Spark jobs on the cluster:

easy-db-lab spark jobs

Options:

• --limit - Maximum number of jobs to display (default: 10)

Check Specific Job Status

easy-db-lab spark status --step-id <step-id>

Without --step-id, shows the status of the most recent job.

Options:

• --step-id - EMR step ID to check
• --verbose - Show detailed step information (equivalent to aws emr describe-step)
• --logs - Download the step's stderr.gz from the EMR log path in S3, save it under logs/<cluster-id>/<step-id>/, and print it. If EMR has not uploaded the log yet, the command says so and prints the S3 path where it will appear.

Retrieving Logs

Query a Spark job's logs from Loki, scoped to the current cluster:

easy-db-lab spark logs --step-id <step-id>

Without --step-id, shows the logs of the most recent job. The step's name is its job's name, and the job's lines are the ones whose service_name is spark-<job-name>; if a job name was run more than once within --since, every run's lines are returned.

• --step-id - EMR step whose job's logs to show
• --limit, -n - Maximum number of lines (default: 100)
• --since - Time range, e.g. 1h, 30m, 1d (default: 1d)

Every line from an EMR node also carries the label source="emr", so easy-db-lab logs query --source emr shows all of them.

The collector on each EMR node also sends the log files that EMR and YARN write on the node:

These lines have source="emr", the node's node_role (spark-master or spark-worker), its host_name and the cluster label, and the file path in log_file_path. A driver that fails before its Java agent sends anything still leaves its exception in the step's stderr:

easy-db-lab logs query --source emr --grep NoSuchMethodError

Architecture

When Spark is enabled, easy-db-lab provisions:

• EMR Cluster: Managed Spark cluster with master and worker nodes
• S3 Integration: Logs stored at s3://<bucket>/spark/emr-logs/
• IAM Roles: Service and job flow roles for EMR operations
• Observability: Each EMR node runs an OTel Collector (host metrics, the node's EMR and YARN log files, OTLP forwarding), OTel Java Agent (auto-instrumentation for logs/metrics/traces), and Pyroscope Java Agent (continuous CPU/allocation/lock profiling). All telemetry flows to the control node's observability stack.

Timeouts and Polling

• Job Polling Interval: 5 seconds
• Maximum Wait Time: 4 hours
• Cluster Creation Timeout: 30 minutes

Spark with Cassandra

A common use case is running Spark jobs that read from or write to Cassandra. Use the Spark Cassandra Connector:

import com.datastax.spark.connector._

val df = spark.read
  .format("org.apache.spark.sql.cassandra")
  .options(Map("table" -> "my_table", "keyspace" -> "my_keyspace"))
  .load()

Ensure your JAR includes the Spark Cassandra Connector dependency and configure the Cassandra host in your Spark application.

Bulk Write Implementations

easy-db-lab provides three different implementations for bulk writing data to Cassandra, each with different characteristics. All three use the same configuration properties (spark.easydblab.*) so you can easily compare performance by just swapping the JAR and main class.

Implementation Comparison

Direct Bulk Writer (Sidecar Transport)

Streams SSTables directly from Spark to Cassandra nodes via the Sidecar REST API on port 9043.

How it works:

1. Spark generates SSTables from source data
2. SSTables are streamed directly to Sidecar endpoints
3. Sidecar validates and imports SSTables into Cassandra

When to use:

• Direct network connectivity between Spark and Cassandra
• Lower latency requirements
• Smaller to medium datasets

Limitations:

• Requires network connectivity from EMR to Cassandra on port 9043
• Streaming backpressure if Sidecar can't keep up

S3 Bulk Writer (S3 Staging Transport)

Stages SSTables in S3, then notifies Cassandra Sidecar to download and import them.

How it works:

1. Spark generates SSTables and bundles them into ZIPs with manifests
2. Bundles are uploaded to S3 bucket (provided by easy-db-lab)
3. Spark pushes import notification to Sidecar REST API
4. Sidecar downloads bundles from S3, validates checksums, filters by token ranges
5. Sidecar imports SSTables into Cassandra

When to use:

• Large-scale bulk loads (terabytes)
• S3 provides durability and staging for retries
• Multi-DC Cassandra clusters

Prerequisites:

• S3 bucket (automatically provisioned by easy-db-lab as clusterState.dataBucket)
• EMR instance profile with S3 write permissions (automatically configured)
• Cassandra Sidecar running on port 9043
• For Cassandra 5.x: storage_compatibility_mode: NONE in cassandra.yaml (the bulk writer requires the new SSTable format)

Credentials:

• EMR uses instance profile (IMDS) to write to S3 - no manual credential configuration needed
• AWS region is auto-detected from EC2 metadata
• Sidecar uses its IAM role to read from S3

Benefits:

• S3 provides durability for large datasets
• Token range filtering ensures data goes to correct nodes
• Checksum validation guarantees data integrity

Standard Connector Writer

Uses the DataStax Spark Cassandra Connector to write data via CQL.

How it works:

1. Spark generates rows as DataFrames
2. Connector batches writes and sends via Cassandra native protocol (port 9042)
3. Cassandra processes writes through normal write path (memtables → SSTables)

When to use:

• Smaller datasets
• Need CDC, triggers, or other write-time features
• Existing Spark Cassandra Connector pipelines

Limitations:

• Significantly slower than bulk writers for large datasets (goes through full write path)
• Compaction overhead after writes complete
• More network round-trips

Spark Modules

The Spark job modules live in the spark-examples repository and share unified configuration via spark.easydblab.* properties. You can compare performance across implementations by swapping the JAR and main class while keeping the same --conf flags.

Module Overview

The Gradle paths above are within the spark-examples build, not this repo.

Getting the job JARs

The job modules live in their own repository, spark-examples, which publishes the shadow (fat) JARs as GitHub Release assets. Download the one you need and pass it to spark submit with --jar:

# Download a published job jar (check the releases page for the latest version)
curl -L -O https://github.com/rustyrazorblade/spark-examples/releases/download/v0.1.0/bulk-writer-sidecar.jar

Available jars: bulk-writer-sidecar.jar, bulk-writer-s3-iam-all.jar, connector-writer.jar, connector-read-write.jar. Releases: https://github.com/rustyrazorblade/spark-examples/releases

Usage

All modules use the same --conf properties for easy comparison.

Direct Bulk Writer (Sidecar)

Streams SSTables directly to Cassandra Sidecar endpoints:

easy-db-lab spark submit \
  --jar bulk-writer-sidecar.jar \
  --main-class com.rustyrazorblade.easydblab.spark.DirectBulkWriter \
  --conf spark.easydblab.contactPoints=host1:9043,host2:9043,host3:9043 \
  --conf spark.easydblab.keyspace=bulk_test \
  --conf spark.easydblab.localDc=us-west-2 \
  --conf spark.easydblab.rowCount=1000000 \
  --wait

Note: Contact points should include the Sidecar port (9043).

IAM S3 Bulk Writer

Stages SSTables in S3 using IAM instance profile credentials, then imports via Sidecar. Cluster topology is auto-discovered from the Cassandra driver — no manual DC configuration needed:

easy-db-lab spark submit \
  --jar bulk-writer-s3-iam-all.jar \
  --main-class com.rustyrazorblade.easydblab.spark.IamBulkWriter \
  --conf spark.easydblab.contactPoints=host1,host2,host3 \
  --conf spark.easydblab.localDc=us-west-2 \
  --conf spark.easydblab.s3.bucket=my-bucket \
  --conf spark.easydblab.rowCount=10000000 \
  --conf spark.easydblab.parallelism=20 \
  --wait

Credentials: Both the EMR executor and the Cassandra Sidecar authenticate independently via their attached IAM roles. No credentials are extracted or transmitted.

Standard Connector Writer

Standard CQL writes via Cassandra native protocol:

easy-db-lab spark submit \
  --jar connector-writer.jar \
  --main-class com.rustyrazorblade.easydblab.spark.StandardConnectorWriter \
  --conf spark.easydblab.contactPoints=host1,host2,host3 \
  --conf spark.easydblab.keyspace=bulk_test \
  --conf spark.easydblab.localDc=us-west-2 \
  --conf spark.easydblab.rowCount=1000000 \
  --wait

Note: Contact points for the connector use the Cassandra native protocol port (9042), not Sidecar.

Convenience Script

The bin/spark-bulk-write script handles JAR lookup, host resolution, and health checks:

# From a cluster directory
spark-bulk-write direct --rows 10000
spark-bulk-write s3 --rows 1000000 --parallelism 20
spark-bulk-write connector --keyspace myks --table mytable

Configuration Properties

All modules share these properties via spark.easydblab.*:

Table Schema

The test data generators produce this schema:

CREATE TABLE <keyspace>.<table> (
    partition_id bigint,
    sequence_id bigint,
    course blob,
    marks bigint,
    PRIMARY KEY ((partition_id), sequence_id)
);

Troubleshooting

S3 Bulk Writer Issues

"Required property not set: spark.easydblab.s3.bucket"

The S3 bulk writer requires an S3 bucket for staging SSTables. Get the bucket from cluster state:

jq -r '.dataBucket' state.json

Then add it to your submit command:

--conf spark.easydblab.s3.bucket=<bucket-from-state>

"Failed to resolve AWS credentials"

The EMR instance profile should provide credentials automatically via IMDS. If this fails:

1. Verify EMR cluster has EasyDBLabEMREC2Role instance profile attached
2. Check IAM role has s3:* permissions on the data bucket
3. Verify instance metadata service (IMDS) is accessible from EMR nodes

"Unable to detect AWS region"

Region is auto-detected from EC2 metadata. This should work automatically on EMR. If it fails, the EMR cluster may not have proper metadata access.

"S3 bucket name must be between 3 and 63 characters"

Bucket names must follow AWS S3 naming rules (3-63 characters, lowercase, DNS-compliant). Verify the bucket name in state.json.

Job succeeds but no data imported

Check Cassandra 5.x compatibility:

1. Verify storage_compatibility_mode: NONE in cassandra.yaml
2. Cassandra 5.x defaults to UPGRADING mode which uses legacy SSTable format
3. The bulk writer requires NONE to use the new SSTable format

To fix:

echo "storage_compatibility_mode: NONE" >> cassandra.patch.yaml
easy-db-lab cassandra update-config
easy-db-lab cassandra restart

Bundles uploaded but Sidecar didn't import

1. Check Sidecar is running: curl http://<cassandra-host>:9043/api/v1/health
2. Verify Sidecar has S3 read permissions (IAM role)
3. Check Sidecar logs for download or validation errors
4. Verify token ranges match between SSTables and Cassandra ring

Direct Bulk Writer Issues

"Connection refused" to port 9043

1. Verify Cassandra Sidecar is running on all nodes
2. Check security groups allow EMR → Cassandra on port 9043
3. Ensure contact points use correct IP addresses (private IPs if in same VPC)

Slow writes / backpressure

The direct transport streams data and can be throttled if Sidecar can't keep up. Consider:

1. Reduce spark.easydblab.parallelism to lower write rate
2. Use S3 staging transport for large datasets
3. Check Cassandra disk I/O and compaction status

Connector Writer Issues

"No route to host" or timeout

1. Check security groups allow EMR → Cassandra on port 9042
2. Verify contact points are reachable from EMR
3. Ensure Cassandra native protocol is enabled

Slow performance

The connector uses the standard write path (not bulk write). For large datasets, use a bulk writer instead.

Kits

A kit is a self-contained package of configuration and scripts that installs, starts, stops, and optionally backs up a workload on your cluster. Each kit defines its full lifecycle in a kit.yaml file using typed steps — no Kubernetes YAML wrangling required.

easy-db-lab ships with built-in kits (ClickHouse, Presto, Trino, TiDB, memcached, Neo4j, sysbench, clickhouse-benchmark). You can also create your own kits for any workload you want to benchmark or test.

Discovering kits

List all available kits:

easy-db-lab kit list

Inspect a kit before installing it — see its args, endpoints, and available commands:

easy-db-lab kit info clickhouse

Run from a cluster workspace, kit info lists each endpoint at its node's private IP. Before the cluster has hosts of that node type, it lists the bare NodePort (:30123).

Installing a kit

easy-db-lab kit install clickhouse --clickhouse-version 25.4 --size 100Gi

Args vary by kit. Run kit info <name> to see what a kit accepts, or pass --help:

easy-db-lab kit install clickhouse --help

After install, the kit's files are written into a subdirectory of the cluster workspace. The kit's lifecycle commands are registered automatically.

Bench kits — benchmarking a database

Bench kits are a special class of kit that run against an already-running database kit. They require a --target flag pointing at the installed database kit you want to benchmark.

# Install sysbench targeting your running TiDB instance
easy-db-lab kit install sysbench --target tidb

# Run the prepare, start, and stop lifecycle as usual
easy-db-lab sysbench-tidb prepare
easy-db-lab sysbench-tidb start
easy-db-lab sysbench-tidb stop

The kit is installed into a directory named <bench-kit>-<target> (e.g. sysbench-tidb). This lets you run the same bench kit against multiple databases simultaneously and compare results:

easy-db-lab kit install sysbench --target tidb
easy-db-lab kit install sysbench --target my-custom-db

# Both run at the same time — compare results in Grafana
easy-db-lab sysbench-tidb start
easy-db-lab sysbench-my-custom-db start

The target must expose a wire protocol endpoint the bench tool can speak — sysbench supports MySQL and PostgreSQL. See Sysbench for the full lifecycle, flags, and metrics.

TARGET_* environment variables

When a bench kit starts, easy-db-lab reads the target database's endpoint configuration and injects it as environment variables into every phase script:

Which variables are populated depends on what endpoints the target kit declares. A kit that supports both JDBC and PostgreSQL wire protocol will populate both sets.

Running kit commands

Every installed kit gains a set of subcommands:

easy-db-lab clickhouse start       # deploy and start the workload
easy-db-lab clickhouse status      # show running state and connection endpoints
easy-db-lab clickhouse stop        # stop and remove the workload
easy-db-lab clickhouse backup --name my-backup   # back up data
easy-db-lab clickhouse restore --name my-backup  # restore from backup
easy-db-lab clickhouse uninstall   # stop and remove all kit resources

memcached

The memcached kit runs one memcached pod on a db node. By default it runs from RAM only and creates no persistent volumes; --extstore-size adds extstore on the db node's NVMe (see extstore below).

Install it, optionally setting the cache size in megabytes with --memory (default 1024, passed to memcached as -m):

easy-db-lab kit install memcached --memory 4096

Start it:

easy-db-lab memcached start

memcached is published on NodePort 31211, so it is reachable on any node's private IP. memcached start, memcached status and kit info memcached all print the endpoint resolved to each db node's private IP:

easy-db-lab kit info memcached
# Endpoints:
#   memcached  db  <db node private IP>:31211  native

Connect from a pod in the cluster, or from your machine over Tailscale:

printf 'set greeting 0 0 5\r\nhello\r\nget greeting\r\nquit\r\n' | nc <db node private IP> 31211

A memcached-exporter sidecar serves Prometheus metrics on port 9150. The collector finds the pod by label and scrapes it, so the series land in Mimir under job="memcached". The kit's METRICS.md lists them.

memcached extstore

extstore lets memcached keep item values on flash once RAM is full. It is off unless you pass --extstore-size, which sets the size of the extstore file (a whole number with an M, G or T suffix):

easy-db-lab kit install memcached --memory 4096 --extstore-size 100G

With extstore on, kit install creates a local PersistentVolume on the db node's NVMe, and the memcached container mounts it at /data and runs with -o ext_path=/data/extstore:<size>.

These install options tune extstore. Each one is passed to memcached only when you set it, so memcached's own default applies otherwise. Without --extstore-size they are ignored.

easy-db-lab kit install memcached --extstore-size 100G --extstore-threads 8 --extstore-item-size 512

memcached uninstall also deletes the extstore volume claim, the PersistentVolume, and its directory on the node.

Stopping and uninstalling

Stop it, or uninstall it to also remove the kit directory. stop deletes the Deployment, its ReplicaSet and pods, the Service and the ConfigMap labelled easydblab/kit=memcached; uninstall deletes those plus the extstore volume claim, then the extstore PersistentVolume:

easy-db-lab memcached stop
easy-db-lab memcached uninstall

Installing memcached a second time fails with a collision error and exits non-zero, leaving the running kit untouched. Pass --force to overwrite the scaffold. Running memcached start while memcached is already running also fails with a collision error; run memcached stop first. stop returns once the memcached pod is gone, so start can follow it straight away.

Installing a custom kit

Place your kit directory under the profile kits folder and it will appear in kit list and be installable by name like any built-in kit:

~/.easy-db-lab/profiles/default/kits/<kit-name>/

Custom kits in the profile directory take precedence over built-in kits with the same name.

mkdir -p ~/.easy-db-lab/profiles/default/kits/my-kit
cp -r /path/to/my-kit/* ~/.easy-db-lab/profiles/default/kits/my-kit/

# Now it appears in kit list and can be installed by name:
easy-db-lab kit install my-kit

Using kits from external projects

If you keep kit definitions alongside a private project (a POC, internal tooling, etc.), you can register that project's kits directory without copying files into your profile.

A typical project structure looks like this:

myapp/
├── src/
├── kits/
│   └── myapp-workload/
│       ├── kit.yaml
│       └── bin/
│           ├── start.sh
│           └── stop.sh
└── README.md

Clone your project and register the kits directory by name:

git clone https://github.com/myorg/myapp ~/myapp
easy-db-lab kit source add myapp ~/myapp/kits

The kits it contains now appear in kit list and can be installed by name:

easy-db-lab kit list
easy-db-lab kit install myapp-workload

Registered sources are persisted in ~/.easy-db-lab/profiles/<profile>/kit-sources.yaml and survive CLI restarts. When you kit install a kit from an external source, its files are copied into the cluster workspace exactly like any other kit — the installed kit is self-contained.

Managing registered sources

# List all registered sources (shows name and path, flags missing paths)
easy-db-lab kit source list

Output looks like:

Registered kit sources:
  myapp  /Users/jon/myapp/kits

If a registered path no longer exists on disk, [missing] appears next to it so you know which sources need attention.

# Remove a source by name
easy-db-lab kit source remove myproject

Updating a path (upsert behavior): Sources are identified by name. If you move or reclone a project to a different location, just re-add the source with the new path — no need to remove the old registration first:

# If you move or reclone the project, just update the path — no need to remove first
easy-db-lab kit source add myapp ~/new-location/myapp/kits
# Updated kit source 'myapp': /new-location/myapp/kits

Resolution priority

When multiple sources provide a kit with the same name, the first match wins:

1. Profile kits directory (~/.easy-db-lab/profiles/<profile>/kits/)
2. Registered additional sources (in registration order)
3. Built-in kits

For a full walkthrough of building and publishing your own kit, see the Kit Development guide.

ClickHouse

easy-db-lab supports deploying ClickHouse clusters on Kubernetes for analytics workloads alongside your database cluster.

Overview

The clickhouse kit deploys ClickHouse with the Altinity ClickHouse operator. The operator runs the servers from a ClickHouseInstallation (CHI) named clickhouse and the coordination service from a ClickHouseKeeperInstallation (CHK) named clickhouse-keeper. Both run only on db nodes.

Quick Start

Create a 3-node cluster and deploy ClickHouse as one shard with 3 replicas:

# Initialize and start a 3-node cluster
easy-db-lab init my-cluster --db 3 --up

# Install the kit: the operator, and the manifests rendered with your settings
easy-db-lab kit install clickhouse

# Deploy ClickHouse (1 shard x 3 replicas, one per db node)
easy-db-lab clickhouse start

Configuring ClickHouse

All ClickHouse settings are options of kit install clickhouse. They are applied when the kit is installed: the manifests in the workspace's clickhouse/ directory are rendered with them, and clickhouse start deploys those rendered manifests.

# Pin the server version and give each replica a 100Gi volume
easy-db-lab kit install clickhouse --version 25.4 --size 100Gi

# Larger S3 cache, no caching on write
easy-db-lab kit install clickhouse --s3-cache 50Gi --s3-cache-on-write false

To change a setting after installing, run kit install clickhouse --force with the new options, then clickhouse stop and clickhouse start.

Starting ClickHouse

easy-db-lab clickhouse start

start creates the Local PersistentVolumes on the db nodes, applies the ClickHouseKeeperInstallation and waits for the Keeper pods to be Ready (up to 300 seconds), then applies the ClickHouseInstallation and waits for its status to reach Completed (up to 900 seconds) and for every server pod to be Ready (up to 300 seconds). Finally it creates the clickhouse-nodeport service. The phase takes no options of its own.

Example with Custom Settings

# 6 db nodes, run 3 replicas
easy-db-lab kit install clickhouse --replicas 3
easy-db-lab clickhouse start

Cluster Topology

ClickHouse is deployed by the Altinity ClickHouse operator as a ClickHouseInstallation named clickhouse, with one shard and --replicas replicas (default: one per db node). The ClickHouse cluster is also named clickhouse, which is the name to use in ON CLUSTER DDL.

Pod Names

The operator names each server pod chi-clickhouse-clickhouse-<shard>-<replica>-0, with shard and replica counted from 0. A 3-replica deployment runs:

List them with:

kubectl get pods -l clickhouse.altinity.com/chi=clickhouse -o wide

Pod-to-Node Placement

Server pods run only on db nodes, each on a Local PersistentVolume. A replica is placed on a db node when it first starts, and its volume keeps it on that node across pod restarts, so data stays local and does not move. Which db node a given replica lands on is chosen by the scheduler, not by its replica number; use kubectl get pods -o wide (above) to see the mapping.

Shell Helpers

After source env.sh, two helpers connect to ClickHouse without looking up pod names or IPs:

# Interactive clickhouse-client in the first running server pod; extra args are passed through
clickhouse-client
clickhouse-client --query "SELECT version()"

# Send one query over HTTP to db0 (NodePort 30123)
clickhouse-query "SELECT 1"
clickhouse-query <<< "SELECT count() FROM system.tables"

Checking Status

easy-db-lab clickhouse status

This prints whether ClickHouse is running (with the count of Ready server pods) and the connection endpoints on each db node: HTTP and JDBC (30123), Native (30900), MySQL wire (30904) and PostgreSQL wire (30905).

Accessing ClickHouse

After deployment, ClickHouse is accessible on NodePorts of every db node. The default user has no password.

The MySQL and PostgreSQL interfaces are protocol-compatible, not dialect-compatible: queries sent over them are parsed as ClickHouse SQL. They are handy for connecting standard clients and drivers, but tools that emit MySQL- or PostgreSQL-specific DDL will not work unmodified.

The kit also adds a sql command that runs a statement over JDBC as the default user:

easy-db-lab clickhouse sql "SELECT version()"
easy-db-lab clickhouse sql --file schema.sql

Creating Tables

The kit deploys one shard, so every replica holds all of the data. Use ReplicatedMergeTree to keep the replicas in sync through ClickHouse Keeper, and create tables ON CLUSTER clickhouse so the DDL runs on every replica:

CREATE TABLE events ON CLUSTER clickhouse (
    id UInt64,
    timestamp DateTime,
    event_type String,
    data String
) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/events', '{replica}')
ORDER BY (timestamp, id);

Key points:

• clickhouse is the cluster name defined in the ClickHouseInstallation
• {shard} and {replica} are macros the operator sets on each server
• An insert on any replica is replicated to the others
• A query on any replica sees the whole table

Distributed Tables

A Distributed table routes queries and inserts across shards. With one shard it adds nothing over querying the replicated table directly, but the pattern works unchanged:

CREATE TABLE events_dist ON CLUSTER clickhouse AS events
ENGINE = Distributed(clickhouse, default, events, rand());

INSERT INTO events_dist VALUES (1, now(), 'click', '{"page": "/home"}');
SELECT count(*) FROM events_dist WHERE event_type = 'click';

Table Engine Comparison

Storage Policies

ClickHouse is configured with three storage policies. You select the policy when creating a table using the SETTINGS storage_policy clause.

Policy Comparison

Local Storage (default)

The default policy stores data on the local disk of the db node, in the replica's Local PersistentVolume. This provides the best performance for latency-sensitive workloads.

CREATE TABLE my_table (...)
ENGINE = MergeTree()
ORDER BY id
SETTINGS storage_policy = 'default';

If you omit the storage_policy setting, tables use this policy.

When to use local storage:

• Performance benchmarking where latency matters
• Temporary or experimental datasets
• Workloads with predictable data sizes that fit on local disks
• When you don't need data to persist after cluster teardown

S3 Storage (s3_main)

The S3 policy stores data in the cluster's S3 data bucket (under clickhouse/), which up creates, with a local cache for frequently accessed data. The cache size defaults to 10Gi and is set with kit install clickhouse --s3-cache. Caching on write is enabled by default (--s3-cache-on-write true), so data written is cached and subsequent reads can be served from cache immediately. This is ideal for large datasets where storage cost matters more than latency. The servers reach S3 with the instance profile credentials; no keys are configured.

CREATE TABLE my_table (...)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/default/my_table', '{replica}')
ORDER BY id
SETTINGS storage_policy = 's3_main';

When to use S3 storage:

• Large analytical datasets (terabytes+)
• Datasets larger than the local disks
• Cost-sensitive workloads where storage cost > compute cost

down sets no expiry on the data bucket and deletes nothing in it, so data in it outlives the cluster until you delete it. Use Backup and Restore to keep a dataset in a place that is not tied to the cluster.

How the cache works:

• Hot (frequently accessed) data is cached locally for fast reads
• Cold data is fetched from S3 on demand
• Cache is automatically managed by ClickHouse
• First query on cold data will be slower; subsequent queries use cache

S3 Tiered Storage (s3_tier)

The S3 tiered policy moves data from local disk to S3 based on disk space. It has a hot volume on the local disk and a cold volume on the cached S3 disk: data is written locally and moved to S3 when local free space runs low, giving local performance for hot data and S3 capacity for cold data.

Configure the tiering behavior when installing the kit:

# Move data to S3 when local disk free space falls below 20% (default)
easy-db-lab kit install clickhouse --s3-tier-move-factor 0.2

# More aggressive tiering - move when free space < 50%
easy-db-lab kit install clickhouse --s3-tier-move-factor 0.5

Then create tables with S3 tiered storage:

CREATE TABLE my_table (...)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/default/my_table', '{replica}')
ORDER BY id
SETTINGS storage_policy = 's3_tier';

When to use S3 tiered storage:

• Workloads with mixed hot/cold data access patterns
• Growing datasets that may outgrow local disk capacity
• Want automatic cost optimization without manual intervention
• Need local performance for recent data with S3 capacity for historical data

How automatic tiering works:

• New data is written to local disks first (fast writes)
• When local disk free space falls below the configured threshold (default: 20%), ClickHouse automatically moves data to S3
• Data on S3 is still queryable but with higher latency
• The local cache (configured with --s3-cache) helps performance for frequently accessed S3 data
• Manual moves are also possible: ALTER TABLE my_table MOVE PARTITION tuple() TO DISK 's3'

Backup and Restore

The kit provides backup and restore commands. Backups are stored in your account-level S3 bucket, outside the per-cluster prefix, so they survive cluster teardown and can be restored into a new cluster.

ClickHouse's native BACKUP/RESTORE SQL is used. The backup destination is a named disk (s3_backup) configured in the ClickHouseInstallation, which points to:

s3://<account-bucket>/clickhouse-backups/<backup-name>/

The S3 disk uses IAM instance profile credentials — no AWS keys are stored anywhere in the cluster configuration. The required S3 permissions on the account-level bucket are handled automatically by the easy-db-lab IAM setup — no manual configuration is needed for standard clusters.

Creating a Backup

easy-db-lab clickhouse backup --name <backup-name>

Without --name, the backup is named backup-<yyyyMMdd-HHmmss>. The command fails if a backup with that name already exists in S3, and prints the aws s3 rm command that deletes it. Otherwise it runs, in the first ClickHouse server pod:

BACKUP DATABASE default TO Disk('s3_backup', '<backup-name>/');

Restoring a Backup

easy-db-lab clickhouse restore --name <backup-name>

In the first ClickHouse server pod, this first drops stale replica entries in Keeper for replicated tables that no longer exist locally (left behind by a DROP TABLE without SYNC, they make the restore fail with REPLICA_ALREADY_EXISTS), then runs:

RESTORE DATABASE default FROM Disk('s3_backup', '<backup-name>/');

Note: The restore command does not drop existing tables first. If tables with conflicting names exist, the restore will fail. Drop or truncate the conflicting tables before restoring.

Common Workflows

Snapshot before a destructive operation:

# Take a named snapshot before running a migration
easy-db-lab clickhouse backup --name pre-migration-snapshot

# Run the migration
# ...

# If something goes wrong, restore
easy-db-lab clickhouse restore --name pre-migration-snapshot

Persist data across cluster rebuilds:

# Back up before tearing down
easy-db-lab clickhouse backup --name my-dataset
easy-db-lab down

# In a new cluster workspace: create the cluster, install, start, and restore
easy-db-lab init my-cluster --db 3 --up
easy-db-lab kit install clickhouse --size 100Gi
easy-db-lab clickhouse start
easy-db-lab clickhouse restore --name my-dataset

Stopping ClickHouse

easy-db-lab clickhouse stop

This deletes the ClickHouseInstallation clickhouse (and with it the server pods) and the clickhouse-nodeport service. Keeper, the operator, and the PersistentVolumes stay in place, so clickhouse start redeploys onto the same volumes.

To remove everything the kit created:

easy-db-lab clickhouse uninstall

This deletes the ClickHouseKeeperInstallation clickhouse-keeper, the kit's PersistentVolumes, and the clickhouse-operator Helm release.

Monitoring

ClickHouse metrics are automatically integrated with the observability stack. On clickhouse start the kit registers two Prometheus scrape jobs. Both use pod discovery, so each pod is scraped once, by the collector on its own node, with instance set to the pod name:

start also installs the kit's Grafana dashboards, one for metrics and one for logs, into a Grafana folder named clickhouse.

Architecture

The ClickHouse deployment includes:

• Altinity ClickHouse operator: Helm release clickhouse-operator in kube-system, installed by kit install clickhouse
• ClickHouse Server: ClickHouseInstallation clickhouse, one cluster named clickhouse with 1 shard and --replicas replicas, on db nodes only
• ClickHouse Keeper: ClickHouseKeeperInstallation clickhouse-keeper with --replicas replicas on db nodes, used for replication and ON CLUSTER DDL (ZooKeeper-compatible); the CHI references it by name
• Service: clickhouse-nodeport, a NodePort service in front of the server pods
• Local PersistentVolumes: One PV per db node for data locality

Storage Architecture

ClickHouse uses Local PersistentVolumes to guarantee pod-to-node pinning:

1. During cluster creation, each db node is labeled with its ordinal (easydblab.com/node-ordinal=0, etc.)
2. Local PVs are created with node affinity matching these ordinals and the type=db label
3. Each PV is labelled app.kubernetes.io/name=clickhouse, and the claims select that label, so they never bind another kit's PV
4. The installation's volumeClaimTemplate requests --size of storage from these PVs

Once a replica's claim binds a PV, that replica always runs on the PV's db node, providing:

• Data locality (no network storage overhead)
• No data movement when pods restart

The binding is made when the replica first starts, so replica numbers do not map to node numbers; see Pod-to-Node Placement.

Ports

Clients connect to the NodePorts on any db node. The container ports are what the pods listen on inside the cluster; the metrics scrapes use them directly.

Install Kafka

The kit install kafka command sets up Apache Kafka in KRaft mode (no ZooKeeper) using the Strimzi operator. Installation provisions local persistent volumes on db nodes and installs the Strimzi operator via Helm. easy-db-lab kafka start then applies the Kafka custom resource and waits for the cluster to become Ready; easy-db-lab kafka stop tears it back down.

Prerequisites

• Cluster is up (easy-db-lab up)
• At least one db node is provisioned
• Environment is sourced: source env.sh

Quick Start

easy-db-lab kit install kafka
easy-db-lab kafka start

Install Flags

Managing Kafka

start

Deploys the Kafka cluster and waits for it to become Ready:

1. Applies the JMX metrics ConfigMap
2. Applies the Kafka and KafkaNodePool custom resources
3. Waits for the Kafka cluster to report Ready (up to 300s)
4. Applies NodePort services for the kafka-exporter and JMX exporter

Grafana dashboards in kafka/dashboards/ are installed automatically after a successful start.

stop

Removes the Kafka and KafkaNodePool resources and NodePort services. Persistent volumes are retained — topic data survives a stop/start cycle.

uninstall

Removes all Kafka resources including PVCs and PVs, then uninstalls the Strimzi operator.

Bootstrap Addresses

Internal (in-cluster pods, e.g. a Cassandra sidecar or another kit):

kafka.default.svc.cluster.local:9092

External (from outside the cluster via Tailscale):

<control-node-private-ip>:32100

The external address is printed in the kafka/README.md generated at install time.

Benchmarking

Producer performance test

easy-db-lab kafka producer-perf

Consumer performance test

easy-db-lab kafka consumer-perf

Topic management

easy-db-lab kafka create-topic --topic my-topic --partitions 3 --replication-factor 3

Metrics

Two Prometheus scrape jobs are registered automatically by kafka start:

Both are scraped by pod discovery on the container port: each pod once, by the collector on its own node, with instance set to the pod name. The NodePorts expose the same endpoints for manual inspection.

Metric names are lowercase. See kafka/METRICS.md for the full catalog.

The Kafka Overview Grafana dashboard is installed automatically and shows broker health, throughput, consumer lag, and topic health panels.

CDC: Cassandra → Kafka

When using Kafka as a CDC target for Cassandra, configure the sidecar connector with the internal bootstrap address to avoid NodePort overhead:

kafka.default.svc.cluster.local:9092

Replication and Multi-Broker Setup

The default install uses a single combined broker/controller node. For replication testing, install with multiple brokers and set the replication factor when creating topics:

easy-db-lab kit install kafka --brokers 3
easy-db-lab kafka start
easy-db-lab kafka create-topic --topic my-topic --partitions 3 --replication-factor 3

--brokers must not exceed the number of db nodes in the cluster.

Cleaning Up

To remove Kafka and free disk space:

easy-db-lab kafka stop
easy-db-lab kit uninstall kafka

kit uninstall deletes the Kafka resources, PVCs/PVs, and the Strimzi operator.

Apache Ignite 3

easy-db-lab supports deploying Apache Ignite 3 clusters on Kubernetes for distributed SQL and in-memory computing workloads.

Overview

Apache Ignite 3 is a distributed database with ACID transactions, distributed SQL, and a pluggable storage engine. It runs as a StatefulSet on K3s with configurable storage profiles ranging from pure in-memory to fully disk-backed persistence.

Metrics are automatically pushed to the cluster's OTel Collector via OTLP and appear in Grafana.

Quick Start

# Initialize and start a 3-node cluster
easy-db-lab init my-cluster --db 3 --up

# Install and start Ignite 3 with default settings
easy-db-lab kit install ignite3
easy-db-lab ignite3 start

Configuration Options

Storage Profiles

Ignite 3 supports three storage engines, selectable at start time:

# Start with pure in-memory storage (fastest, no persistence)
easy-db-lab ignite3 start --storage aimem

# Start with disk-based storage for large datasets
easy-db-lab ignite3 start --storage rocksdb

SQL Queries

Run SQL directly against the cluster using the thin client JDBC driver:

# Execute a SQL statement
easy-db-lab ignite3 sql "CREATE TABLE t1 (id INT PRIMARY KEY, val VARCHAR)"
easy-db-lab ignite3 sql "INSERT INTO t1 VALUES (1, 'hello')"
easy-db-lab ignite3 sql "SELECT * FROM t1"

# Execute SQL from a file
easy-db-lab ignite3 sql --file query.sql

Lifecycle

# Start the cluster
easy-db-lab ignite3 start

# Stop (preserves PVCs and data for aipersist/rocksdb profiles)
easy-db-lab ignite3 stop

# Start again — existing data is available immediately
easy-db-lab ignite3 start

# Remove all resources including data
easy-db-lab kit uninstall ignite3

Endpoints

Metrics

Ignite 3 metrics are pushed to the cluster's OTel Collector via OTLP automatically at start time. No additional configuration is needed. Metrics appear in Mimir and are accessible from Grafana.

Neo4j

The neo4j kit runs Neo4j Community Edition as a single-replica StatefulSet on a db node. Its data directory is on a local persistent volume that kit install provisions. Authentication is off (NEO4J_AUTH=none), so clients connect without credentials.

Quick Start

easy-db-lab init my-cluster --db 3 --up

easy-db-lab kit install neo4j
easy-db-lab neo4j start

Configuration Options

easy-db-lab kit install neo4j --version 5.26.0

start refuses unsupported versions before it applies anything:

• 4.x and earlier use different configuration keys from 5.x and the calendar-versioned releases.
• 5.0 through 5.5 are refused too; 5.6.0 is the first supported 5.x release. The kit loads the Java agent through NEO4J_server_jvm_additional. Images before 5.6.0 replace the stock server.jvm.additional lines in neo4j.conf with that value instead of appending to them, so Neo4j would start without its stock JVM flags.

Endpoints

Neo4j is published on two NodePorts, reachable on any node's private IP:

neo4j start, neo4j status and kit info neo4j print both endpoints resolved to each db node's private IP: <db node private IP>:30687 for Bolt and http://<db node private IP>:30474 for HTTP.

The server advertises <first db node private IP>:30687 as its Bolt address, so a neo4j:// client that fetches the routing table gets an address it can dial, not a pod IP.

Connecting

Over Bolt, with cypher-shell or any Neo4j driver:

cypher-shell -a bolt://<db node private IP>:30687 "RETURN 1"

Over HTTP, with the transactional Cypher endpoint:

curl -s -H 'Content-Type: application/json' \
  -d '{"statements":[{"statement":"RETURN 1"}]}' \
  http://<db node private IP>:30474/db/neo4j/tx/commit

Neo4j Browser is at http://<db node private IP>:30474/.

Metrics

The OpenTelemetry Java agent from the base AMI is mounted into the pod and loaded by the Neo4j JVM. It pushes metrics over OTLP to the collector on the pod's own node every 5 seconds. The series land in Mimir under job="neo4j".

Lifecycle

# Stop: deletes the StatefulSet, Services, pods, and ConfigMap labelled easydblab/kit=neo4j.
# The data volume is kept.
easy-db-lab neo4j stop

# Start again with the existing data
easy-db-lab neo4j start

# Uninstall: deletes those plus the data volume claim, then the PersistentVolume and its
# directory on the db node
easy-db-lab neo4j uninstall

Installing Neo4j a second time fails with a collision error and exits non-zero, leaving the running kit untouched. Pass --force to overwrite the scaffold. Running neo4j start while Neo4j is already running also fails with a collision error; run neo4j stop first. stop returns once the Neo4j pod is gone, so start can follow it straight away.

Install Presto

The install presto command scaffolds a Presto deployment using the Presto Helm chart. It generates scripts and a values.yaml in a local presto/ directory. Once installed, use easy-db-lab presto start and easy-db-lab presto stop to manage it.

Presto is stateless — it does not use persistent volumes. Queries run in-memory on app (type=app) nodes.

Prerequisites

• Cluster is up (easy-db-lab up)
• App nodes are provisioned (at least one ServerType.Stress node)
• kubectl and helm are available in your PATH (or run via the easy-db-lab container)
• Environment is sourced: source env.sh

Quick Start

easy-db-lab kit install presto
easy-db-lab presto start

Flags

What Gets Generated

presto/
├── README.md                    # Usage instructions for this cluster
├── values.yaml                  # Helm values for the Presto chart
├── catalogs/
│   ├── cassandra.properties     # Cassandra connector config
│   └── clickhouse.properties    # ClickHouse JDBC connector config
└── bin/
    ├── start.sh                 # Deploy sequence
    ├── stop.sh                  # Teardown sequence
    └── update-catalogs.sh       # Re-renders Presto catalog config from running kits

Managing Presto

After installation, use the CLI to start and stop:

# Deploy Presto
easy-db-lab presto start

# Tear down Presto
easy-db-lab presto stop

start

helm upgrade --install presto prestodb/presto --version 0.4.0 -f values.yaml
# update-catalogs.sh runs automatically after helm install
kubectl wait --for=condition=Ready pods -l app=presto,component=coordinator --timeout=180s

No platform create-pvs step — Presto is stateless.

stop

helm uninstall presto

Automatic Catalog Management

Presto has built-in support for two catalogs that wire up automatically:

• Cassandra — uses the Cassandra connector, pointing at the cluster's Cassandra nodes
• ClickHouse — uses the ClickHouse JDBC connector, pointing at the cluster's ClickHouse nodes

When either of those kits starts or stops, Presto detects the change via a post-workload-start / post-workload-stop hook and re-renders its catalog configuration automatically. No restart of Presto is needed.

# Start Cassandra — Presto wires up the cassandra catalog automatically
easy-db-lab cassandra start

# Start ClickHouse — Presto wires up the clickhouse catalog automatically
easy-db-lab kit install clickhouse
easy-db-lab clickhouse start

You do not need to restart Presto or manually update any config when adding or removing either kit.

Node Placement

Presto workers are scheduled on type=app nodes using nodeSelector: type: app. The coordinator runs on the control plane or app nodes depending on cluster size.

Connecting

After easy-db-lab presto start completes, the Presto coordinator is accessible within the cluster. Use kubectl port-forward or the SOCKS5 proxy to connect from your workstation:

kubectl port-forward svc/presto 8080:8080

Then connect with any Presto-compatible client at localhost:8080.

Adding Presto Catalogs for Custom Kits

If you install a custom kit via --from and want Presto to connect to it, drop a presto-catalog.properties file in the kit's installed directory:

my-kit/
├── bin/
│   ├── start.sh
│   └── stop.sh
└── presto-catalog.properties    # Presto picks this up automatically

The file follows the standard Presto connector properties format:

connector.name=jdbc
connection-url=jdbc:postgresql://localhost:5432/mydb
connection-user=presto
connection-password=

update-catalogs.sh scans all sibling kit directories for this file. When found, the catalog is added to Presto's configuration under a name matching the kit directory name. No restart of Presto is required — the script runs helm upgrade with the merged catalog values.

Install Trino

The install trino command scaffolds a Trino deployment using the Trino Helm chart. It generates scripts and a values.yaml in a local trino/ directory. Once installed, use easy-db-lab trino start and easy-db-lab trino stop to manage it.

Trino is stateless — it does not use persistent volumes. Queries run in-memory on app (type=app) nodes.

Prerequisites

• Cluster is up (easy-db-lab up)
• App nodes are provisioned (at least one ServerType.Stress node)
• kubectl and helm are available on the control node
• Environment is sourced: source env.sh

Quick Start

easy-db-lab kit install trino
easy-db-lab trino start

Flags

What Gets Generated

trino/
├── README.md                    # Usage instructions for this cluster
├── values.yaml                  # Helm values for the Trino chart
├── catalogs/
│   ├── cassandra.properties     # Cassandra connector config
│   └── clickhouse.properties    # ClickHouse JDBC connector config
└── bin/
    ├── start.sh                 # Deploy sequence
    ├── stop.sh                  # Teardown sequence
    ├── uninstall.sh             # Remove Helm release
    └── update-catalogs.sh       # Re-renders Trino catalog config from running kits

Managing Trino

After installation, use the CLI to start and stop:

# Deploy Trino
easy-db-lab trino start

# Stop Trino (scale to zero, no data loss)
easy-db-lab trino stop

Automatic Catalog Management

Trino has built-in support for two catalogs that wire up automatically:

• Cassandra — uses the Cassandra connector, pointing at the cluster's Cassandra nodes
• ClickHouse — uses the ClickHouse JDBC connector, pointing at the cluster's ClickHouse nodes

When either of those kits starts or stops, Trino detects the change via a post-workload-start / post-workload-stop hook and re-renders its catalog configuration automatically. No restart of Trino is needed.

# Start Cassandra — Trino wires up the cassandra catalog automatically
easy-db-lab cassandra start

# Start ClickHouse — Trino wires up the clickhouse catalog automatically
easy-db-lab kit install clickhouse
easy-db-lab clickhouse start

Node Placement

Trino workers are scheduled on type=app nodes using nodeSelector: type: app.

Connecting

After easy-db-lab trino start completes, the Trino coordinator is accessible within the cluster. Connect using the SOCKS5 proxy or a port-forward:

kubectl port-forward svc/trino 8080:8080

Then connect with any Trino-compatible client at jdbc:trino://localhost:8080.

You can also use the built-in SQL command:

easy-db-lab trino sql "SELECT count(*) FROM cassandra.mykeyspace.mytable"

Adding Trino Catalogs for Custom Kits

If you install a custom kit via --from and want Trino to connect to it, drop a trino-catalog.properties file in the kit's installed directory:

my-kit/
├── bin/
│   ├── start.sh
│   └── stop.sh
└── trino-catalog.properties    # Trino picks this up automatically

The file follows the standard Trino connector properties format:

connector.name=postgresql
connection-url=jdbc:postgresql://localhost:5432/mydb
connection-user=trino
connection-password=

update-catalogs.sh scans all sibling kit directories for this file. When found, the catalog is added to Trino's configuration under a name matching the kit directory name. No restart of Trino is required — the script runs helm upgrade with the merged catalog values.

Presto vs Trino

Trino is the open-source fork of Presto (previously known as PrestoSQL). Both are supported as independent kits. Choose Trino for the active open-source community and Presto for compatibility with Meta's Presto ecosystem.

TiDB

The tidb kit deploys a TiDB HTAP cluster using the TiDB Operator. TiDB combines a MySQL-compatible SQL layer with two storage engines: TiKV (row store, for OLTP) and TiFlash (columnar store, for analytics) — letting you run transactional and analytical queries against the same data.

Prerequisites

• Cluster is up (easy-db-lab up)
• At least 1 db node (runs TiKV and TiFlash)
• At least 1 app node (runs TiDB and PD)

Quick Start

easy-db-lab kit install tidb
easy-db-lab tidb start
easy-db-lab tidb sql "SELECT tidb_version()"

tidb start deploys the TiDB Operator-managed cluster and waits for each component (PD, TiKV, TiDB, TiFlash) to become Ready. TiFlash takes the longest — expect a few minutes on first start while images pull.

Flags

Cluster Layout

Connecting

TiDB speaks the MySQL wire protocol, exposed as NodePort 30400 on every cluster node:

mysql -h <node-ip> -P 30400 -u root

Or use the built-in SQL command, which resolves the endpoint for you:

easy-db-lab tidb sql "SHOW DATABASES"

Using TiFlash

Tables are not automatically replicated to TiFlash. Enable replication per table:

ALTER TABLE my_table SET TIFLASH REPLICA 1;

Once the replica is in place, TiDB's optimizer routes analytical queries to TiFlash automatically. To force it for a specific query:

SELECT /*+ read_from_storage(tiflash[my_table]) */ count(*) FROM my_table;

Lifecycle

easy-db-lab tidb start       # deploy the TiDB cluster
easy-db-lab tidb status      # show running state and endpoints
easy-db-lab tidb stop        # tear down the TiDB cluster
easy-db-lab tidb uninstall   # stop the cluster if running, delete its PVCs, remove the TiDB Operator

uninstall of a running TiDB kit runs tidb stop first. It still refuses to run when a TidbCluster exists that the kit does not know is running — run tidb stop first.

Monitoring

The kit registers four Prometheus scrape jobs with the cluster's observability stack automatically — no configuration needed:

Metrics are available in Grafana and Mimir as soon as the kit starts.

TiDB also exports traces to Tempo (via the OTel Collector's Jaeger receiver, since TiDB v8.x has no native OTLP support). Search for them in Grafana with service.name=TiDB — the tag is case-sensitive.

Benchmarking

TiDB declares the sql capability, so bench kits can target it directly. See Sysbench:

easy-db-lab kit install sysbench --target tidb

Sysbench

The sysbench kit runs sysbench OLTP benchmarks against a running database kit. It is a bench kit: it does not deploy a database itself, but targets one you've already started, connecting over the MySQL or PostgreSQL wire protocol. Benchmark pods run inside the Kubernetes cluster, and per-interval results are pushed to Mimir so you can watch throughput and latency live in Grafana.

Prerequisites

• Cluster is up (easy-db-lab up) and has an app node (init --app.count 1 or more). Every sysbench pod (the run, prepare, createdb and cleanup pods) runs on an app node (type=app), so it does not compete with the database it measures. With no app node, prepare, start and stop fail before they create a pod, with sysbench needs an app node (type=app); this cluster has none. A stop that fails this way leaves the kit marked as running.
• A database kit with the sql capability is installed and running (e.g. TiDB)
• The target kit exposes a MySQL or PostgreSQL wire protocol endpoint — sysbench connects over the wire protocol, not JDBC

Quick Start

# Start a database to benchmark
easy-db-lab kit install tidb
easy-db-lab tidb start

# Install sysbench pointed at it
easy-db-lab kit install sysbench --target tidb

# Load data, run the benchmark, clean up
easy-db-lab sysbench-tidb prepare
easy-db-lab sysbench-tidb start
easy-db-lab sysbench-tidb stop

The kit installs as sysbench-<target> — the instance name and the CLI subcommand both include the target, so multiple sysbench instances can run against different databases at the same time. See Bench kits for how cross-kit targeting works.

Flags

--target is baked in at install time — to point sysbench at a different database, install another instance. Every other flag is passed per invocation, so you can vary them run to run without reinstalling:

easy-db-lab sysbench-tidb start --threads 32 --duration 300

Lifecycle

prepare

easy-db-lab sysbench-tidb prepare

Creates the sbtest database on the target if it doesn't exist, then loads the test tables (--tables tables with --scale thousand rows each). Run this once before the first benchmark run.

start

easy-db-lab sysbench-tidb start

Runs the benchmark for --duration seconds, streaming sysbench's interval output to your terminal. Each 10-second interval report (TPS, QPS, p99 latency, errors/s) is also pushed to Mimir.

When the run finishes, the final sysbench summary (SQL statistics, throughput, latency percentiles, and errors) is written to last-run.txt in the kit's workspace directory (e.g. sysbench-tidb/last-run.txt), prefixed with the run's parameters and its exit code. Each run overwrites the file, a failed run too, so a run's numbers survive after the terminal output scrolls away and the file never shows an older run.

When the run pod ends, start reads its exit code. When sysbench exits non-zero (for example after FATAL: event queue is full), start says the run failed, gives the exit code, names last-run.txt (which then holds the end of the run's log when sysbench printed no summary) and exits non-zero, and does not print Run summary written to .... A run pod that cannot pull its image fails start at once, with the reason and the pod's events; one that stays Pending fails it after 10 minutes.

Only one run per instance can be in progress. start fails with a collision error while a run pod for the same instance (e.g. sysbench-tidb) is still pending or running; a finished run does not block the next one. Instances pointed at different targets run side by side.

stop

easy-db-lab sysbench-tidb stop

Kills any running benchmark pod and runs sysbench cleanup, dropping the test tables from the target database. If the target database is already stopped, stop still finishes and removes its pods; it prints why the cleanup did not finish (the cleanup pod's log, or its events if it never started) and says the test tables were left.

Rate limiting and overload testing

By default (--rate=0) sysbench is thread-bound: each of the --threads worker threads issues transactions as fast as the target will answer them, so throughput settles at whatever the database can sustain. Setting --rate to a non-zero value switches sysbench to a fixed target rate — it generates events on a schedule of that many transactions per second and hands them to the worker threads, regardless of how fast the target is actually responding.

That distinction matters when the requested rate exceeds what the target can sustain. The generated events queue up faster than the workers can drain them, sysbench's internal event queue fills, and the run hard-aborts with:

FATAL: event queue is full

The abort is fast — under 15 seconds into the run in the case that prompted this section, regardless of the --duration you asked for. A --rate set well above capacity does not produce a sustained high-latency window; it produces a run that dies almost immediately with no useful results.

An aborted run is easy to miss after the fact, because it does not look like a failure downstream. last-run.txt holds only the seeded parameter header with no SQL statistics block, and the run's p50/p95/p99 series on the Grafana dashboard flatline at 0 — which reads as a suspiciously excellent result rather than a crash. If latency drops to zero and the summary is truncated, check the pod output for the FATAL line.

For overload and latency testing, drive the target past its limit with concurrency instead of with a target rate: leave --rate=0 and raise --threads until latency climbs. A thread-bound run applies backpressure naturally — slower responses mean fewer transactions issued — so it degrades into a high-latency window instead of overflowing the event queue. It is not immune to aborting for other reasons: sysbench still exits on unhandled SQL errors, which a heavily overloaded target is more likely to return. If you do want a fixed rate, first measure the target's sustainable throughput with a thread-bound run, then set --rate at or just above that measured number rather than far above it.

Comparing Databases

Because each install is a separate named instance, you can benchmark several databases simultaneously and compare them side by side in Grafana:

easy-db-lab kit install sysbench --target tidb
easy-db-lab kit install sysbench --target my-custom-db

easy-db-lab sysbench-tidb prepare && easy-db-lab sysbench-tidb start
easy-db-lab sysbench-my-custom-db prepare && easy-db-lab sysbench-my-custom-db start

Of the built-in kits, TiDB and ClickHouse expose MySQL and PostgreSQL wire endpoints. Any custom kit that declares a mysql or postgresql endpoint and the sql capability in its kit.yaml can be targeted the same way.

Note that ClickHouse's wire interfaces parse queries as ClickHouse SQL, and sysbench's built-in oltp_* workloads issue MySQL-specific DDL during prepare — running them against ClickHouse unmodified will fail at table creation. Benchmarking ClickHouse with sysbench requires a custom Lua workload with ClickHouse-compatible schemas.

Metrics & Dashboard

The kit ships a Sysbench Benchmark Grafana dashboard, installed automatically. During a run, these metrics are pushed through the OTel collector to Mimir, labelled by instance (kit):

The kit label carries the instance name (e.g. sysbench-tidb), so runs against different targets plot as separate series on the same panel.

sysbench's interval reports only emit the single configured percentile (p99), so p50/p95 cannot be sampled per interval. The kit runs sysbench with --histogram and parses the final latency histogram to derive whole-run p50/p95/p99, pushed once when the run completes.

clickhouse-benchmark

The clickhouse-benchmark kit runs ClickHouse's own clickhouse-benchmark tool against a running ClickHouse kit, over the ClickHouse native protocol. It is a bench kit: it deploys no database, but targets the clickhouse kit you have already started. The ClickBench dataset and its 43 queries are built in, and you can run your own query file instead.

The benchmark runs in a pod on an app node, so it does not compete with the database it measures. Every 10 seconds the kit pushes the tool's figures to Mimir, so you can watch throughput and latency live in Grafana.

Prerequisites

• The cluster is up (easy-db-lab up) and has an app node (init --app.count 1 or more). Every pod of this kit runs on an app node (type=app). With no app node, prepare and start fail before they create a pod, with clickhouse-benchmark needs an app node (type=app); this cluster has none.
• The clickhouse kit is installed and started.

Quick Start

easy-db-lab kit install clickhouse
easy-db-lab clickhouse start

easy-db-lab kit install clickhouse-benchmark --target clickhouse

easy-db-lab clickhouse-benchmark-clickhouse prepare
easy-db-lab clickhouse-benchmark-clickhouse start
easy-db-lab clickhouse-benchmark-clickhouse stop

The kit installs as clickhouse-benchmark-<target>. The target must declare a native and an http endpoint; kit install refuses a target that lacks one and names every missing type. See Bench kits for how cross-kit targeting works.

Flags

--target and --version are set at install time. The prepare and start flags apply to one run, so you can change them from run to run without reinstalling.

Lifecycle

prepare

easy-db-lab clickhouse-benchmark-clickhouse prepare
easy-db-lab clickhouse-benchmark-clickhouse prepare --storage-policy s3_main --recreate

prepare creates the ClickBench hits table on every replica (ON CLUSTER clickhouse, as a ReplicatedMergeTree table) with the storage policy you give, and then loads it on the server with INSERT INTO hits SELECT * FROM s3(...). The target has one shard, so every replica holds the whole table. The load runs from a prepare pod with no client-side time limit, so a long load is waited out. When the load is done, prepare waits for every replica to catch up and checks that they all hold the same number of rows. It fails, and prints each replica's count, when they differ.

Before it creates the pod, prepare checks the target over its HTTP endpoint:

• When hits already exists, prepare fails, names the table and its storage policy, and tells you to use --recreate. It does not change the table. With --recreate it drops hits on every replica first.
• While a benchmark run of this instance is in progress, prepare fails and names the run pod. Run stop, or wait for the run to finish.

When a statement fails, prepare shows the server's error and exits non-zero. It never drops hits. When the load itself fails, it also says hits may be partly loaded; run prepare --recreate to reload it.

hits lives on the target's disks. When you stop and start the clickhouse kit, the table is gone (issue 985), so run prepare again before the next start.

Loading from your own S3 files

With no --source, the server reads the public ClickBench files, https://clickhouse-public-datasets.s3.amazonaws.com/hits_compatible/athena_partitioned/hits_{0..99}.parquet, anonymously. A --source you give is read with the cluster's IAM role (use_environment_credentials), so it must be in a bucket that role can read; the role can read the easy-db-lab-* buckets. Recent ClickHouse servers refuse to use the server's own credentials in a query unless the setting s3_allow_server_credentials_in_user_queries is on, so for a --source load prepare turns that setting on for the INSERT on servers that have it. The anonymous default load never uses it. The files must have the ClickBench hits columns. A --source that holds a ' or a \ is refused before any pod is created.

start

easy-db-lab clickhouse-benchmark-clickhouse start
easy-db-lab clickhouse-benchmark-clickhouse start --concurrency 16 --duration 300 --randomize=false

start runs the 43 ClickBench queries against the target's native endpoint, from a pod on an app node, and streams the tool's output to your terminal. With the ClickBench queries it first checks that hits exists and tells you to run prepare if it does not.

--duration and --iterations can both be set; the run stops at the first limit it reaches.

Only one run per instance can be in progress. A second start fails while a run pod of the same instance is pending or running; a finished run does not block the next one.

When the run pod ends, start reads the pod's exit code. When it is not 0, start says the run failed, gives the exit code and exits non-zero. Without --continue-on-errors, the first query that fails ends the run with the server's error code.

Your own queries

easy-db-lab clickhouse-benchmark-clickhouse start --queries my-queries.sql

--queries sends a file from your machine to the run pod, so it works from a Homebrew install. Put one query per line; a query cannot span lines. A relative path is relative to the current directory. start does not check for hits when you give --queries. The file and the run's scripts must fit in one ConfigMap, 1,000,000 bytes.

Settings

easy-db-lab clickhouse-benchmark-clickhouse start --settings max_threads=8,use_query_cache=0

Each key=value entry becomes --key=value on the clickhouse-benchmark command line, so it applies to every query. A value cannot hold a comma. A key must be letters, digits and _, so use max_threads, not max-threads. These keys are refused, because start sets them itself or the kit does not support them: host, port, user, password, database, secure, queries_file, query, concurrency, iterations, timelimit, delay, json, randomize, continue_on_errors, cumulative, reconnect, query_id, confidence, stage, max_consecutive_errors and help.

stop

easy-db-lab clickhouse-benchmark-clickhouse stop

stop deletes this instance's benchmark and prepare pods and their ConfigMaps. It leaves hits in place and does not touch other kits' pods.

Results

When a run ends, start writes last-run.txt into the kit's workspace directory (for example clickhouse-benchmark-clickhouse/last-run.txt). It holds the run's settings, the exit code and the final report: queries executed, QPS, rows and MiB per second, and the latency percentiles. A failed run writes it too. Each run replaces the file.

The results come from the run pod's full log. The kubelet rotates pod logs at about 10 MiB; the final report is printed last, so it is always kept, but early interval reports of a long run can be missing from the log.

Dashboard

The first successful start installs the clickhouse-benchmark dashboard into a Grafana folder for the kit, so the dashboard shows data from the second run on. Its panels show the ten gauges the kit pushes for each 10-second report:

• clickhouse_benchmark_qps, clickhouse_benchmark_rows_per_second and clickhouse_benchmark_mib_per_second: throughput in the interval.
• clickhouse_benchmark_queries_executed: queries run so far in the run.
• clickhouse_benchmark_errors: queries that failed in the interval.
• clickhouse_benchmark_latency_p50_ms, _p90_ms, _p95_ms, _p99_ms and _p999_ms: query latency in milliseconds.

Every gauge carries kit=<instance>, so the instance filter separates benchmark instances. A link goes to the clickhouse dashboard for the same cluster, to see the server side of the same run.

Install PostgreSQL

The postgres kit deploys PostgreSQL on K8s db nodes via the CloudNativePG (CNPG) operator. Data is persisted on PersistentVolumes — stopping and restarting preserves your dataset.

Prerequisites

• Cluster is up (easy-db-lab up) with at least one db node

Quick Start

easy-db-lab kit install postgres
easy-db-lab postgres start

Flags

Managing PostgreSQL

# Start PostgreSQL
easy-db-lab postgres start

# Stop PostgreSQL (data is preserved)
easy-db-lab postgres stop

# Remove operator and delete all PersistentVolumes
easy-db-lab postgres uninstall

start

1. Creates PersistentVolumes on db nodes via platform-pvs
2. Applies the CNPG Cluster custom resource
3. Waits for all pods to reach Ready
4. Applies the NodePort service for external access

stop

Deletes the CNPG Cluster CR and the NodePort service. PersistentVolumes are retained — running postgres start again resumes from the existing dataset.

uninstall

Deletes PersistentVolumes and uninstalls the CNPG operator Helm release. The operator is shared by postgres and every postgres-<extension> instance, so it is kept while any CNPG Cluster is left and removed with the last one.

Connecting

The PostgreSQL primary is exposed as a NodePort on port 30432 of each db node.

# JDBC URL
jdbc:postgresql://<db-node-ip>:30432/postgres

# psql via SOCKS5 proxy or port-forward
kubectl port-forward svc/postgres-nodeport 5432:5432
psql -h localhost -U postgres postgres

The default user is postgres with password postgres (stored in the postgres-credentials K8s Secret).

Running SQL

Use the built-in sql capability to execute queries directly:

easy-db-lab postgres sql "SELECT version()"
easy-db-lab postgres sql --file query.sql

Extensions

PostgreSQL extensions that require custom container images are activated at install time via --extension on kit install postgres. Each extension installs as a separate named instance (e.g. postgres-duckdb) that starts and stops independently.

Listing available extensions

easy-db-lab postgres extensions

Outputs a table of alias names, image templates, shared_preload_libraries, and CREATE EXTENSION statements.

Built-in aliases

Each instance has its own NodePorts, so plain postgres (30432 / 30987) and every extension instance can run at the same time.

Example

# Install with an extension
easy-db-lab kit install postgres --extension duckdb

# Start, stop, and use the named instance
easy-db-lab postgres-duckdb start
easy-db-lab postgres-duckdb sql "SELECT duckdb_version()"
easy-db-lab postgres-duckdb stop

Versioning

Built-in alias images use __PG_MAJOR__ as a placeholder for the PostgreSQL major version (e.g. 17). This is substituted automatically from the --version flag set at install time.

Presto Integration

When both postgres and presto are running, Presto automatically exposes a postgres catalog using the PostgreSQL JDBC connector pointed at the CNPG primary service. No manual configuration is needed.

easy-db-lab kit install postgres
easy-db-lab postgres start

easy-db-lab kit install presto
easy-db-lab presto start

# postgres catalog is available automatically
easy-db-lab presto sql "SHOW CATALOGS"

Platform Substrate

The platform substrate is the set of Kubernetes primitives that easy-db-lab provisions on every cluster. It provides a stable foundation so any kit can be deployed without bespoke manifest code.

Two-Layer Model

┌─────────────────────────────────────┐
│         Kit Layer              │
│  (ClickHouse, Presto, custom, …)    │
│  deployed via install + helm/kubectl│
└────────────────┬────────────────────┘
                 │ uses
┌────────────────▼────────────────────┐
│        Platform Substrate           │
│  StorageClasses · Node Labels · PVs │
│  provisioned at cluster `up` time   │
└─────────────────────────────────────┘

The platform substrate is provisioned automatically when you run easy-db-lab up. You do not need to configure it manually.

StorageClasses

Two StorageClasses are created at up time:

local-storage-wfc is used for StatefulSet kits. Kubernetes must know which node a pod schedules on before binding the volume — WaitForFirstConsumer enforces this ordering.

Node Labels

All cluster nodes are labeled at up time:

Use nodeSelector: type: db (or app) in pod specs to constrain placement. The ordinal label is used by the PV pre-binding mechanism so each StatefulSet replica lands on the right node.

Persistent Volumes

Per-kit PVs are created lazily at install time, not at cluster-up time. Run platform create-pvs before starting a stateful kit:

easy-db-lab platform create-pvs --kit clickhouse --size 100Gi

This creates one PV per db node with:

• Path: /mnt/db1/<kit> on each host
• StorageClass: local-storage-wfc
• Node affinity: type=<node-type> and easydblab.com/node-ordinal=N for deterministic binding. Both pools carry ordinals, so the node type keeps a db PV off the app node with the same ordinal.
• ClaimRef: pre-bound to <volumeClaimTemplateName>-<kit>-N

The command is safe to re-run. If a PV exists with a stale claimRef (the PVC was deleted), the UID is cleared and the PV is returned to Available.

platform Commands

platform create-pvs

easy-db-lab platform create-pvs --kit <name> --size <Gi> [--node-type db|app] [--pvc-name <name>]

Creates one PV per node of the specified type. Defaults to db nodes. For stateless kits that need app-node storage, use --node-type app. --pvc-name sets the volumeClaimTemplate name to pre-bind against (default: data).

platform info

easy-db-lab platform info

Displays StorageClasses, available PV counts per node pool, node selector labels, and the ordinal label key. Use this to verify substrate readiness before deploying a kit.

platform cni

easy-db-lab platform cni

Shows the pod-network datapath. On a Cilium cluster it reads the cilium-config ConfigMap and the CiliumNode objects from the control node and prints the routing mode, IPAM mode, kube-proxy replacement, masquerade interfaces, native routing CIDR, the Hubble UI URL, and per node the ENI count, subnet CIDRs, and IPs allocated, used, and available. On a Flannel cluster it prints one line that names Flannel. Cilium is the default datapath; init --cni=flannel selects Flannel. Read-only. See Pod Networking (CNI).

Custom Templates

The install command can render templates from a custom directory:

easy-db-lab kit install --from ./my-kit/ --kit my-kit --size 50Gi

The directory must contain a kit.yaml. Without it the install is rejected before anything is written to the workspace, because kit.yaml is what marks an installed directory as a kit.

Template Variable Contract

All templates receive these standard variables from cluster state:

Unresolved __VAR__ placeholders emit a warning but do not fail the render.

Template Directory Layout

my-kit/
├── kit.yaml
├── README.md.template
├── values.yaml.template
└── bin/
    ├── start.sh.template
    └── stop.sh.template

Files without .template suffix are copied verbatim.

Profile Templates

Place templates in ~/.easy-db-lab/profiles/<profile>/kits/<name>/ to make them discoverable via kit list. Additional template directories can be registered with kit source add. Resolution priority: profile templates override additional sources, which override built-in templates of the same name.

Port Exposure Model

Kits use standard pod networking (not hostNetwork). Client and metrics ports are surfaced on each EC2 instance's network interface in one of two ways:

• NodePort services — stateful db kits (ClickHouse, TiDB) expose their ports through a NodePort service, remapping native ports into the NodePort range (30000–32767). Example: ClickHouse HTTP 8123 → 30123.
• hostPort patches — helm-based app kits (Presto, Trino) patch hostPort mappings onto the coordinator pod at start time, keeping the native port (e.g. 8080 → 8080).

Why ports must reach the host

The OTel collector DaemonSet runs with hostNetwork: true so it can scrape host processes and kit metrics endpoints, and so host JVMs such as Cassandra can push OTLP to it at localhost:4318. Built-in kits are scraped by pod discovery: each collector scrapes the kit pods on its own node directly, on the pod IP and container metrics port, so a metrics port does not need to reach the host for the collector (see Kit Observability). The NodePorts and hostPorts that still expose metrics ports are there for manual inspection (curl <node-ip>:<port>/metrics). Client ports do need to reach the host, and a NodePort-range port can never collide with a database listening on its native port on the host.

Port Assignments

When adding a new kit, choose ports that do not conflict with any host process or existing kit in the table above. Each kit's ports are declared in its kit.yaml (metrics and endpoints sections).

Kit Observability

Each kit declares its metrics targets in kit.yaml. metrics is a list — kits with multiple components declare one entry per scrape target, each with a unique job name:

metrics:
  - type: scrape     # pod service discovery — each pod scraped directly, per-pod `instance`
    job: tidb-sql
    pod-selector: "app.kubernetes.io/component=tidb,app.kubernetes.io/instance=tidb"
    port: 10080      # container metrics port, not a NodePort
    path: /metrics
  - type: scrape
    job: tikv
    pod-selector: "app.kubernetes.io/component=tikv,app.kubernetes.io/instance=tidb"
    port: 20180
    path: /metrics

If job is omitted, the kit name is used.

Static NodePort vs. pod discovery. By default a scrape target is a static localhost:<port> NodePort — every collector scrapes it and instance is the collector's hostname. When a target sets pod-selector (a comma-separated K8s label selector), the OTel collector instead uses Prometheus pod service discovery (kubernetes_sd_configs, role: pod): each collector scrapes only the matching pods co-located on its own node, and instance becomes the pod name. Use this when a component has multiple pods behind one service and you need per-pod attribution — a NodePort load-balances scrapes across all pods, so a single store/pod cannot be distinguished. TiKV uses this so each of the 3 stores reports under its own instance (e.g. tidb-tikv-0). Every built-in kit uses pod discovery: a static job runs on every collector, so even a single-instance kit behind a NodePort reports one duplicate series per node.

Three modes are supported:

Metrics Registration Lifecycle

When a kit with type: scrape targets starts successfully, easy-db-lab:

1. Creates one ConfigMap easydblab-metrics-<job> per scrape target in the default namespace, labeled easydblab.com/workload-metrics=true and easydblab.com/kit=<kit>, containing the job name, port, and path.
2. Regenerates the OTel collector ConfigMap to include a Prometheus scrape job per target.
3. Applies the updated OTel ConfigMap so the running collector picks it up.

When the kit stops:

1. Deletes all of the kit's metrics ConfigMaps via the easydblab.com/kit label selector.
2. Regenerates and applies the OTel collector ConfigMap without the kit's scrape jobs.

This is fully automatic — no manual OTel configuration is required when starting or stopping kits.

Verifying the Substrate

# Check StorageClasses and PV availability
easy-db-lab platform info

# List all nodes and their labels
kubectl get nodes --show-labels

# List PVs for a kit
kubectl get pv | grep clickhouse

Monitoring

Grafana Dashboards

Grafana is deployed automatically as part of the observability stack (k8 apply). It is accessible on port 3000 of the control node.

Cluster Identification

When running multiple environments side by side, Grafana displays the cluster name in several places to help you identify which environment you're looking at:

• Browser tab - Shows the cluster name instead of "Grafana"
• Dashboard titles - Each dashboard title is prefixed with the cluster name
• Sidebar org name - The organization name in the sidebar shows the cluster name
• Home dashboard - The System Overview dashboard is set as the home page instead of the default Grafana welcome page

Tenant pickers and the current cluster

Every dashboard has a picker for each kind of data it shows: Metrics, Logs and Traces. Each picker lists the cluster's own tenant (the default), every tenant in the shared store by name, and one "all tenants" choice. Change the pickers to read another tenant's tests; every panel, variable, annotation and link on the dashboard follows them, and a link to another dashboard keeps them. Profiles have one datasource only, so there is no profiles picker.

A dashboard with a cluster variable opens on the current cluster, so it shows this cluster's data with no selection by hand. The variable lists every cluster of the tenant the Metrics picker selects. You can select more than one cluster, or "All". With the "all tenants" datasource in each picker and "All" in cluster, a dashboard shows every cluster of every tenant. Every place a dashboard shows a cluster names it by its short name: its name and the first 8 characters of its id, for example test-1a2b3c4d. That covers the cluster pickers, legends, panel titles and tables. The pickers still select the full id, so queries and links read the whole <name>-<id>.

Tests

The Tests dashboard (Infrastructure folder) lists every test (cluster) of the tenant the Metrics picker selects, with its start, end and duration: its first and last up sample within Lookback (default 180 days). The dashboard opens on the last 24 hours; the list and the Test picker still cover the whole Lookback. Torn-down clusters are listed too. Click a cluster for:

• System Overview or Cassandra Overview, opened on that test's window;
• Compare with the current cluster, which opens Cluster Comparison with that test as the baseline and the current cluster as the candidate;
• Show documents, which selects the test and shows its documents below the list.

Comparing two tests

Cluster Comparison, A/B Comparison and System A/B Comparison compare two runs of any lengths: pick them in Baseline and Candidate (any cluster of the tenant within Lookback; both default to the current cluster). Three rows at the top show:

• Overlay: both runs on one time axis, from a common start. The axis covers the longer run.
• Side by side: each run on its own time range, at its real times.
• Summary and documents: each figure over each run's whole window, the difference in percent, and both runs' documents.

A/B Comparison compares builds within one cluster, so its deltas are per cluster. To compare two clusters, use Cluster Comparison.

These rows need the dashboard's relative time range (the default, ending now); an absolute range turns their own time ranges off. In the overlay, annotations sit at their real time, so they line up only with a run that did not move.

Test documents

Attach notes and results to a test as markdown files:

easy-db-lab report upload results.md notes.md

The files go to the test's folder in the account bucket, reports/<tenant>/<name>-<id>/, each with an HTML copy, and one index.html holds them all, each under its own heading. Only .md files named with letters, digits, ., _ and - are accepted, and index.md is refused. Each name can occur only one time in a command. The command names every file it refuses and uploads nothing. If an upload fails, the command names the failed file and each document it already stored; run the command again. A file with the name of a stored document replaces it. The command uses your own AWS credentials and needs only the workspace, so it works after down. up writes the index of a new test, which says "No documents yet".

The Tests dashboard and the comparison dashboards show the documents through a read-only web server in the Grafana pod (port 3080 of the control node). The server answers only GET requests for document paths under /reports/; it answers 400 for a path with other characters, and it drops the query string. Pick the tenant folder with Documents tenant; it defaults to the cluster's own tenant.

Two report upload runs for one test at the same moment can each rebuild the index without the other's document. The document itself is stored; the next upload puts it back in the index.

System Dashboard

Shows CPU, memory, disk I/O, network I/O, and load average for all cluster nodes via OpenTelemetry metrics.

System Overview and System A/B Comparison have a Role picker. It selects hosts by the role in their name:

Pick one or more roles to show every host of those roles, with no host selection by hand. The host pickers list only hosts of the selected roles, and every panel shows only them. The default is All, which shows every host. A link from these dashboards to another dashboard keeps the selected roles.

AWS CloudWatch Overview

A combined dashboard showing S3, EBS, and EC2 metrics via CloudWatch. Available after running easy-db-lab up.

S3 metrics:

• Throughput: BytesDownloaded, BytesUploaded
• Request Counts: GetRequests, PutRequests
• Latency: FirstByteLatency (p99), TotalRequestLatency (p99)

EBS volume metrics:

• IOPS: VolumeReadOps, VolumeWriteOps (mirrored read/write chart)
• Throughput: VolumeReadBytes, VolumeWriteBytes (mirrored read/write chart)
• Queue Length: VolumeQueueLength
• Burst Balance: BurstBalance (percentage)

EC2 status checks:

• Status Check Failures: StatusCheckFailed_Instance, StatusCheckFailed_System (red threshold at >= 1)

Use the dropdowns at the top to select S3 bucket, EC2 instances, and EBS volumes.

How it works:

• S3 request metrics are automatically enabled for the cluster's prefix in the account S3 bucket during easy-db-lab up
• EBS and EC2 metrics are published automatically by AWS for all instances and volumes
• Grafana queries CloudWatch using the EC2 instance's IAM role (no credentials needed)
• During easy-db-lab down, the S3 metrics configuration is automatically removed to stop CloudWatch billing

Note: S3 request metrics take approximately 15 minutes to appear in CloudWatch after being enabled. EBS and EC2 metrics are available immediately.

EMR Overview

Shows Spark/EMR node metrics via OpenTelemetry. Available when an EMR cluster is provisioned. Each EMR node runs an OTel Collector that collects host metrics and receives JVM telemetry from the OTel and Pyroscope Java agents.

Host Metrics:

• CPU Usage: Per-node CPU utilization percentage
• Memory Usage: Used and cached memory per node
• Disk I/O: Read/write throughput per node (mirrored chart)
• Network I/O: Receive/transmit throughput per node (mirrored chart)
• Load Average: 1m and 5m load per node
• Filesystem Usage: Root filesystem utilization percentage

Spark JVM Metrics:

• JVM Heap Memory: Used and committed heap per node/pool
• GC Duration Rate: Garbage collection duration rate per collector
• JVM Threads: Thread count per node
• JVM Classes Loaded: Class count per node

Use the Hostname dropdown to filter by specific EMR nodes.

OpenSearch Overview

Shows OpenSearch domain metrics via CloudWatch. Available when an OpenSearch domain is provisioned.

Metrics displayed:

• Cluster Health: ClusterStatus (green/yellow/red), FreeStorageSpace
• CPU / Memory: CPUUtilization, JVMMemoryPressure
• Search Performance: SearchLatency (p99), SearchRate
• Indexing Performance: IndexingLatency (p99), IndexingRate
• HTTP Responses: 2xx, 3xx, 4xx, 5xx (color-coded)
• Storage: ClusterUsedSpace

Use the Domain dropdown to select which OpenSearch domain to view.

Cassandra Overview

A deep-dive into Cassandra cluster health. Shows:

• Cluster Overview: Request throughput, errors by status, read and write latency spread across nodes, pending compactions, active tasks by pool
• Hardware / Operating System: CPU, load average, memory, disk throughput, network I/O
• Per-Node Latency: Read and write latency per node
• Data Status: Data size, SSTable count, compaction backlog against completion rate, SSTables per read
• Cassandra Internals: Thread pool pending and blocked tasks, dropped messages, hinted handoff
• JVM / Garbage Collection: Application throughput, GC time and pause percentiles, heap memory
• Storage & Capacity: Live against total disk space per table, storage growth rate per node

The metrics come from the OpenTelemetry Java Agent, which runs inside the Cassandra JVM and reads Cassandra's own MBeans. It exports over OTLP to the node's OTel collector on port 4318, so no Prometheus scrape port is involved. One agent serves every Cassandra release, and it needs no per-release selection: a version installed with cassandra install --url or --branch reports the same metrics as a stock release.

Series carry the Prometheus label job="cassandra", and each node also reports a cassandra_build label naming the build it is running. That label is what lets a dashboard compare two versions in one cluster. See OpenTelemetry for the metric names, the rule file, and how a node derives its own labels.

If the agent jar or the rule file is missing, Cassandra says so on stderr at startup — check journalctl -u cassandra on the node when a dashboard is empty.

Latency arrives as pre-aggregated percentile gauges read from Cassandra's own EstimatedHistogram, the same reservoir nodetool proxyhistograms prints. histogram_quantile() does not apply to them, and they cannot be re-aggregated into a cluster-wide percentile.

Annotations

Annotations mark a moment on the dashboards' timeline. Use them as A/B config-change markers; for example, drop one before and one after you change a Cassandra setting, so a latency shift lines up with the change that caused it.

Create an annotation with grafana annotate:

# A point marker at the current time
easy-db-lab grafana annotate --text "raised concurrent_writes to 128" --tags config

# A region marker spanning a window
easy-db-lab grafana annotate --text "load test" --tags test --time -30m --time-end now

A plain global marker (no --dashboard and no --panel scope) is automatically tagged easydblab, in addition to any --tags you pass. Every annotation is also mirrored to Loki as it is created (source="annotation"), and the core dashboards read their markers from Loki, so a global marker renders on every core dashboard and stays readable from S3 after the cluster is gone. A scoped marker (--dashboard or --panel) is not auto-tagged; it renders on its target dashboard. See Command Reference for all options.

Backing up annotations

The cluster is ephemeral, but the annotations are worth keeping. Back them up to the tenant's annotations directory in the account bucket (grafana/annotations/<tenant>/):

easy-db-lab grafana backup

This backup also runs automatically before teardown. When you run easy-db-lab down, the tool mirrors the annotations to Loki and stops the OTel collector, then flushes Loki and Mimir, waits for Tempo to upload its last traces, and backs up the annotations, all at once, before it removes any infrastructure. If any of that fails, down removes nothing and never starts a backend again; the report names every signal that is not saved. Pass --force to skip these steps and tear down anyway; it lists what it will not save before the confirmation prompt. See down for details.

Where observability data is stored

Every cluster belongs to an observability tenant, chosen at init --tenant <name> (default default) and fixed for the life of the cluster. Clusters that share a tenant share one store: their metrics, logs, traces, profiles and annotations sit side by side and can be read together.

All of a cluster's observability data lands in the account bucket. Each backend has its own root at the top level of the bucket, named for the tool whose file format it holds:

An annotations backup never overwrites another: when one of the same cluster already exists under that second's name (a grafana backup just before down, say), the new one takes the next free second.

Cluster configuration stays under clusters/<name>-<id>/config/. The observability stack writes nothing to the per-cluster data bucket. easy-db-lab status shows the tenant and every location.

Native multi-tenancy. Mimir, Loki, Tempo and Pyroscope run with multi-tenancy on; the tenant is their tenant ID. Every writer sends it in the X-Scope-OrgID header — the OTel collector's metrics, logs and trace exporters, Tempo's metrics generator, the Alloy eBPF profiler, the Pyroscope Java agent (stress jobs, the Cassandra sidecar, EMR Spark, the Trino and Presto kits) and the Cassandra JFR shipper — on redirected clusters too. Every Grafana datasource sends it on every query, so a query from the cluster's Grafana returns its tenant's data. A query that sends no tenant, such as one made directly against a backend's HTTP API, reads nothing; add -H 'X-Scope-OrgID: <tenant>'. Pyroscope's own UI on port 4040 asks for the tenant once per browser; see Profiling.

Nothing is deleted automatically. No S3 lifecycle, expiry or retention rule is set on any bucket, and down deletes no data. The cluster's IAM role is denied deletes under mimir/, loki/, tempo/, grafana/ and reports/. pyroscope/ is left out: Pyroscope's own compaction writes a merged block and then removes the segments it merged, which loses nothing. Mimir runs no compactor and no retention. Loki's compactor is idle, its retention is off and its delete API is not served. Tempo runs with compaction — which is also what runs retention in Tempo 3 — turned off for every tenant, so no block is ever deleted. Pyroscope runs pure v2 storage with the metastore's retention cleanup off and no retention period; its v2 compaction merges this cluster's segments into blocks, which keeps their data.

Uploads within minutes. Each backend uploads while the cluster runs, so other clusters in the tenant can read the data: a metric sample is in mimir/ in about 2 minutes, a span in tempo/ in about 2 minutes, and a log line of a stream that keeps writing in loki/ in about an hour, with its index file at Loki's next 15-minute index rotation. Profiles are in pyroscope/ as soon as Pyroscope accepts them.

Durability across restarts. Tempo cuts a block at most every minute and uploads it while the cluster runs. Both of its write-ahead logs live on the control node's disk (/mnt/db1/tempo), so a restart of the Tempo pod does not lose spans it received but has not yet uploaded. Tempo acknowledges a push while the spans are still in memory and writes them to the WAL about 2 seconds later (10 seconds at most for a trace that is still receiving spans). A graceful restart writes every in-memory trace to the WAL first and loses nothing. A Tempo process killed outright loses only the spans from those last 2 seconds. The WAL is not fsynced, so a crash of the control node itself can also lose WAL data the operating system had not yet written to disk. Tempo keeps every span it receives: its per-tenant limits on live traces and trace size are off, and its ingestion rate limit is 1 GB/s, so a stress load is not thinned out there. The stress job itself traces only 1% of its requests, so request rates for cassandra-easy-stress on the Trace RED and Tempo dashboards show 1% of the real rate. It also stores every attribute value in full: the distributor's max_attribute_bytes is 0, where Tempo's default of 2048 bytes would truncate long values such as process.command_line. Any span Tempo does drop is counted in tempo_discarded_spans_total. The Pyroscope metastore index lives on /mnt/db1/pyroscope, so profiles written before a Pyroscope restart are still returned after it.

Known limit: Pyroscope after teardown. Pyroscope v2 finds blocks only through the cluster's local metastore index and cannot rebuild it from S3. After down, the profiles stay in pyroscope/ but cannot be queried without that index, which down does not save.

Restarts. up and grafana update-config restart an observability workload only when its configuration changed: each workload's pod template carries a hash of the pod template itself (image, probes, environment, volumes) and of the ConfigMaps it reads. A dashboard edit no longer restarts Tempo or Pyroscope. Both commands print, for each workload, whether its configuration changed (and it is rolling) or is unchanged (and it is left running). A new image or probe counts as a change, because Kubernetes rolls the workload for it.

Versions. Tempo 3.0.3, Pyroscope 2.3.1, Grafana 13.2.2 with image renderer v5.12.4, Alloy v1.20.0, Beyla 3.36.0, Mimir 3.2.1, Loki 3.7.8, the OpenTelemetry collector 0.161.0 (cluster, stress-job sidecar and EMR), and Fluent Bit 5.1.2. No observability image runs latest. The collector also scrapes its own telemetry (:8888, job otel-collector), so exporter send failures are visible in Mimir. In local mode it also scrapes the Mimir (:9009, job mimir), Loki (:3100, job loki), Tempo (:3200, job tempo) and Pyroscope (:4040, job pyroscope) servers' own metrics. Only the collector on the node running each server scrapes it, so each job has one target, and instance is the server's pod name. A redirect cluster runs none of these servers and renders none of these jobs.

eBPF Observability

The cluster deploys eBPF-based agents on all nodes for deep system observability:

Beyla (L7 Network Metrics)

Grafana Beyla uses eBPF to automatically instrument network traffic and provide RED metrics (Rate, Errors, Duration) for:

• Cassandra CQL protocol (port 9042) and inter-node communication (port 7000)
• ClickHouse HTTP (port 8123) and native (port 9000) protocols

Metrics are scraped by the OTel collector and stored in Mimir.

Beyla observes from outside the process only: its Java agent injection is turned off (javaagent.enabled: false), so it never attaches an agent to the database JVM.

ebpf_exporter (Low-Level Metrics)

Cloudflare's ebpf_exporter provides kernel-level metrics via eBPF:

• TCP retransmits — count of retransmitted TCP segments
• Block I/O latency — histogram of block device I/O operation latency
• VFS latency — histogram of filesystem read/write operation latency

These metrics are scraped by the OTel collector and stored in Mimir.

kube-state-metrics

kube-state-metrics runs on the control node of every cluster and exposes the state of Kubernetes objects as Prometheus metrics:

• Pods — kube_pod_status_phase, kube_pod_container_status_restarts_total, kube_pod_container_status_waiting_reason
• Workloads — kube_deployment_status_replicas_available, kube_daemonset_status_number_ready, kube_statefulset_status_replicas_ready, kube_job_status_succeeded
• Nodes and storage — kube_node_status_condition, kube_persistentvolumeclaim_status_phase

The OTel collector scrapes it once, through pod discovery on the control node, so each series appears one time. The metrics carry the cluster label like every other scrape.

Cilium metrics

On a Cilium cluster (the default CNI), the OTel collector also scrapes the Cilium agent (localhost:9962 on every node), Hubble (localhost:9965 on every node), and the Cilium operator (port 9963 on the node that runs it). A Flannel cluster has none of these jobs. See Pod Networking (CNI) for the job definitions, the Hubble UI NodePort, and platform cni.

See Profiling for continuous profiling with Pyroscope.

Redirecting Telemetry to an External Stack

By default each cluster stands up its own local observability backends on the control node: Mimir, Loki, Tempo, the Pyroscope server, and Grafana. Redirect mode ships all telemetry to an external observability stack instead. A redirect cluster stands up no local backends and no local Grafana.

Redirect mode is useful when you run several clusters and want one place to view them all. A common case is a second data center that reports into the first data center's stack.

Enabling Redirect Mode

Set the redirect target once, at init, with --redirect-telemetry <host>:

easy-db-lab init --redirect-telemetry 10.0.0.9 --up

<host> is the hostname or IP of the external stack. The tool derives the four signal endpoints from that host and the known stack ports:

• Metrics: http://<host>:9009/api/v1/push (Mimir remote write)
• Logs: http://<host>:3100/otlp (Loki OTLP ingest)
• Traces: <host>:4320 (Tempo OTLP gRPC receiver)
• Profiles: http://<host>:4040 (Pyroscope ingest)

The traces endpoint uses the OTLP receiver port 4320, not Tempo's query port 3200.

Overriding Individual Endpoints

If the external stack does not use the easy-db-lab port layout, override any signal. An override wins over the derived value; the other signals still derive from the base host.

easy-db-lab init --redirect-telemetry 10.0.0.9 \
  --redirect-metrics-endpoint http://metrics.example.com:9009/api/v1/push \
  --redirect-traces-endpoint tempo.example.com:4320 \
  --up

The four override options are --redirect-metrics-endpoint, --redirect-logs-endpoint, --redirect-traces-endpoint, and --redirect-profiles-endpoint.

Origin Identity

Every signal carries the source cluster's name as its origin identifier. Metrics and traces carry a cluster label; logs carry a cluster field; profiles carry a cluster label. Use that identifier to tell one cluster's telemetry from another on the shared stack.

Commands That Need the Local Stack

A redirect cluster has no local backends or Grafana to act on. These commands refuse to run on a redirect cluster and tell you why:

• grafana update-config
• logs query

down also skips its pre-teardown save (annotation mirror, collector stop, flushes, Tempo drain and annotations backup) on a redirect cluster.

Run these commands against the external stack instead. Kit dashboards are not installed on a redirect cluster; the dashboards live on the external stack's Grafana.

Validation

The tool validates the four endpoints for well-formedness at init and again at the start of up, before it creates any AWS resource. If a signal endpoint is missing or malformed, the tool names the offending signal and stops without standing anything up. Validation checks structure only; it does not probe the external stack for reachability. An unreachable but well-formed endpoint surfaces later as a collector send failure.

Profiling

Continuous profiling is provided by Grafana Pyroscope, deployed automatically as part of the observability stack.

There are two capture mechanisms in the fleet, and which one applies depends on the workload:

• Cassandra uses runtime-controlled async-profiler, attached to the already-running JVM and driven by the cassandra profile command group. Nothing profiling-related is injected at JVM startup.
• Every other JVM workload — stress jobs, Presto/Trino, Spark/EMR, the Cassandra Sidecar — still uses the Pyroscope Java agent, loaded with -javaagent at startup.
• Everything else (ClickHouse, TiDB/TiKV/PD, and all host processes) is covered by the Grafana Alloy eBPF profiler.

Architecture

Profiling data is collected from these sources and sent to the Pyroscope server on the control node (port 4040):

• async-profiler, attached at runtime (Cassandra) — A systemd timer on every Cassandra node runs edl-profiling-reconcile, which attaches async-profiler to the live Cassandra JVM, writes rotating JFR chunks to /mnt/db1/cassandra/profiles, uploads completed chunks to Pyroscope's /ingest, and prunes the directory. What is being profiled changes without restarting Cassandra.
• Pyroscope Java agent (Stress jobs) — Runs as a -javaagent inside cassandra-easy-stress K8s Jobs. Collects CPU, allocation, and lock profiles. The agent JAR is mounted from the host via a hostPath volume.
• Pyroscope Java agent (Presto) — Runs as a -javaagent inside both the Presto coordinator and worker JVMs, injected via JAVA_TOOL_OPTIONS during presto start. Profiles appear under service_name=presto with component=coordinator or component=worker.
• Pyroscope Java agent (Spark/EMR) — Runs as a -javaagent on Spark driver and executor JVMs, installed via EMR bootstrap action to /opt/pyroscope/pyroscope.jar. Profiles appear under service_name=spark-<job-name>.
• Grafana Alloy eBPF profiler — Runs as a DaemonSet on all nodes via Grafana Alloy. Profiles all processes at the system level using eBPF, including kernel stack frames. Pod processes are attributed per pod/container/service_name; see the eBPF Agent section below.

Cassandra profiling at runtime

A cluster is profiling as soon as it comes up — setup-instances seeds each Cassandra node with profiling enabled and -e cpu --alloc 512k --lock 10ms. Nothing needs to be run to get CPU, allocation and lock-contention flame graphs. Only wall-clock sampling is exclusive of a CPU event, so all three of these run in one recording.

Why it is not a startup agent

The Pyroscope Java agent takes exactly one primary profiler.event, fixed for the JVM's lifetime. Changing what was being profiled therefore meant restarting Cassandra, which discards page cache and compaction state — precisely what an operator on a benchmarking rig is trying to observe. Attaching at runtime makes the event set a runtime property instead. It also takes profiling out of the startup path entirely, so a bad profiler configuration can no longer abort Cassandra's start.

The cost, worth knowing: attach-based profiling dies with the JVM, so a node-resident reconciler re-establishes it. After a Cassandra restart or a mode switch, expect up to one reconcile interval (60s by default) with no profiling. That is expected behaviour, not a bug.

Switching modes

There is one async-profiler session per JVM, so one mode at a time. Switch by stopping and starting:

# what is running right now, per node
easy-db-lab cassandra profile status

# switch the whole cluster to wall-clock profiling
easy-db-lab cassandra profile stop
easy-db-lab cassandra profile start -- -e wall -i 10ms

# back to CPU, with allocation sampling alongside it
easy-db-lab cassandra profile stop
easy-db-lab cassandra profile start -- -e cpu --alloc 512k

# only two nodes, rotating every 30 seconds
easy-db-lab cassandra profile start --hosts db0,db1 --loop 30s -- -e cpu

stop finalizes the in-flight JFR chunk before detaching, so a mode switch loses no data — the outgoing chunk still ships, and fetch and flamegraph can reach it once nothing is writing to it. stop also leaves your retention and byte ceiling exactly as start set them; it only turns profiling off. If a node's configuration document cannot be read — truncated by an interrupted write, say — stop has nothing to preserve and falls back to the defaults, and it says so with a Profiling.DesiredStateUnreadable event naming the node and the bounds it is applying, rather than resetting them silently.

Every profiling command applies to all Cassandra nodes by default; --hosts narrows it to a subset.

async-profiler's own parameters pass through

Anything after -- is handed to asprof byte-identically. easy-db-lab does not model, enumerate, or re-parse async-profiler's option surface, so options added by newer async-profiler releases work with no change here.

easy-db-lab cassandra profile start -- -e wall -i 10ms
easy-db-lab cassandra profile start -- -e cpu --lock 10ms --alloc 512k

Reserved parameters

Five parameters are rejected, at the CLI, before any node is contacted. Four of them easy-db-lab supplies itself, because the node-side JFR shipper depends on them; the fifth changes which JFR event types reach that shipper at all:

Rejection covers both of async-profiler's spelling systems (CLI flags and the comma-separated agent-option form) and reserved options smuggled inside another value after a comma, such as -e cpu,file=/tmp/elsewhere.jfr. That last case is the reason the check exists at all: without it, profiling would appear to work while chunks landed somewhere the shipper never looks and Pyroscope stayed silently empty.

The first four remove no capability. Every profiling parameter async-profiler offers remains reachable, and the one knob from that set people actually want — rotation — is a first-class option on the command itself:

easy-db-lab cassandra profile start --loop 30s -- -e cpu

Retention on the node

Shipped chunks are pruned by age and by total size: their data is already in Pyroscope. A chunk that has not shipped, or that Pyroscope rejected, is never deleted — it is data you cannot get back.

When the directory reaches --max-bytes and only unshipped or rejected chunks are left, the node stops recording instead of deleting anything, so an unreachable Pyroscope still cannot fill /mnt/db1 and take the database down. Recording resumes on its own once the directory is back under the bound — when shipping recovers and shipped chunks are pruned, or when you fetch and remove chunks. The stop is reported by cassandra profile status (STOPPED), a typed Profiling.RecordingStoppedAtBound event, the jfr_recording_stopped journal line, and the edl_jfr_recording_stopped_at_bound metric.

easy-db-lab cassandra profile start --retention 30 --max-bytes 1073741824 -- -e cpu

Profiles live in /mnt/db1/cassandra/profiles, deliberately not artifacts/ — that directory is world-writable and holds operator-dropped heap dumps and jstacks, which automatic pruning must never delete.

Fetching chunks and building flame graphs

# download the 5 most recent completed chunks from every node into ./profiles/<host>/
easy-db-lab cassandra profile fetch

easy-db-lab cassandra profile fetch --last 20 --hosts db0

# convert the 10 most recent chunks into a flame graph on the node and download it
easy-db-lab cassandra profile flamegraph --last 10

# split stacks by thread — see below
easy-db-lab cassandra profile flamegraph --last 10 -- --threads

Conversion runs on the node, because jfrconv is installed there and not on your machine, and because the rendered HTML is far smaller than the raw JFR.

jfrconv arguments pass through after -- the same way async-profiler's do. Only its input and output are reserved — positional arguments and -o/--output are rejected, because easy-db-lab supplies the input chunks (--last) and the output destination (--format).

The chunk currently being written is never offered by fetch or flamegraph: async-profiler finalizes constant pools when a chunk closes, so an unfinalized chunk is unreadable by any tool. Everything else on the node is offered, including chunks already shipped to Pyroscope — shipping marks a chunk with a -shipped.jfr infix rather than moving it aside, so --last 20 really does give you twenty chunks. Once you stop profiling, the final chunk of the run becomes available too, since nothing is writing to it any more.

Thread-pool attribution

jfrconv --threads on a fetched chunk is the only way to see Cassandra thread-pool attribution (ReadStage-3, CompactionExecutor:4, MutationStage-7).

Pyroscope's JFR ingest discards thread identity: the JFR carries sampledThread and Grafana's parser decodes it, but the pprof converter never reads it. Passing -t to async-profiler does not help either — in JFR output mode it deliberately puts thread identity in the constant pool rather than into the stack. So the continuous-profiling path structurally cannot show you which pool a stack came from, and the flamegraph path exists alongside it precisely for this:

easy-db-lab cassandra profile flamegraph --last 20 -- --threads

Status

easy-db-lab cassandra profile status

Reports per node: what you asked for (desired) and what is actually attached to the JVM (attached), reported separately, the process being profiled, how long the session has been running, your arguments verbatim, the full command line as actually invoked (including the -o jfr --loop ... -f ... the tool appended), chunks pending/shipped/rejected, bytes held on disk, and the most recent shipping and attach errors.

It also reports pruned: — how many shipped chunks the node reclaimed for age and for size — and on disk: as bytes held against the --max-bytes ceiling.

A node that stopped recording at its byte ceiling is marked STOPPED and reports attached: no (stopped at the size bound). It emits a typed Profiling.RecordingStoppedAtBound event, and no Profiling.AttachFailed: nothing is wrong with the JVM. Fix shipping, raise --max-bytes, or fetch and remove chunks; recording resumes on the next pass once the directory is back under the ceiling.

The report is a snapshot the node's reconciler left behind, not a live reading, so status also prints updated: — how long ago that node last completed a pass. If the reconciler has stopped running, the report is marked STALE above everything else in it and names the unit to check, and status emits a typed Profiling.StateStale event. Without that, a node whose timer is masked keeps reporting attached: yes with a session age that grows convincingly against the current clock while nothing is running.

A node whose configuration document cannot be read is marked CONFIG instead, never STALE, and emits Profiling.NodeConfigUnreadable. The two are opposite diagnoses. A pass that cannot read /etc/easy-db-lab/profiling.json refuses only the attach/detach decision — it keeps shipping and pruning under the bounds the last good pass recorded, and keeps reporting every metric — so the reconciler is running perfectly and the file it reads is what is wrong. Run cassandra profile start to rewrite it. A node with no document at all has simply never been configured; it is not marked CONFIG.

desired and attached are two lines rather than one because they answer different questions. A node showing desired: enabled with attached: no is a node whose profiler will not attach — usually PrivateTmp hiding the attach socket, a java.io.tmpdir mismatch, or perf_event_paranoid — and status emits a typed Profiling.AttachFailed event with the reconciler's captured error, rather than letting it look like a deliberate stop.

A node that has just started is marked WAITING instead, and reports attached: no (waiting for the database to become ready). The reconciler will not attach to a database whose native transport is not yet listening, because async-profiler attaches with jattach, jattach signals SIGQUIT, and a process that has not installed a handler for that signal yet is killed by it — which is how attaching to a starting node used to kill it. This state clears itself on the first pass after the database is up, so it emits a typed Profiling.AttachDeferred event, which is not an error, and no Profiling.AttachFailed. If it does not clear, see edl_jfr_attach_deferred_total below.

Shipping and attach problems are also visible without the CLI. The reconciler writes structured logfmt lines to journald, which Fluent Bit ships to Loki, and exports counters to the node's OTel collector, which forwards them to Mimir:

Each successful attach also logs event=jfr_session_started ... ready=<code>, naming which readiness answer allowed it. ready=listening is the healthy answer. A node — or a whole cluster — attaching on ready=uptime is profiling ten minutes late every time: the socket check is not finding the native transport, and the ten-minute backstop is doing work it was only meant to do for a transport that is deliberately off.

Every series carries the node's host_name and the cluster label, the same as the rest of the stack.

The cpu+wall hazard

Combining a CPU event and wall-clock sampling in one recording corrupts the CPU sample weights, silently, by up to three orders of magnitude.

The defect is upstream, in Pyroscope's JFR ingest. Pyroscope delegates to github.com/grafana/jfr-parser (pinned at v0.18.0), whose pprof/parser.go:56 declares the sample value buffer once, outside the event loop:

var values = [2]int64{1, 0}

The wall-clock branch assigns values[0] = WallClockSample.Samples and never restores it, and the execution-sample branch reuses the buffer without re-initialising it. After the first wall sample, every subsequent CPU sample carries that wall event's batch count as its weight. Wall sampling coalesces idle threads into batches of up to 1000, and Cassandra runs hundreds of mostly-idle pool threads — hence the magnitude.

Three things follow:

• The corruption is scoped to a single upload. Separate sessions are uploaded separately and are each clean. Switching modes with stop/start is the supported approach and is not affected.
• --nobatch is not a mitigation, and easy-db-lab rejects it. It does not merely disable coalescing: it changes the emitted event type, so wall samples arrive as jdk.ExecutionSample — indistinguishable from CPU samples — and Pyroscope merges them all into process_cpu. It causes exactly the failure it appears to prevent. async-profiler itself refused the flag until 4.5 accepted it, so the guard now lives in easy-db-lab.
• Getting CPU, wall, and allocation profiles requires more than one session, in sequence. cpu+alloc in one session is fine; wall belongs in its own.

Note also that CPU and wall sampling have very different costs. A CPU sample only fires on a running thread, so its rate is bounded by core count. A wall tick targets threads regardless of run state, so its rate is bounded by thread count — far higher on a Cassandra node. That is why wall is treated as its own mode rather than something left on by default.

Accessing profiles

Profiling dashboard

1. Open Grafana (port 3000)
2. Navigate to Dashboards and select the Profiling dashboard
3. Use the Service dropdown to select a service (e.g. cassandra, cassandra-easy-stress, clickhouse-server)
4. Use the Hostname dropdown to filter by specific nodes
5. Select a time range

The dashboard includes panels for:

• CPU Flame Graph — CPU time spent in each method
• Wall Clock Flame Graph — Wall-clock time, for finding I/O and blocking
• Memory Allocation Flame Graph — Heap allocation hotspots
• Lock Contention Flame Graph — Time spent waiting for monitors
• Mutex Contention Flame Graph — Mutex delay analysis

Grafana Explore

1. Open Grafana (port 3000) and navigate to Explore
2. Select the Pyroscope datasource
3. Choose a profile type (e.g. process_cpu, memory, mutex, wall)
4. Filter by labels: service_name, hostname, cluster

Cassandra profiles arrive under service_name=cassandra with hostname and cluster labels.

Pyroscope runs native multi-tenancy, and the Grafana Pyroscope datasource sends the cluster's tenant (init --tenant, default default) on every query, so Explore and the dashboards show the cluster's tenant. Every profile writer sends the same tenant in X-Scope-OrgID. A request made directly against Pyroscope's HTTP API (port 4040) must send it too: curl -H 'X-Scope-OrgID: <tenant>' ....

Pyroscope UI

Pyroscope 2's own UI is served on port 4040, and the profiling dashboard links to it as Pyroscope UI. On the first visit the UI finds that the server runs multi-tenancy (its first query is refused with 401 no org id) and shows an Enter a Tenant ID dialog. Enter the cluster's tenant: the value of init --tenant, default if none was given. The UI remembers it per browser (in localStorage, as pyroscope:tenantID) and sends it as X-Scope-OrgID on every query; the navbar can change it. The dashboard link cannot preselect the tenant, because the UI reads no URL parameter for it.

Profile types

Cassandra (runtime async-profiler)

Which types are populated depends on the session that is running, since one session runs at a time.

Thread identity is not available through this path; use flamegraph -- --threads (see above).

Java agent (stress jobs, Presto, Spark, sidecar)

eBPF agent (all processes)

The eBPF agent profiles all processes on every node, including ClickHouse and other kit databases (TiDB, TiKV, PD). Since these are written in C++/Go, only CPU profiles are available.

Processes running inside Kubernetes pods are attributed to their pod: they carry namespace, pod, container, and a service_name derived as <namespace>/<container> (for example tidb-cluster/tikv). Host processes that don't run in a pod are still profiled, just without pod labels.

Stress job profiling

Stress jobs are automatically profiled via the Pyroscope Java agent. No configuration is needed — when you start a stress job, the agent is mounted from the host node and configured to send profiles to the Pyroscope server. Profiles appear under service_name=cassandra-easy-stress with cluster and job_name labels.

Configuration

Cassandra runtime profiling

You should not need to edit these by hand; cassandra profile start/stop own the desired-state document, and stopping is recorded as an explicit disabled state rather than by deleting the file.

eBPF agent

The eBPF profiler runs as a privileged Grafana Alloy DaemonSet (pyroscope-ebpf) and profiles all processes on each node. Configuration is in the pyroscope-ebpf-config ConfigMap (Alloy River format). It uses discovery.kubernetes to discover the pods on each node, discovery.process (joined to those pods by container id) to discover host processes, and pyroscope.ebpf to collect CPU profiles. The DaemonSet runs under the pyroscope-ebpf ServiceAccount, whose ClusterRole grants read access to pods so samples can be attributed to a pod/container/service_name.

Pyroscope server

The Pyroscope server (2.3.1) runs on the control node with pure v2 storage in the account bucket under s3://<account-bucket>/pyroscope/, and native multi-tenancy. Nothing is deleted by age: the metastore's retention cleanup is off and there is no retention period. Its metastore index lives on the control node at /mnt/db1/pyroscope, so profiles stay queryable across a Pyroscope restart. Pyroscope accepts a push only after its segment is in S3, so down needs no flush for profiles and reports that. After down the profiles stay in S3, but v2 cannot find them without the cluster's index (see Where observability data is stored). Configuration is in the pyroscope-config ConfigMap.

Data flow

Cassandra is the one path where nothing is injected into the JVM at startup and nothing pushes from inside the process — the reconciler on each node reads finished chunks off disk and uploads them.

Cassandra JVM
    │ (async-profiler attached at runtime, writes JFR chunks)
    ▼
/mnt/db1/cassandra/profiles/*.jfr
    │ (edl-profiling-reconcile.timer, every 60s: ship completed chunks, prune)
    ├──────────────► Pyroscope Server (:4040) ──┐
    │                                            │
    └──(fetch / flamegraph)──► your workspace    │
                                                 │
Stress Jobs   ──(Java agent)────────────────────►│
Presto JVMs   ──(Java agent)────────────────────►│
Spark JVMs    ──(Java agent)────────────────────►│
All Processes ──(eBPF agent)────────────────────►│
                                                 │
                                                 ▼
                                            S3 storage
                                          Grafana (:3000)
                                     Pyroscope datasource
                                    + Profiling dashboard

Metrics (Mimir)

Grafana Mimir stores the metrics from every node in the cluster. It runs on the control node and writes its blocks to the account bucket, so a cluster's metrics outlive the cluster. Every cluster's Mimir reads the whole shared store, so each cluster's Grafana shows the metrics of every tenant and of every past cluster.

How metrics get there

The OpenTelemetry Collector runs on every node. It gathers host metrics, scrapes every database, kit and infrastructure exporter, and receives OTLP pushes (the OpenTelemetry Java agent, stress jobs, kits). It sends everything to Mimir with Prometheus remote write, at http://mimir.default.svc.cluster.local:9009/api/v1/push, with the cluster's tenant in the X-Scope-OrgID header. Tempo's metrics generator writes the span metrics (traces_spanmetrics_*) and the service graph to the same place.

Every series carries a cluster label (<name>-<clusterId>), so clusters that share a tenant stay distinguishable.

Configuration

Mimir 3.2.1 runs as one process on the control node:

• Ports: 9009 (HTTP), 9097 (gRPC), 7947 (memberlist, loopback only), 11211 (memcached, loopback only)
• Local data: /mnt/db1/mimir on the control node (the write-ahead log, the last 15 minutes of blocks, and the store-gateway's index headers)
• Object storage: s3://<account-bucket>/mimir/<tenant>/
• Tenancy: native multi-tenancy; the tenant is the cluster's observability tenant

Mimir cuts a one-minute block and ships it within seconds, so a sample is in S3 about 2 minutes after it is written, while the cluster runs. A head with no writes for 2 minutes is compacted, so the last partial block ships too.

The read path

Queries read the ingester and, through the store-gateway, every tenant's blocks in S3. The store-gateway finds blocks through each tenant's bucket index, which the account compactor rewrites every minute, and it syncs every minute. The ingester keeps local blocks for 15 minutes; older data is read from S3. At 15 minutes a query opens far fewer one-minute local blocks than it did at 2 hours, and the local copies it drops are already in S3. A block the compactor merged away is no longer read 10 minutes after its deletion mark (5 minutes for a running query), so a read opens the merged block instead of its many 1-minute sources.

Mimir queues up to 5000 queries per tenant (query_scheduler.max_outstanding_requests_per_tenant; the default is 100). One load of the heaviest dashboard sends about 150 queries, and the query frontend splits each by day and runs as many as 14 parts at once. Queries are not sharded: on one Mimir process, sharding split each query into about 20 parts and made a dashboard load several times slower. With the default, a full dashboard load filled the queue and Mimir refused the rest with HTTP 429, so panels showed errors. The limit of 5000 holds two full loads of that dashboard at once.

A memcached sidecar in the Mimir pod (2 GB, on the control node's loopback, port 11211) holds Mimir's caches. The store-gateway caches the index, chunks and bucket metadata it reads from S3, and the query frontend caches query results. So a dashboard refresh reads from memory what the last load already read from S3. Mimir also keeps its S3 connections open and reuses them, instead of opening a new TLS connection for each read. The caches only hold copies; a restart empties them, and nothing is lost.

A tenant that has no bucket index yet returns no stored data rather than an error. A stale bucket index is accepted for about 10 years, so metrics queries still succeed while the compactor is stopped; blocks shipped after it stopped become readable once it runs again.

Nothing in the cluster deletes a block. Mimir on a cluster runs no compactor and no retention. The account compactor merges blocks and removes the sources only after it wrote the merged block. Every sample stays in S3 until you delete it yourself.

Querying metrics

Grafana

Grafana's default datasource is Mimir (uid mimir). It sends the cluster's own tenant on every query, so it returns this tenant's data from every cluster in it. Dashboards narrow to one cluster with their cluster variable.

Grafana also has one metrics datasource per tenant in the shared store, Mimir (<tenant>) (uid mimir-<tenant>), and one for all of them, Mimir (all tenants) (uid mimir--all), which sends every tenant joined with |. up and grafana update-config list the tenants under mimir/ in the account bucket and rebuild these datasources, so run grafana update-config to pick up a tenant that appeared after up. A UID longer than 40 characters is shortened to mimir-<prefix>-<8 hex characters>.

Dashboards reach these datasources through their Metrics picker, not a fixed uid. The picker opens on Mimir (the cluster's own tenant); pick another tenant, or all tenants, and every metric panel, variable and link on the dashboard follows. See Monitoring.

HTTP API

Mimir serves the Prometheus API under /prometheus. A query that carries no tenant reads nothing, so always send X-Scope-OrgID:

source env.sh

# Every metric name
with-proxy curl -H 'X-Scope-OrgID: <tenant>' \
  "http://control0:9009/prometheus/api/v1/label/__name__/values"

# An instant query
with-proxy curl -H 'X-Scope-OrgID: <tenant>' \
  "http://control0:9009/prometheus/api/v1/query?query=up"

easy-db-lab status prints the tenant and the Mimir URL.

Teardown

down flushes Mimir before it removes anything, at the same time as it saves the other signals: it stops the ingester, which cuts and ships every block it holds before it answers, then scales Mimir to 0. It checks nothing else. The metrics are recorded as saved the moment this succeeds, so a later down skips them. If a step fails, down removes nothing and Mimir stays as that step left it: it is never started again, and the report names the step, each backend's state, and down --force. See down.

Troubleshooting

No metrics appearing

1. Check that Mimir is running and ready:

   kubectl get pods -l app.kubernetes.io/name=mimir
   kubectl logs -l app.kubernetes.io/name=mimir
   

2. Check that the collector is sending. It scrapes its own telemetry into Mimir (job otel-collector); a failing exporter shows up in otelcol_exporter_send_failed_metric_points_total, and in the collector's log:

   kubectl logs -l app.kubernetes.io/name=otel-collector
   

3. Check that the cluster-config ConfigMap carries the tenant:

   kubectl get configmap cluster-config -o yaml
   

An empty result from the HTTP API

The request had no X-Scope-OrgID header, or named another tenant.

Logs (Loki)

Grafana Loki stores the logs from every node in the cluster. It runs on the control node and writes its chunks and index to the account bucket, so a cluster's logs outlive the cluster and can be read from any later cluster in the same tenant.

How logs get there

The OpenTelemetry Collector runs on every node and reads:

Fluent Bit reads the systemd journal on every node and forwards it to the collector with source="journald". OTLP logs (the OpenTelemetry Java agent, Spark jobs) arrive at the collector directly.

Cassandra's application logs (system.log, debug.log) reach Loki only over OTLP, from the OpenTelemetry Java agent in the Cassandra JVM. They carry service_name="cassandra" and no source label, and are stored once. The collector tails only the JVM GC log from disk, because the JVM writes it directly and the agent never sees it.

The collector sends everything to Loki's OTLP endpoint, http://loki.default.svc.cluster.local:3100/otlp, with the cluster's tenant in the X-Scope-OrgID header.

Labels

Four resource attributes become stream labels. Loki adds service_name, and for pod logs the Kubernetes labels (k8s_namespace_name, k8s_pod_name, k8s_container_name):

Every other attribute (the systemd unit, the log file, the trace and span ids, the severity) is stored as structured metadata. LogQL reads it with a label filter after the selector, such as | systemd_unit="docker.service".

Configuration

Loki 3.7.8 runs as one process (target: all) on the control node:

• Ports: 3100 (HTTP), 9098 (gRPC)
• Local data: /mnt/db1/loki on the control node (write-ahead log, open chunks and the index not yet shipped)
• Object storage: s3://<account-bucket>/loki/; chunks sit under the tenant, the TSDB index under index/
• Tenancy: native multi-tenancy; the tenant is the cluster's observability tenant

Loki flushes a chunk to S3 once it spans an hour, so a stream that keeps writing reaches S3 within about an hour. Its index uploads on Loki's own 15-minute rotation, and Loki re-reads the other clusters' index from S3 every minute. A line from another cluster in the tenant becomes readable here once its index file is in S3.

Out-of-order window. Loki rejects a line that is more than 30 minutes older than the newest line of its stream (half of the 1-hour chunk age). The window is wide so that a slow Loki restart does not drop logs. A line replayed or backdated by more than 30 minutes on a stream that is still writing is refused. The annotation mirror is not affected: each annotation is a stream of its own.

Nothing in the cluster deletes a chunk or an index file. The cluster's Loki compactor is idle, retention is off, and the delete API is not served. The account compactor merges each day's index files into one, with retention off, and removes the sources only after the merged file is written. Every log line stays in S3 until you delete it yourself.

Every cluster's Loki reads every tenant's chunks and index from the shared store; nothing about this changes with the account compactor.

Querying logs

The CLI

easy-db-lab logs query reads this cluster's logs:

# The last hour
easy-db-lab logs query

# By source, host or systemd unit (`cassandra` is the application log, `cassandra-gc` the JVM GC log)
easy-db-lab logs query --source cassandra --host db0
easy-db-lab logs query --source cassandra-gc --host db0
easy-db-lab logs query --source journald --unit docker.service

# Lines containing text
easy-db-lab logs query --grep "OutOfMemory"

# Time range and limit
easy-db-lab logs query --since 30m --limit 500

# A raw LogQL query, sent unchanged
easy-db-lab logs query -q '{service_name="cassandra"} |= "Exception"'

Every option but --query is scoped to this cluster. A raw query is sent as written, so it reads every cluster in the tenant unless it names a cluster.

Grafana

The Loki datasource (uid loki) sends the cluster's own tenant on every query. Grafana also has one logs datasource per tenant in the shared store, Loki (<tenant>) (uid loki-<tenant>), and one for all of them, Loki (all tenants) (uid loki--all). Each links a trace ID to the Tempo datasource of the same tenant view. See the Mimir page for how the tenants are found. Dashboards reach them through their Logs picker, which opens on Loki; the annotation markers follow it too.

The Log Investigation dashboard (Dashboards → Log Investigation) filters by:

For anything else, open Explore, choose the Loki datasource, and write LogQL:

{cluster="<name>-<clusterId>", service_name="cassandra"} |= "Exception"
{cluster=~".+", host_name="db0"} | systemd_unit="cassandra.service"
sum by (source) (count_over_time({cluster="<name>-<clusterId>"}[5m]))

A LogQL selector needs at least one matcher that cannot match an empty value, which is why "all clusters" is written cluster=~".+". See the LogQL documentation.

A trace in Tempo links to its logs through the trace_id.

HTTP API

A query that carries no tenant reads nothing, so always send X-Scope-OrgID:

source env.sh
with-proxy curl -G -H 'X-Scope-OrgID: <tenant>' \
  --data-urlencode 'query={service_name="cassandra"}' \
  "http://control0:3100/loki/api/v1/query_range"

Annotations

Every Grafana annotation is also written to Loki as its own stream (source="annotation", with the cluster label and an annotation_id), so the annotations stay readable from S3 after the cluster and its Grafana are gone. The core dashboards read them back from Loki. Loki accepts entries only from the last 8760 hours to 24 hours ahead, so an annotation outside that window is not mirrored; the tool warns with its id, and the annotation stays in Grafana's JSON backup.

Teardown

down flushes Loki before it removes anything, at the same time as it saves the other signals: it stops the ingester, which answers only once every open chunk is in S3, then scales Loki to 0 and waits for the pod to go, which builds and uploads its index. It checks nothing else. The logs are recorded as saved the moment this succeeds, so a later down skips them. If a step fails, down removes nothing and Loki stays as that step left it: it is never started again, and the report names the step, each backend's state, and down --force. See down.

Troubleshooting

No logs appearing

1. Check that Loki is running and ready:

   kubectl get pods -l app.kubernetes.io/name=loki
   kubectl logs -l app.kubernetes.io/name=loki
   

2. Check that the collector is sending. A failing exporter shows up in otelcol_exporter_send_failed_log_records_total in Mimir, and in the collector's log:

   kubectl logs -l app.kubernetes.io/name=otel-collector
   

3. Check that the cluster-config ConfigMap carries the tenant:

   kubectl get configmap cluster-config -o yaml
   

Connection errors from logs query

logs query reaches Loki through the cluster's SOCKS proxy, which it starts when needed. Check that the cluster is up (easy-db-lab status) and that the control node answers (ssh control0 hostname).

Account Compactor

Every cluster writes its metrics, logs and traces to one shared store in the account bucket. The account compactor keeps that store readable: it merges small objects into larger ones and keeps Mimir's bucket index current, so every cluster's Grafana can read every tenant's metrics.

What it runs

The compactor is one ECS Fargate service per AWS account, easy-db-lab-compactor in the ECS cluster easy-db-lab. It runs in the account bucket's region, in its own small VPC, easy-db-lab-compactor: one public subnet, an internet gateway, and a security group with no ingress. The subnet is in the first availability zone, by name, where Fargate runs; up never places it in a zone without Fargate, such as use1-az3.

The service runs exactly 1 task (ARM64, 2 vCPU, 8 GiB, 100 GiB of disk). ECS never runs a second one: a new task starts only after the old one stopped. The task runs:

Mimir and Loki use the same configuration files as the clusters, with flag overrides, so the compactor always reads the store the way the clusters write it. Profiles are not compacted here: Pyroscope compacts its own segments on each cluster.

Logs go to the CloudWatch Logs group /easy-db-lab/compactor, with no retention policy.

No data is deleted by age

Compaction is on; retention and every other deletion path are off:

• Mimir: -compactor.blocks-retention-period=0 and -compactor.partial-block-deletion-delay=0.
• Loki: retention disabled, deletion mode disabled.
• Tempo: block retention 876000h (100 years), no per-trace size limit, and empty tenants are never deleted.

The compactor removes a source object only after it wrote a merged copy that holds its data. Its task role, EasyDBLabCompactorTaskRole, is the only role that may delete under mimir/, loki/ and tempo/, and it has no access to grafana/ or pyroscope/. The account bucket policy denies those deletes to every cluster-side role: the EC2 instance role, the EMR service role and the EMR EC2 role. Deletes you make yourself from your workstation are not affected.

Lifecycle

• up starts the compactor when it is not running: it creates it when it is missing and starts it at 1 task when it is stopped. When it already runs, up leaves it as it is. up does not wait for the task to run.
• A new compactor configuration or version takes effect only when the service starts. To apply it to a running compactor, run observability compactor stop, then observability compactor start.
• down, after its teardown succeeds, stops the compactor when no other cluster uses the account bucket. It counts the VPCs tagged easy_cass_lab=1, in every enabled region, whose bucket tag names the bucket. The compactor's own VPC has no bucket tag, so it is never counted.
• down --all keeps the compactor's VPC, as it keeps the packer VPC.

Metrics queries keep working while the compactor is stopped. Blocks shipped after it stopped become readable once it runs again.

Commands

easy-db-lab observability compactor start    # start it as up does
easy-db-lab observability compactor stop     # set its desired count to 0
easy-db-lab observability compactor status   # state, why the task stopped, service events, the last 20 log lines

They work outside a cluster workspace. See Observability Commands.

Permissions

The compactor needs the EasyDBLabCompactor user policy: ECS, its CloudWatch Logs group, iam:PassRole on its two roles, and ECS's service-linked role. easy-db-lab show-iam-policies compactor prints it.

Cost

A 2 vCPU, 8 GiB ARM64 Fargate task costs about $0.10 an hour, while any cluster exists.

Kubernetes

easy-db-lab uses K3s to provide a lightweight Kubernetes cluster for deploying supporting services like ClickHouse, monitoring, and stress testing workloads.

Overview

K3s is automatically installed on all nodes during provisioning:

• Control node: Runs the K3s server (Kubernetes control plane)
• Cassandra nodes: Run as K3s agents with label type=db
• Stress nodes: Run as K3s agents with label type=app

Accessing the Cluster

kubectl

After running source env.sh, kubectl is automatically configured:

source env.sh
kubectl get nodes
kubectl get pods -A

The kubeconfig is downloaded to your working directory. source env.sh puts the workspace's bin/ first on your PATH, and the kubectl there is a wrapper that reaches the cluster through the SOCKS5 tunnel, or directly on a Tailscale cluster. See Tool Wrappers.

k9s

k9s provides a terminal-based UI for Kubernetes:

source env.sh
k9s

k9s uses the workspace kubeconfig and, like kubectl, runs through the workspace wrapper.

Port Forwarding

easy-db-lab uses a SOCKS5 proxy for accessing the private Kubernetes cluster.

Starting the Proxy

Any easy-db-lab command that needs the cluster starts the tunnel on its own. source env.sh does not start it. To start it for your shell or a browser:

easy-db-lab socks start

Manual Proxy Control

# Start the SOCKS5 tunnel, or reuse the running one, and print its port
easy-db-lab socks start

# Check the recorded port (after source env.sh)
socks5-status

# Stop the tunnel; the cluster keeps running
easy-db-lab socks stop

Running Commands Through the Proxy

kubectl, helm, cilium, curl, skopeo and k9s use the tunnel through the workspace wrappers. ssh, scp, sftp and rsync have wrappers too, which use the workspace sshConfig and do not need the tunnel. For other commands:

# Route any command through the proxy
with-proxy curl http://10.0.1.50:8080/api

Pushing Docker Images with Jib

easy-db-lab includes a private Docker registry accessible via HTTPS. You can push custom images using Jib.

Gradle Configuration

Add Jib to your build.gradle.kts:

plugins {
    id("com.google.cloud.tools.jib") version "3.4.0"
}

jib {
    from {
        image = "eclipse-temurin:21-jre"
    }
    to {
        // Use the control node's registry
        image = "control0:5000/my-app"
        tags = setOf("latest", project.version.toString())
    }
    container {
        mainClass = "com.example.MainKt"
    }
}

Pushing to the Registry

# Build and push to the cluster registry
./gradlew jib

# Or build locally first
./gradlew jibDockerBuild

Using Images in Kubernetes

Reference your pushed images in Kubernetes manifests:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: my-app
    image: control0:5000/my-app:latest

Node Labels

Nodes are automatically labeled for workload scheduling:

Using Node Selectors

Schedule pods on specific node types:

apiVersion: v1
kind: Pod
metadata:
  name: stress-worker
spec:
  nodeSelector:
    type: app
  containers:
  - name: worker
    image: my-stress-tool:latest

Useful Commands

# List all nodes
kubectl get nodes

# List pods in all namespaces
kubectl get pods -A

# Watch pod status
kubectl get pods -w

# View logs
kubectl logs <pod-name>

# Execute command in pod
kubectl exec -it <pod-name> -- /bin/bash

# Port forward a service locally
kubectl port-forward svc/my-service 8080:80

Architecture

Networking

• K3s server runs on the control node
• All nodes communicate over the private VPC network
• External access is via SOCKS5 proxy through the control node

Storage

• Local path provisioner for persistent volumes
• Data stored on node-local NVMe drives at /mnt/db1/

Kubeconfig

The kubeconfig file is:

• Downloaded automatically during cluster setup
• Stored as kubeconfig in your working directory
• Backed up to S3 for recovery

Pod Networking (CNI)

The K3s cluster runs one pod-network datapath. You select it at init time with --cni. Cilium is the default. --cni=flannel selects Flannel.

# Default: Cilium in ENI IPAM native-routing mode (no encapsulation)
easy-db-lab init my-cluster --db 3

# K3s's built-in Flannel VXLAN overlay
easy-db-lab init my-cluster --db 3 --cni=flannel

A cluster whose saved state records no CNI was provisioned before Cilium existed and is treated as Flannel.

AMIs built before the Cilium node fixes lack the base image's cloud-init hotplug setting and the systemd-networkd ENI drop-ins that Cilium's secondary ENIs need. If your AMI predates them, rebuild it with easy-db-lab build-image before provisioning a Cilium cluster.

On a Cilium cluster, up checks every node for those fixes before it starts K3s, and stops with an error naming each node that lacks them and the files it is missing:

• /etc/systemd/network/05-cilium-eni-primary.network
• /etc/systemd/network/06-cilium-eni-unmanaged.network
• /etc/cloud/cloud.cfg.d/90-easydblab-no-network-hotplug.cfg

To recover, rebuild the images with easy-db-lab build-image, then run easy-db-lab down and easy-db-lab up so the nodes launch from the new AMI. Or initialize the cluster with --cni=flannel, which needs none of them. Without the check, such a node joins the cluster normally and drops off the network only once Cilium attaches its second ENI.

Flannel

Flannel is the K3s built-in CNI. Pod traffic between nodes is encapsulated in VXLAN. Nothing in this page beyond the platform cni one-liner applies to a Flannel cluster.

Cilium

With Cilium (the default, or --cni=cilium), Cilium runs in ENI IPAM native-routing mode. Each pod gets a real VPC-routable secondary IP on its node's ENIs, so the VPC routes cross-AZ pod traffic with no tunnel. K3s keeps its own kube-proxy; Cilium's kube-proxy replacement is off.

up is safe to re-run on a Cilium cluster. If Cilium is already installed, the hook upgrades the release in place with the same flags instead of failing on the release name.

Two K3s add-ons are not installed on a Cilium cluster: Traefik and ServiceLB. ServiceLB publishes every node IP as a LoadBalancer ingress, and Cilium's port-0 wildcard for a LoadBalancer IP rejects pod traffic to node IPs, which blocks kubelet probes. A Flannel cluster keeps both.

With Tailscale enabled, up installs a small nftables chain on the control node (table ip edl_tailscale, loaded at boot by edl-tailscale-masquerade.service). The control node is the tailnet subnet router, and Cilium's own NAT chain would otherwise pass tailnet packets to the db nodes without masquerading them, so the db nodes' private IPs would time out from your machine while the control node worked. The chain masquerades Tailscale-forwarded traffic before Cilium sees it. It is installed only when Tailscale is enabled; re-running up replaces it in place.

hostPort works on a Cilium cluster through portmap chaining. With kube-proxy replacement off, Cilium does not implement hostPort itself, so it is installed with cni.chainingMode=portmap: the portmap CNI plugin runs after cilium-cni for every pod and publishes its host ports on the node. The plugin binary is the one K3s bundles; each node links it into /opt/cni/bin when K3s starts. Kits that publish a port through hostPort (Trino, Presto, Flink) are reachable at their node's private IP and scraped, as on Flannel. A pod started before the chaining was applied keeps its old networking until it is restarted.

Cilium's devices are pinned to the ENA interfaces (ens+). The tailnet interface tailscale0 is not a Cilium device, so its 1280 MTU does not lower the NIC MTU below the 9001 AWS offers.

Reading the datapath back

platform cni reads the live configuration and the per-node ENI state off the cluster. It is read-only.

easy-db-lab platform cni

On a Cilium cluster it prints the routing mode, IPAM mode, kube-proxy replacement, masquerade interfaces, native routing CIDR, and the Hubble UI URL. Then it prints one block per node: ENI count, subnet CIDRs, and IPs allocated, used, and available. The values come from the cilium-config ConfigMap and the CiliumNode objects, so they show what the cluster runs, not what init requested.

On a Flannel cluster it prints one line that names Flannel and exits 0.

Hubble UI

Hubble UI is exposed as a NodePort on port 31234. Open it on any node's private IP over the tailnet or the SOCKS tunnel; no port-forward is needed:

http://<control node private IP>:31234

platform cni prints the exact URL.

Metrics

On a Cilium cluster the OTel collector scrapes three Cilium jobs:

The metrics land in Mimir with the cluster label like every other scrape. A Flannel cluster renders none of these jobs.

Logs

The Cilium agent, operator, and Hubble pods run in kube-system. Their stdout and stderr are collected by the same container log pipeline as every other pod, so they are in Loki under the stream label k8s_namespace_name="kube-system". Nothing needs to be enabled.

Install markers on the timeline

up marks the Cilium install on the Grafana timeline with two annotations tagged cilium: Cilium install started and Cilium install finished (or Cilium install failed: <error>). Cilium installs before Grafana exists, so the timestamps are taken when the install runs and the annotations are posted once the observability stack is up. A telemetry-redirect cluster has no local Grafana and posts nothing. See Annotations.

kube-state-metrics

Every cluster, on either CNI, runs kube-state-metrics on the control node. It turns the state of Kubernetes objects (pods, deployments, nodes, PVCs, jobs) into Prometheus metrics such as kube_pod_status_phase and kube_node_status_condition. The OTel collector scrapes it once, through pod discovery on the control node, and the metrics carry the cluster label. See Monitoring.

Network Connectivity

This guide covers how to connect to your easy-db-lab cluster from your local machine.

Overview

easy-db-lab clusters run in a private AWS VPC. By default, up picks a random 10.X.0.0/16 block that no existing VPC in the region uses, and prints the one it chose. If creating the VPC fails, it retries a few times, each time on a new random unused block. To choose the block yourself, pass --cidr; it is used as-is and not retried:

easy-db-lab init --cidr 10.14.0.0/20 ...

The block must be /20 or larger, and it must name its prefix length and be in network form: init rejects 10.0.0.0 (no prefix) and 10.0.0.5/16 (host bits set; the error names 10.0.0.0/16) before it calls AWS.

There are two methods to access your cluster:

Both methods, and every command easy-db-lab runs on a node, travel over SSH. If your network blocks or re-routes outbound SSH (port 22), switch the profile's SSH transport to SSM Session Manager. It works underneath either method.

Tailscale VPN (Recommended)

Tailscale provides a persistent VPN connection to your cluster. Once connected, you can access cluster resources directly—no proxy configuration needed.

Why Tailscale?

• Native access - Use any tool (browsers, kubectl, ssh) without proxy configuration
• Persistent - Connection survives terminal sessions
• Team sharing - Share cluster access with teammates
• Reliable - No SSH tunnels to maintain or reconnect

Setup (One-Time)

Step 1: Configure Tailscale ACL

Go to Tailscale ACL Editor and add:

{
  "tagOwners": {
    "tag:easy-db-lab": ["autogroup:admin"]
  },
  "autoApprovers": {
    "routes": {
      "10.0.0.0/8": ["tag:easy-db-lab"]
    }
  }
}

The autoApprovers section automatically approves subnet routes, so you don't need to manually approve each cluster.

Step 2: Create OAuth Client

1. Go to Tailscale OAuth Settings
2. Click Generate OAuth Client
3. Configure:
   • Description: easy-db-lab
   • Scopes: Auth Keys write (auth_keys), to create the control node's key, and Devices › Core write (devices:core), so down and tailscale stop can remove the control node's device
   • Tags: Add tag:easy-db-lab
4. Click Generate and save the Client ID and Client Secret

Step 3: Configure easy-db-lab

easy-db-lab profile setup

Enter your Tailscale OAuth credentials when prompted.

Step 4: Connect this machine

Your own machine must be on the tailnet too. A Tailscale cluster routes every connection over the tailnet and starts no SOCKS proxy, so a logged-out client has no route to the cluster at all.

tailscale up
tailscale status

easy-db-lab up checks this before it creates any AWS resource. If the local client is logged out, or the tailscale command is missing, up stops immediately and names the cause.

Usage

Tailscale starts automatically with easy-db-lab up. Once connected:

# Direct access to private IPs
ssh ubuntu@10.0.1.50
curl http://10.0.1.50:3100/ready
kubectl get pods

# Web UIs work directly in your browser
# http://10.0.1.50:3000 (Grafana)

easy-db-lab down and easy-db-lab tailscale stop remove the cluster's control node from the tailnet, using the device ID recorded when Tailscale started on it (tailscale stop does this even when Tailscale is already down), so a later tailscale start does not leave the old device beside the new one. If the OAuth client is not allowed to delete devices, or no credentials are configured, the command exits non-zero and says so; the device ID stays in the cluster state, so running it again once the scope is granted removes it. tailscale start likewise removes a previously recorded device when the control node registers as a new one; if that removal fails it reports the old device (and the devices:core scope, when the OAuth client lacks it) so you can remove it by hand, records the new one, and still succeeds, so up carries on.

Manual Control

easy-db-lab tailscale start
easy-db-lab tailscale status
easy-db-lab tailscale stop

Cluster commands cannot reach this cluster until 'easy-db-lab tailscale start'.

Troubleshooting Tailscale

"requested tags are invalid or not permitted" - Add the tag to your ACL (Step 1).

Can't reach private IPs - Check subnet route is approved in Tailscale admin, or add autoApprovers to your ACL.

"the local Tailscale client is not connected" - up found your own machine off the tailnet. Run tailscale up, confirm tailscale status reports it connected, then run easy-db-lab up again.

"the 'tailscale' command was not found" - Install Tailscale from tailscale.com/download. If you use the macOS App Store build, its CLI is not on the PATH; see the Tailscale CLI docs.

"cannot reach it ... over the tailnet" - The control node joined the tailnet, but this machine has no route to the cluster's private network. up waits about two minutes for the route to arrive before it reports this. Approve the subnet route for the control node in Tailscale admin.

Using a custom tag:

easy-db-lab tailscale start --tag tag:my-custom-tag

SOCKS Proxy (Alternative)

If you don't want to set up Tailscale, the SOCKS proxy provides connectivity via an SSH tunnel through the control node.

┌─────────────────┐     SSH Tunnel      ┌──────────────┐
│  Your Machine   │ ──────────────────► │ Control Node │
│ 127.0.0.1:<port>│                     │  (control0)  │
└────────┬────────┘                     └──────┬───────┘
         │                                     │
    SOCKS5 Proxy                         Private VPC
         │                                     │
         ▼                                     ▼
   kubectl, curl                          VPC network

Quick Start

source env.sh
kubectl get pods
curl http://control0:3100/ready

The easy-db-lab CLI is the only thing that starts or stops the tunnel. Any command that needs to reach the cluster starts it, or reuses the one already running, before the command does its work. The CLI reuses a running tunnel only after a connection through it reaches the control node. If the tunnel's SSH connection died while its local port stayed open, the CLI stops it, starts a new one, and records the new port.

When the CLI replaces a tunnel, it prints the old port, the new port and the reason: the tunnel went to another control node, it no longer reached the control node, its process had exited, or its PID now belongs to another process. If your browser uses the tunnel, change its SOCKS port to the new one. If the old tunnel does not stop, the CLI prints an error with its PID; end it with kill -9 <pid>. The CLI no longer records that tunnel, so it will not stop an ssh the tunnel leaves: the error also gives the pkill command that ends it.

The tunnel reconnects by itself. The CLI runs it through <workspace>/bin/edl-socks-tunnel, which easy-db-lab writes with the tool wrappers. That script runs ssh -N -D 127.0.0.1:<port> and, when ssh exits, starts it again on the same port after a two-second pause. The tunnel listens on 127.0.0.1 only, and every client of it uses 127.0.0.1, never localhost: if another process holds the port on one loopback address, a client of localhost could reach that process instead. The SSH keepalives make ssh exit about 90 seconds after its connection dies, for example after a laptop sleep or a network change. Because the port does not change, wrapped tools in an open shell work again as soon as the connection comes back, with no CLI command. socks stop and down stop the script and its ssh.

The script restarts ssh only for a tunnel the CLI has verified. The CLI records the port in .socks5-proxy.env only after a connection through the tunnel reaches the control node, and the script checks that record. A tunnel that never came up (its port was taken, its key was refused) ends at once, and the CLI reports the failure or retries on a new port. If another process takes the port while ssh waits to restart, the script ends after that one attempt. The wrapped tools then fail with the socks start message, which names <workspace>/logs/socks5-proxy.log, and the next CLI command starts a new tunnel. The log records each ssh start and why the script ended, with a UTC timestamp on every line.

The tunnel is shared by every command in the workspace, so Ctrl-C does not end it. Pressing Ctrl-C during an easy-db-lab command stops that command and leaves the tunnel running. Use easy-db-lab socks stop to stop it.

To start it for everything else (your shell, a browser), run:

easy-db-lab socks start

socks start prints the tunnel's local port. On a Tailscale cluster it starts nothing and says that no tunnel is needed.

Each new tunnel listens on a random free local port, from 20000 to 29999; no port is preferred. The range is below the ports the OS gives outgoing connections (32768 and up on Linux, 49152 and up on macOS), so no outgoing connection can take the tunnel's port while ssh waits to restart. If 20 random ports in the range are all in use, the CLI fails with a message that names the range. Each workspace records its own port, so several clusters can run side by side, and two workspaces are unlikely to want the same port. If they do, the second one picks another port.

The CLI records the tunnel in the workspace's .socks5-proxy.env, as EDL_SOCKS_PORT (the port) and EDL_SOCKS_PID (the PID of the tunnel script), next to EDL_TAILSCALE_ACTIVE. The CLI is the only writer of this file; the wrappers and env.sh read it. Each write holds a lock on .socks5-proxy.env.lock, so two easy-db-lab commands in the same workspace never lose each other's values.

A wrapper checks only that some process still holds the recorded PID, because a shell script cannot read another process's arguments portably. The CLI checks more: it reuses a recorded tunnel only when the process with that PID is the workspace's tunnel script, with the recorded port and sshConfig. When another program now holds the PID, the next CLI command that needs the tunnel starts a new one.

Tool Wrappers

easy-db-lab writes a small wrapper script for each of these tools into the workspace's bin/ directory, at up and before every kit command:

source env.sh puts bin/ first on your PATH, and kit scripts get it first on theirs.

The ssh, scp and sftp wrappers add -F <workspace>/sshConfig before your arguments, and the rsync wrapper adds -ave "ssh -F <workspace>/sshConfig", so ssh db0 reaches the node by its alias. They reach the nodes over SSH itself, not the SOCKS tunnel, so they leave your proxy variables and KUBECONFIG unchanged. If the workspace has no sshConfig file, they name the missing file and exit 1.

Each call of the other wrapped tools reads the workspace's .socks5-proxy.env, which the CLI writes, and then:

• on a SOCKS cluster, routes that one call through the tunnel;
• on a Tailscale cluster, runs the tool unchanged, so it connects directly;
• with no tunnel recorded, or when the recorded tunnel script (EDL_SOCKS_PID) is no longer running, fails with exit code 1 and a message that says to run easy-db-lab socks start. It never uses the port of a tunnel that is gone, because another process may hold that port now.

The kubectl, helm, cilium and k9s wrappers always use the workspace kubeconfig. They replace any KUBECONFIG you exported, because a workspace's wrapper reaches only that workspace's cluster. A bare <workspace>/bin/kubectl get ns works with no setup: you do not need to source env.sh or set KUBECONFIG. If the workspace has no kubeconfig file, the wrapper names the missing file and exits 1. The curl and skopeo wrappers do not change KUBECONFIG.

The wrappers replace any proxy variables you set in your shell for that one call, so an exported NO_PROXY cannot send cluster traffic around the tunnel. They do not change your shell, and tools without a wrapper, aws included, connect directly. A wrapped tool sends all of its traffic through the tunnel, public URLs included.

command kubectl in a sourced shell also runs the wrapper, because the wrapper is a real file on your PATH, not a shell function. To run the real binary directly, call it by its full path, for example /usr/local/bin/kubectl.

If you sourced an env.sh from an older version in an open shell, source the new one again: it removes the old kubectl, helm, cilium, curl, skopeo, k9s, ssh, scp, sftp and rsync shell functions.

Manual Proxy Usage

For other commands, use the with-proxy helper from env.sh. On a Tailscale cluster it runs the command directly:

with-proxy wget http://10.0.1.50:8080/api
with-proxy http http://control0:3000/api/health

Kit commands over SOCKS

Kit lifecycle commands work transparently on SOCKS-only clusters, with no extra flags or setup, whether or not Tailscale is enabled.

kit install, <kit> start / stop and other lifecycle phases. Their shell steps, phase scripts and hooks run kubectl, helm and similar tools on your machine to apply manifests, wait on pods, and read pod state. Each of them runs with the workspace bin/ first on PATH and the absolute workspace kubeconfig in KUBECONFIG, so those tools go through the wrappers above. The CLI verifies the tunnel, and restarts it on a new port if it died, before the first step runs. Indirect calls (timeout, xargs, nested scripts) go through the wrappers too. See How local scripts reach the cluster.

easy-db-lab postgres start

sql. The sql command opens a short-lived in-process loopback bridge that forwards the JDBC connection through the existing tunnel to the database's private IP, then tears it down when the query finishes. This works for raw-TCP drivers (PostgreSQL, MySQL) as well as HTTP-based ones (Trino, ClickHouse):

easy-db-lab postgres sql "SELECT 1"

On Tailscale-enabled clusters the same commands connect directly to the private IP with no proxy, so behavior is identical either way. In neither path are the JVM-global socksProxyHost / socksProxyPort properties touched — routing is scoped per client.

Browser Access

Run easy-db-lab socks start and note the port it prints. Then configure your browser's SOCKS5 proxy:

Then access cluster services:

• Grafana: http://control0:3000
• Mimir: http://control0:9009/prometheus (send X-Scope-OrgID: <tenant>)
• Loki: http://control0:3100 (send X-Scope-OrgID: <tenant>)

Proxy Management

easy-db-lab socks start   # Start the tunnel, or reuse the running one, and print its port
socks5-status             # Show the recorded port (after source env.sh)
easy-db-lab socks stop    # Stop the tunnel; the cluster keeps running

easy-db-lab down stops the tunnel too. After socks stop, wrapped tools fail with the socks start message until a CLI command or socks start starts the tunnel again.

If the tunnel process does not end, socks stop prints its PID and exits non-zero. The PID and port stay recorded. Stop the process with kill <pid>, then run easy-db-lab socks stop again. down goes on with the teardown in that case and prints the same PID.

If the tunnel script was killed from outside (for example with kill -9), its ssh can keep running on its own. socks stop, down and the next tunnel start find that ssh by the recorded port and sshConfig, and stop it.

The Generated sshConfig

The sshConfig generated for your cluster applies its options to the cluster's hosts only. It starts with one Host db* app* control* !*.* block: a wildcard per node type, so it covers every node the cluster grows to, and !*.*, which keeps out every name with a dot, such as db.example.com, and every IP address. OpenSSH host patterns have no character classes, so db* alone would also match db.example.com. That block sets StrictHostKeyChecking=no, UserKnownHostsFile=/dev/null, User ubuntu, the cluster's IdentityFile, the keepalives and ConnectTimeout. Each node then has its own Host block with its address and, under the ssm transport, its ProxyCommand.

Cluster hosts never read your personal ~/.ssh/config or the system /etc/ssh/ssh_config, so no personal setting (such as ControlMaster, ProxyJump, ForwardAgent or another IdentityFile) reaches a node or the SOCKS tunnel. Every other host, such as github.com for a git that runs through the workspace's ssh wrapper, a dotted name like db.example.com, or an IP address, resolves exactly as it would without -F. The file ends with two Match originalhost blocks that include your personal config and then the system config for those names only. To see what applies to a host, run ssh -F <workspace>/sshConfig -G <host>.

Because the patterns are wildcards, a name of your own with no dot that starts with db, app or control, such as a personal alias dbprod, counts as a cluster host through the workspace wrappers: it gets the cluster's SSH settings, and not your own settings for it. Call such a host by its dotted name, or with the real ssh (for example /usr/bin/ssh dbprod).

Host Key Verification

For the cluster's nodes, the generated sshConfig sets UserKnownHostsFile=/dev/null alongside StrictHostKeyChecking=no. ssh — and therefore the SOCKS tunnel, whose edl-socks-tunnel script runs ssh -N -D against that config — never reads or writes your ~/.ssh/known_hosts for cluster nodes.

This matters because AWS recycles public IPs across ephemeral cluster lifetimes. Without this setting, a recycled IP that previously belonged to a different cluster (with a different host key) would make ssh hard-fail with REMOTE HOST IDENTIFICATION HAS CHANGED — StrictHostKeyChecking=no only auto-adds unknown hosts, it doesn't override a changed key for a host already recorded. Every cluster is short-lived and gets fresh host keys on every provision, so there is nothing to verify against across runs.

If you connected to easy-db-lab clusters before this change, their host keys may still be in your ~/.ssh/known_hosts. They're no longer read by the tool, so you can prune them any time — look for entries matching your cluster's Hostname lines in the generated sshConfig.

Tunnel Failures

If the SOCKS tunnel can't be established, the command that needed it fails immediately with a non-zero exit code rather than silently continuing against a proxy port nothing is listening on. The error names the SOCKS proxy as the failing component and points at socks5-proxy.log in your cluster workspace directory — that file holds the ssh -v transcript from the tunnel attempt and is the fastest way to find the real cause (a host-key mismatch, a security group blocking port 22, the control node not yet accepting SSH, and so on). The error quotes only the lines of the failed attempt.

If ssh is still trying to connect when the CLI stops waiting, the error says so: it names the control node, the address and port that ssh was dialing (on the ssm transport, that it was connecting through its ProxyCommand), and the wait. On the direct transport, when the cluster was not created with init --open, the most likely cause is a changed public IP. The cluster's security group allows SSH only from the IPs it lists, so after your public IP changes (a new network, a VPN, a new DHCP lease), ssh waits until its connect timeout and then fails.

To fix it, run this from the cluster's workspace:

easy-db-lab ingress allow

It adds an SSH rule for this machine's current public IP; give --cidr to add another address or range. It needs only AWS API access, not SSH. When you no longer need the rule for the old IP, remove it with easy-db-lab ingress revoke --cidr <old-ip>. See ingress allow and ingress revoke.

The log keeps the history of every tunnel in the workspace; nothing truncates it. Each launch adds a header line with a UTC timestamp, the port and the control node, and the tunnel script adds a timestamped line before each ssh run and each time ssh exits. Look for the last header to find the current tunnel.

easy-db-lab status is the one exception: it still reports everything it can reach over SSH and the AWS SDK even when the tunnel is down, marking only the section that requires the private Kubernetes API (stress jobs) as unavailable. See the status command reference for details.

Troubleshooting SOCKS Proxy

"no SOCKS tunnel is recorded for the workspace" or "the recorded SOCKS tunnel (PID ...) is not running":

easy-db-lab socks start    # Start the tunnel and record its port

"Connection refused" errors: The tunnel's ssh cannot reach the control node right now. The tunnel restarts ssh by itself; if that does not help, start a new tunnel:

socks5-status              # Shows the recorded port and whether anything listens on it
easy-db-lab socks start    # Starts a new tunnel and records its port
ssh control0 hostname      # Verify SSH works

Proxy not working after network change:

easy-db-lab socks stop
easy-db-lab socks start

Commands timing out:

1. Check cluster status: easy-db-lab status
2. Verify SSH works: ssh control0 hostname
3. Restart the tunnel: easy-db-lab socks stop && easy-db-lab socks start

easy-db-lab command fails with a SOCKS proxy error: As of this change, easy-db-lab commands that need the tunnel (up, kit commands, Grafana config updates, etc.) abort immediately if the tunnel can't be established, instead of silently running against a dead proxy port. Check socks5-proxy.log in your cluster workspace directory for the ssh -v transcript — it shows the actual reason the tunnel failed. See Host Key Verification above for the most common cause on a newly-provisioned cluster.

SSH over SSM Session Manager

Everything easy-db-lab does on a node travels over SSH: provisioning, the SOCKS proxy, ssh db0, and the c0 aliases. By default SSH connects straight to each node's public IP on port 22. Some networks don't allow that, typically a corporate egress proxy that only routes outbound traffic to destinations registered with it in advance. A freshly provisioned instance can never be pre-registered, so on those networks up times out waiting for SSH even though the instances are healthy.

The ssm SSH transport tunnels every SSH connection through AWS Systems Manager Session Manager. The SSM agent on each node connects out to AWS over HTTPS, so your machine never needs to reach a node on port 22. Nothing else about SSH changes: the keys, the generated sshConfig, and every command stay the same.

Requirements

• On your machine: the AWS CLI v2 and the Session Manager plugin.

  brew install awscli
  brew install --cask session-manager-plugin
  

  For other platforms, see the AWS CLI and Session Manager plugin install guides. up checks for both before it creates any AWS resource.

• IAM: your user or role must be allowed to open sessions. The EC2 policy printed by easy-db-lab show-iam-policies ec2 allows ssm:StartSession with the SSH and port-forwarding session documents, only to instances tagged easy_cass_lab=1 (every cluster node and AMI build instance is), and allows ssm:TerminateSession and ssm:ResumeSession on your own sessions only (matched on the session's aws:ssmmessages:session-id tag, which works for IAM users and SSO or assumed roles alike), plus ssmmessages:OpenDataChannel.

• On the cluster: nothing. Every node's instance role carries a small inline Session Manager policy (SessionManagerInstance), which lets the SSM agent register and carry sessions and grants nothing else. up, and the IAM check that AMI builds run first, add it to roles created by older versions. The node image already runs the SSM agent.

The SSM sessions authenticate as your profile does: through its named AWS profile (including SSO) when one is set, otherwise with its static keys. With static keys the AWS CLI is told to ignore your ~/.aws/config, so a [default] SSO or role setting there cannot replace them.

Enabling it

Run profile setup and answer ssm at the SSH transport prompt:

SSH transport (direct, ssm)? [direct] ssm

easy-db-lab profile show reports the current transport.

AMI builds

AMI builds (profile setup, build-image, build-base, build-cassandra) follow the same setting. Under ssm, Packer reaches its temporary build instance through Session Manager instead of its public IP.

Packer runs in a container, and the stock Packer image has no Session Manager plugin. The first ssm AMI build therefore builds a derived image, localhost/easy-db-lab/packer-ssm, which adds the plugin. Both the Packer base image and the plugin are pinned to fixed versions, and the plugin download is checked against a known SHA-256. Later builds reuse the image. Building it needs your container engine to reach the Alpine package mirror and AWS's plugin download at s3.amazonaws.com. AMI builds need nothing extra installed on your machine itself.

Troubleshooting SSM

ssh db0 fails with "Connection timed out during banner exchange": the Session Manager session opened but carried no data for 30 seconds (ConnectTimeout in the generated sshConfig). Run the command again; a new session usually works.

Each ProxyCommand in sshConfig runs through edl-ssm-proxy, a small wrapper in your profile directory. It ends the aws and session-manager-plugin processes when ssh exits or is killed, so none are left behind.

up stops with "these tools cannot run on this machine": each tool it lists says why. "not found on PATH" comes with the install command; install it, then run easy-db-lab up again. A tool that "exited" is installed but broken, and the line shows its own error output. A tool that "did not finish" hung on --version.

TargetNotConnected while up waits for SSH: a freshly booted node's SSM agent takes a short while to register with AWS. up keeps retrying, exactly as it does while sshd starts (a refused connection under the direct transport), and each "SSH still not up yet" line shows the error from the last attempt. If it never connects, check that the node can reach the internet over HTTPS.

AccessDeniedException mentioning ssm:StartSession: your IAM identity is missing the SSM permissions. Compare it against easy-db-lab show-iam-policies ec2.

Checking SSM outside easy-db-lab: take an instance ID from easy-db-lab status and open a port-forwarding session to it. The policy allows only the SSH and port-forwarding documents, so a plain shell session (no --document-name) is denied by design.

aws ssm start-session --target <instance-id> \
  --document-name AWS-StartPortForwardingSession \
  --parameters portNumber=22,localPortNumber=2222

It should print Waiting for connections....

If that also fails from your network, the proxy is blocking Session Manager itself (its data channel is a WebSocket to ssmmessages.<region>.amazonaws.com on port 443). Ask your network administrators to allow that endpoint.

Comparison

Shell Aliases

After running source env.sh, you get access to several helpful aliases and functions for managing your cluster.

SSH Aliases

SSH aliases for all Cassandra nodes are automatically created as c0-cN. The ssh command is not required. For example:

c0 nodetool status

This runs nodetool status on the first Cassandra node.

Cluster Management Functions

Examples

Run a command on all nodes

c-all "df -h"

Check cluster status

c-status

Collect artifacts for performance testing

c-collect-artifacts my-test-run

This is useful when doing performance testing to capture the state of the system at a given moment.

Graceful Rolling Restarts

For true rolling restarts, we recommend using cstar instead of c-restart.

Server

easy-db-lab includes a server mode that provides AI assistant integration via MCP (Model Context Protocol), REST status endpoints, and live metrics streaming. This enables Claude to directly interact with your clusters, and provides programmatic access to cluster status.

The server exposes tools for all supported databases — Cassandra, ClickHouse, OpenSearch, and Spark — as well as cluster lifecycle management and observability.

Starting the Server

To start the server, run:

easy-db-lab server

By default, the server picks an available port. To specify a port:

easy-db-lab server --port 8888

The server automatically generates a .mcp.json configuration file in the current directory with the connection details.

Adding to Claude Code

Once the server is running, start Claude Code from the same directory:

claude

Claude Code automatically detects and uses the .mcp.json file generated by the server.

Available Tools

The server exposes commands annotated with @McpCommand as MCP tools to Claude. Tool names use underscores and are derived from the command's package namespace.

Cluster Lifecycle

Cassandra Management

Cassandra Stress Testing

ClickHouse

OpenSearch

Spark

Kubernetes

Utilities

Tool Naming Convention

MCP tool names are derived from the command's package location:

• Top-level commands: status, hosts, ip, clean, init, up
• Cassandra commands: cassandra_ prefix (e.g., cassandra_start, cassandra_use)
• Nested commands: cassandra_stress_ prefix (e.g., cassandra_stress_start)
• Hyphens become underscores: update-config → cassandra_update_config

Benefits of Server Integration

Example Workflow

1. Start the server in one terminal:

   easy-db-lab server
   

2. In another terminal, start Claude Code from the same directory:

   claude
   

   Claude Code automatically detects the .mcp.json file generated by the server.

3. Ask Claude to help manage your cluster:

   • "Initialize a new 5-node cluster with i4i.xlarge instances"
   • "Check the status of all nodes"
   • "Select Cassandra version 5.0 and start it"
   • "Start a KeyValue stress test for 1 hour"
   • "Deploy ClickHouse and check its status"
   • "Create an OpenSearch domain and monitor its progress"
   • "Submit a Spark job to the EMR cluster"

Live Metrics Streaming

When Redis is configured via the EASY_DB_LAB_REDIS_URL environment variable, the server publishes live cluster metrics to the Redis pub/sub channel every 5 seconds. Metrics are queried from Mimir using the same PromQL expressions as the Grafana dashboards.

Enabling

export EASY_DB_LAB_REDIS_URL=redis://localhost:6379/easydblab-events
easy-db-lab server

Metrics events are published to the same channel as command events. Consumers filter by the event.type field.

Event Types

Only metrics for running services are published. If the cluster is running ClickHouse instead of Cassandra, no Cassandra metrics events are emitted.

Metrics.System

Published every 5 seconds with per-node CPU, memory, disk I/O, and filesystem metrics:

{
  "timestamp": "2026-03-08T14:22:05.123Z",
  "commandName": "server",
  "event": {
    "type": "Metrics.System",
    "nodes": {
      "db-0": {
        "cpuUsagePct": 34.2,
        "memoryUsedBytes": 17179869184,
        "diskReadBytesPerSec": 52428800.0,
        "diskWriteBytesPerSec": 104857600.0,
        "filesystemUsedPct": 45.2
      },
      "db-1": {
        "cpuUsagePct": 28.7,
        "memoryUsedBytes": 16106127360,
        "diskReadBytesPerSec": 41943040.0,
        "diskWriteBytesPerSec": 83886080.0,
        "filesystemUsedPct": 42.8
      }
    }
  }
}

Metrics.Cassandra

Published every 5 seconds when the cluster is running Cassandra:

{
  "timestamp": "2026-03-08T14:22:05.187Z",
  "commandName": "server",
  "event": {
    "type": "Metrics.Cassandra",
    "readP99Ms": 1.247,
    "writeP99Ms": 0.832,
    "readOpsPerSec": 15234.5,
    "writeOpsPerSec": 12087.3,
    "compactionPending": 3,
    "compactionCompletedPerSec": 1.5,
    "compactionBytesWrittenPerSec": 52428800.0
  }
}

Field Reference

System — per node:

Cassandra — cluster-wide:

Auto-Shutdown on Infrastructure Removal

When running the server in unattended or automated scenarios, you can enable automatic shutdown if the cluster's AWS infrastructure is torn down:

easy-db-lab server --auto-shutdown

When --auto-shutdown is set, the server checks whether the cluster VPC still exists on each status refresh cycle (controlled by --refresh). If the VPC is no longer found, the server emits a shutdown event and exits cleanly with code 0.

This is useful when:

• Running the server alongside an automated test workflow that tears down infrastructure when done
• Leaving the server running overnight and wanting it to stop automatically after easy-db-lab down

Note: The check is skipped if no cluster state exists or the VPC name cannot be determined. AWS API errors during the check are logged and ignored — only a confirmed "VPC not found" result triggers shutdown.

Notes

• The server requires Docker to be installed
• Your AWS profile must be configured (easy-db-lab profile setup)
• The server runs in the foreground and logs to stdout
• Use Ctrl+C to stop the server

Command Reference

Complete reference for all easy-db-lab commands.

Global Options

Profile Commands

Profile inspection and setup. Not to be confused with cassandra profile, which controls runtime profiling of a running cluster.

profile

Parent command for the profile group. Run without a subcommand to list what is available.

easy-db-lab profile

profile show

Report the active profile: its name, its directory, and the settings it holds.

easy-db-lab profile show

Reads only the profile directory, so it runs from anywhere — no cluster workspace needed. Secret values are never printed: AxonOps and Tailscale are reported as ENABLED or DISABLED, and the AWS access key and secret are not shown at all. The SSH transport (direct or ssm) is shown as-is.

The profile reported is the one named by EASY_DB_LAB_PROFILE, or default when that is unset:

EASY_DB_LAB_PROFILE=staging easy-db-lab profile show

If the profile has no settings.yaml, the command says so and points at easy-db-lab profile setup. If the file is present but unreadable, it reports that instead, naming the file.

profile setup

Set up user profile interactively.

easy-db-lab profile setup

Guides you through:

• Email and AWS credentials collection
• AWS credential validation
• SSH transport selection (direct or ssm; see SSH over SSM Session Manager)
• Key pair generation
• IAM role creation
• Packer VPC infrastructure setup
• AMI validation/building

show-iam-policies

Display IAM policies with your account ID populated.

easy-db-lab show-iam-policies [policy-name]

Aliases: sip

build-image

Build both base and Cassandra AMI images.

easy-db-lab build-image [options]

Cluster Lifecycle Commands

init

Initialize a directory for easy-db-lab.

easy-db-lab init [cluster-name] [options]

Use a new, empty directory as the workspace, for example under clusters/. init fails, and writes nothing, in a directory that already has a bin/ file or directory: easy-db-lab writes its tool wrappers into the workspace's bin/. This also keeps a source checkout from being used as a workspace. With --clean, the check runs after the cleanup, which removes a bin/ that holds only easy-db-lab's wrappers.

The database and application node groups are configured through a namespaced --db.* / --app.* scheme. Every pre-existing flag continues to work as an alias carrying its established default. When both a namespaced option and its legacy alias are supplied for the same setting, the namespaced option always wins, regardless of the order they appear on the command line.

Architecture is no longer a flag. Each node group's CPU architecture is derived automatically from that group's resolved instance type at init time (via the EC2 DescribeInstanceTypes SupportedArchitectures field) and persisted per group in cluster state. A cluster whose database and application groups have different architectures is provisioned correctly, each group booting from the AMI for its own architecture. An instance type whose architecture cannot be determined fails at init, before any instance is created.