easy-db-lab

easy-db-lab creates lab environments for database evaluations in AWS. It provisions infrastructure, deploys databases, and sets up a full observability stack so you can focus on testing, benchmarking, and learning.

Supported databases include Apache Cassandra, ClickHouse, TiDB, and OpenSearch, with Trino, Presto, and Apache Spark available for analytics workloads.

If you are looking for a tool to aid in stress testing Cassandra clusters, see the companion project cassandra-easy-stress.

If you're looking for tools to help manage Cassandra in production environments please see Reaper, cstar, and K8ssandra.

Quick Start

1. Install easy-db-lab
2. Set up your profile - Run easy-db-lab setup-profile
3. Follow the tutorial

Features

Database Support

• Apache Cassandra: Versions 3.0, 3.11, 4.0, 4.1, 5.0, and trunk builds. Includes custom build support, Cassandra Sidecar, and integration with cassandra-easy-stress for benchmarking.
• ClickHouse: Sharded clusters with configurable replication, distributed tables, and S3-tiered storage.
• TiDB: MySQL-compatible HTAP clusters deployed via the TiDB Operator, with TiKV row storage for OLTP and TiFlash columnar storage for analytics.
• Trino: Distributed SQL query engine with federation support for querying across data sources.
• Presto: Distributed SQL query engine for in-memory analytics.
• OpenSearch: AWS OpenSearch domains for search and analytics.
• Apache Spark: EMR-based Spark clusters for analytics workloads.

AWS Integration

• EC2 Provisioning: Automated provisioning with configurable instance types
• EBS Storage: Optional EBS volumes for persistent storage
• S3 Backup: Automatic backup of configurations and state to S3
• IAM Integration: Managed IAM policies for secure operations

Kubernetes (K3s)

• Lightweight K3s: Automatic K3s cluster deployment across all nodes
• kubectl/k9s: Pre-configured access with SOCKS5 proxy support
• Private Registry: HTTPS Docker registry for custom images
• Jib Integration: Push custom containers directly from Gradle

Monitoring and Observability

• VictoriaMetrics: Time-series database for metrics storage
• VictoriaLogs: Centralized log aggregation
• Tempo: Distributed trace storage with trace-to-metrics correlation
• Pyroscope: Continuous profiling of database and system processes
• Grafana: Pre-configured dashboards for Cassandra, ClickHouse, and system metrics
• OpenTelemetry: Distributed tracing and metrics collection
• eBPF Observability: Beyla for L7 network metrics and ebpf_exporter for low-level kernel metrics
• AxonOps: Optional integration with AxonOps for Cassandra monitoring and management

Developer Experience

• Shell Aliases: Convenient shortcuts for cluster management (c0, c-all, c-status, etc.)
• Server: Integration with Claude Code for AI-assisted operations
• Restore Support: Recover cluster state from VPC ID or S3 backup
• SOCKS5 Proxy: Secure access to private cluster resources

Stress Testing

• cassandra-easy-stress: Native integration with Apache stress testing tool
• Sysbench: OLTP benchmarks against any MySQL- or PostgreSQL-compatible kit, with live results in Grafana
• Kubernetes Jobs: Run stress tests as K8s jobs for scalability
• Artifact Collection: Automatic collection of metrics and diagnostics

Installation

Prerequisites

System Requirements

AWS Requirements

• AWS Account: A dedicated AWS account is recommended for lab environments
• AWS Credentials: Either static access key & secret, or a named profile — including an AWS SSO (IAM Identity Center) profile
• IAM Permissions: Permissions to create EC2, IAM, S3, and optionally EMR resources

Optional

• AxonOps Account: For free Cassandra monitoring. Create an account at axonops.com

Install from Release

Download a tarball from the releases page and add the bin directory to your $PATH:

export PATH="$PATH:/path/to/easy-db-lab/bin"

Build from Source

git clone https://github.com/rustyrazorblade/easy-db-lab.git
cd easy-db-lab
./gradlew assemble

The built distribution will be in build/distributions/.

Next Steps

Run the interactive setup to configure your profile:

easy-db-lab setup-profile

See the Setup Guide for detailed instructions.

Setup

This guide walks you through the initial setup of easy-db-lab, including AWS credentials configuration, IAM policies, and AMI creation.

Overview

The setup-profile command handles all initial configuration interactively. It will:

1. Collect your email and AWS credentials
2. Validate your AWS access
3. Create necessary AWS resources (key pair, IAM roles, Packer VPC)
4. Build or validate the required AMI

Prerequisites

Before running setup:

• AWS Account: An AWS account with appropriate permissions (see IAM Policies below)
• Java 21+: Required to run easy-db-lab
• Docker: Required only if building custom AMIs

Step 1: Run Setup Profile

Run the interactive setup:

easy-db-lab setup-profile

Or use the shorter alias:

easy-db-lab setup

The setup wizard will prompt you for:

setup-profile validates your credentials against AWS immediately (and then provisions resources), so your credentials must be usable before you run it. For static keys this is automatic; for an SSO profile, run aws sso login first (see below).

What Gets Created

During setup, the following AWS resources are created:

• EC2 Key Pair: For SSH access to instances
• IAM Role: For instance permissions (easy-db-lab-instance-role)
• Packer VPC: Infrastructure for building AMIs
• AMI (if needed): Takes 10-15 minutes to build

Configuration Location

Your profile is saved to:

~/.easy-db-lab/profiles/default/settings.yaml

Using AWS SSO (IAM Identity Center)

If your AWS access is provisioned through AWS SSO (IAM Identity Center) rather than long-lived access keys, easy-db-lab authenticates through a named AWS profile backed by an SSO session. You do not copy temporary credentials out of the AWS access portal — the tool resolves them automatically from your SSO login.

1. Define an SSO profile in ~/.aws/config:

[sso-session my-sso]
sso_start_url = https://my-company.awsapps.com/start
sso_region = us-east-1
sso_registration_scopes = sso:account:access

[profile edl]
sso_session = my-sso
sso_account_id = 123456789012
sso_role_name = YourRoleName
region = us-west-2

2. Log in to start an SSO session (this opens a browser):

aws sso login --profile edl

3. Run setup-profile and enter edl when prompted for the AWS profile name (leave the access key and secret blank):

easy-db-lab setup-profile

Day-to-day usage

After the one-time setup, you do not sign in for every command. An aws sso login session lasts for hours (your organization sets the exact window), and easy-db-lab resolves and refreshes credentials from it automatically:

# Once per work session (e.g. each morning), or whenever the session expires:
aws sso login --profile edl

# Then run commands freely — no per-command authentication:
easy-db-lab up
easy-db-lab cassandra start
easy-db-lab down --auto-approve

When the session expires, the next command fails with an authentication error directing you to log in again. Re-run aws sso login --profile edl and continue.

Step 2: Getting IAM Policies

If you need to request permissions from your AWS administrator, use the show-iam-policies command to display the required policies with your account ID populated:

easy-db-lab show-iam-policies

This displays three policies:

Filter by Policy Name

To show a specific policy:

easy-db-lab show-iam-policies ec2    # Show EC2 policy only
easy-db-lab show-iam-policies iam    # Show IAM policy only
easy-db-lab show-iam-policies emr    # Show EMR policy only

Recommended IAM Setup

For teams with multiple users, we recommend creating managed policies attached to an IAM group:

1. Create an IAM group (e.g., "EasyDBLabUsers")
2. Create three managed policies from the JSON output
3. Attach all policies to the group
4. Add users to the group

Step 3: Build Custom AMI (Optional)

If setup couldn't find a valid AMI for your architecture, or if you want to customize the base image, build one manually:

easy-db-lab build-image

Build Options

Examples

# Build AMD64 AMI (default)
easy-db-lab build-image

# Build ARM64 AMI for Graviton instances
easy-db-lab build-image --arch ARM64

# Build in specific region
easy-db-lab build-image --region eu-west-1

Environment Variables

Verify Installation

After setup completes, verify by running:

easy-db-lab

You should see the help output with available commands.

Next Steps

Once setup is complete, follow the Tutorial to create your first cluster.

Tutorial: Getting Started

This tutorial walks you through creating a database cluster from scratch, covering initialization, infrastructure provisioning, and database configuration. The examples below use Cassandra, but the same infrastructure supports ClickHouse, OpenSearch, and Spark.

Part 1: Initialize Your Cluster

The init command creates local configuration files for your cluster. It does not provision AWS resources yet.

easy-db-lab init my-cluster

This creates a 3-node Cassandra cluster by default.

Init Options

The database and application node groups use a namespaced --db.* / --app.* scheme. Every older flag still works as an alias with its established default; when both a namespaced option and its legacy alias are given for the same setting, the namespaced option wins, regardless of order.

Examples

Basic 3-node cluster:

easy-db-lab init my-cluster

5-node cluster with 2 stress nodes:

easy-db-lab init my-cluster --db 5 --stress 2

Production-like cluster with EBS storage:

easy-db-lab init prod-test --db 5 --ebs.type gp3 --ebs.size 500 --ebs.iops 3000

ARM64 cluster for Graviton instances: The architecture is derived automatically from the instance type — no flag needed.

easy-db-lab init my-cluster --db.instance-type r7g.2xlarge

Mixed-architecture cluster (arm64 database, x86_64 application):

easy-db-lab init my-cluster --db.instance-type r7g.2xlarge --app.instance-type c6id.2xlarge

Initialize and provision in one step:

easy-db-lab init my-cluster --up

Storage Requirements

Database instances need a data disk separate from the root volume. This can come from either:

• Instance store (local NVMe) — Instance types with a d suffix (e.g., i3.xlarge, m5d.xlarge, c5d.2xlarge) include local NVMe storage and require no extra configuration.
• EBS volumes — Attach an EBS volume using --ebs.type for any other instance type.

If the selected instance type has no instance store and --ebs.type is not specified, up will fail. For example:

easy-db-lab init my-cluster --db.instance-type c5.2xlarge --ebs.type gp3 --ebs.size 200

Part 2: Launch Infrastructure

Once initialized, provision the AWS infrastructure:

easy-db-lab up

This command creates:

• S3 Storage: Cluster data stored under a dedicated prefix in the account S3 bucket
• VPC: With subnets and security groups
• EC2 Instances: Cassandra nodes, stress nodes, and a control node (an m5d.xlarge; K3s data is stored on its NVMe drive to avoid filling the root volume)
• K3s Cluster: Lightweight Kubernetes across all nodes

What Happens During up

1. Configures account S3 bucket with cluster prefix
2. Creates VPC with public subnets in your availability zones
3. Provisions EC2 instances in parallel
4. Waits for SSH availability
5. Configures K3s cluster on all nodes
6. Writes SSH config and environment files

Up Options

Environment Setup

After up completes, source the environment file:

source env.sh

This configures your shell with:

• SSH shortcuts: ssh db0, ssh db1, ssh stress0, etc.
• Cluster aliases: c0, c-all, c-status
• SOCKS proxy configuration

See Shell Aliases for all available shortcuts.

Part 3: Configure Cassandra 5.0

With infrastructure running, configure and start Cassandra.

Step 1: Select Cassandra Version

easy-db-lab cassandra use 5.0

This command:

• Sets the active Cassandra version on all nodes
• Downloads configuration files to your local directory
• Applies any existing patch configuration

Available versions: 3.0, 3.11, 4.0, 4.1, 5.0, 5.0-HEAD, 6.0-HEAD, trunk

Step 2: Customize Configuration (Optional)

Edit cassandra.patch.yaml to customize settings:

# Example: Change token count
vim cassandra.patch.yaml

Common customizations:

Step 3: Apply Configuration

easy-db-lab cassandra update-config

This uploads and applies the patch to all Cassandra nodes.

To apply and restart Cassandra in one command:

easy-db-lab cassandra update-config --restart

Step 4: Start Cassandra

easy-db-lab cassandra start

Step 5: Verify Cluster

Check cluster status:

ssh db0 nodetool status

Or use the shell alias (after sourcing env.sh):

c-status

You should see all nodes in UN (Up/Normal) state.

Part 4: Working with Your Cluster

SSH Access

After sourcing env.sh:

ssh db0          # First Cassandra node
ssh db1          # Second Cassandra node
ssh stress0      # First stress node (if provisioned)
ssh control0     # Control node

Cassandra Management

# Stop Cassandra on all nodes
easy-db-lab cassandra stop

# Start Cassandra on all nodes
easy-db-lab cassandra start

# Restart Cassandra on all nodes
easy-db-lab cassandra restart

Filter to Specific Hosts

Most commands support the --hosts filter:

# Apply config only to db0 and db1
easy-db-lab cassandra update-config --hosts db0,db1

# Restart only db2
easy-db-lab cassandra restart --hosts db2

Download Configuration Files

To download the current configuration from nodes:

easy-db-lab cassandra download-config

This saves configuration files to a local directory named after the version (e.g., 5.0/).

Part 5: Shut Down

When finished, destroy the cluster infrastructure:

easy-db-lab down

Quick Reference

Next Steps

• Kubernetes Access - Access K3s cluster with kubectl and k9s
• Shell Aliases - All available CLI shortcuts
• ClickHouse - Deploy ClickHouse for analytics
• Spark - Set up Apache Spark via EMR

Legacy Integrations

The following databases have native integrations that predate the Kits system. They are fully functional but will eventually be migrated to kits.

• Cassandra
• OpenSearch
• Spark

Configuring Cassandra

This page covers Cassandra version management and configuration. For a step-by-step walkthrough, see the Tutorial.

Supported Versions

easy-db-lab supports the following Cassandra versions:

Quick Start

# Select Cassandra 5.0
easy-db-lab cassandra use 5.0

# Generate configuration patch
easy-db-lab cassandra write-config

# Apply configuration and start
easy-db-lab cassandra update-config
easy-db-lab cassandra start

# Verify cluster
ssh db0 nodetool status

Version Management

Select a Version

easy-db-lab cassandra use <version>

Examples:

easy-db-lab cassandra use 5.0       # Latest stable
easy-db-lab cassandra use 4.1       # LTS version
easy-db-lab cassandra use trunk     # Development branch

This command:

1. Sets the active Cassandra version on all nodes
2. Downloads current configuration files locally
3. Applies any existing cassandra.patch.yaml

Specify Java Version

easy-db-lab cassandra use 5.0 --java 11

List Available Versions

easy-db-lab ls

Configuration

The Patch File

Cassandra configuration uses a patch file approach. The cassandra.patch.yaml file contains only the settings you want to customize, which are merged with the default cassandra.yaml.

Generate a new patch file:

easy-db-lab cassandra write-config

Options:

• -t, --tokens: Number of tokens (default: 4)

Example patch file:

cluster_name: "my-cluster"
num_tokens: 4
concurrent_reads: 64
concurrent_writes: 64
trickle_fsync: true

Apply Configuration

easy-db-lab cassandra update-config

Options:

• --restart, -r: Restart Cassandra after applying
• --hosts: Filter to specific hosts

Apply and restart in one command:

easy-db-lab cassandra update-config --restart

Download Configuration

Download current configuration files from nodes:

easy-db-lab cassandra download-config

Files are saved to a local directory named after the version (e.g., 5.0/).

Starting and Stopping

# Start on all nodes
easy-db-lab cassandra start

# Stop on all nodes
easy-db-lab cassandra stop

# Restart on all nodes
easy-db-lab cassandra restart

# Target specific hosts
easy-db-lab cassandra start --hosts db0,db1

Cassandra Sidecar

The Apache Cassandra Sidecar is automatically installed and started alongside Cassandra. The sidecar provides:

• REST API for Cassandra operations
• S3 import/restore capabilities
• Streaming data operations
• Metrics collection (Prometheus-compatible)

Sidecar Access

The sidecar runs on port 9043 on each Cassandra node:

# Check sidecar health
curl http://<cassandra-node-ip>:9043/api/v1/__health

Sidecar Management

The sidecar is managed via systemd:

# Check status
ssh db0 sudo systemctl status cassandra-sidecar

# Restart
ssh db0 sudo systemctl restart cassandra-sidecar

Sidecar Configuration

Configuration is located at /etc/cassandra-sidecar/cassandra-sidecar.yaml on each node. Key settings:

• Cassandra connection details
• Data directory paths
• Traffic shaping and throttling
• S3 integration settings

Custom Builds

To run a custom Cassandra build (your own fork, a feature branch, or a prebuilt tarball), add a version entry and rebuild the AMI. easy-db-lab bakes every listed version into the image — there is no separate build-from-path command.

You don't edit the repository's cassandra_versions.yaml. Instead, drop one or more YAML files into your profile's extras directory:

~/.easy-db-lab/profiles/<profile>/cassandra_versions/

(The default profile is default.) At build time these are merged with the built-in versions. Each version must be unique across the built-in list and your extras, or the build fails.

1. Add a version entry

Create e.g. ~/.easy-db-lab/profiles/default/cassandra_versions/my-build.yaml.

Build from a git branch (cloned and compiled with ant during the AMI build):

- version: "my-build"
  java: "11"
  python: "3.10.6"
  url: "https://github.com/myuser/cassandra.git"
  branch: "my-feature-branch"
  ant_flags: "-Duse.jdk11=true"   # optional, passed to ant

Or install a prebuilt tarball:

- version: "my-build"
  java: "11"
  python: "3.10.6"
  url: "https://example.com/apache-cassandra-my-build-bin.tar.gz"

An entry with no url/branch downloads the matching official Apache release.

2. Rebuild the Cassandra AMI

easy-db-lab build-cassandra

3. Select the custom build

easy-db-lab cassandra use my-build

Next Steps

• Tutorial - Complete walkthrough
• Shell Aliases - Convenient shortcuts for Cassandra management

OpenSearch

AWS OpenSearch can be provisioned as a managed domain for full-text search and log analytics.

Commands

Starting OpenSearch

easy-db-lab opensearch start

This creates an AWS-managed OpenSearch domain linked to your cluster's VPC. The domain takes several minutes to provision.

Checking Status

easy-db-lab opensearch status

Stopping OpenSearch

easy-db-lab opensearch stop

This deletes the OpenSearch domain. Data stored in the domain will be lost.

Spark

easy-db-lab supports provisioning Apache Spark clusters via AWS EMR for analytics workloads.

Enabling Spark

There are two ways to enable Spark:

Option 1: During Init (before up)

Enable Spark during cluster initialization with the --spark.enable flag. The EMR cluster will be created automatically when you run up:

easy-db-lab init --spark.enable
easy-db-lab up

Init Spark Configuration Options

Example with Custom Configuration

easy-db-lab init \
  --spark.enable \
  --spark.master.instance.type m5.2xlarge \
  --spark.worker.instance.type m5.4xlarge \
  --spark.worker.instance.count 5

Option 2: After up (standalone spark init)

Add Spark to an existing environment that is already running. This is useful when you forgot to pass --spark.enable during init, or when you decide to add Spark later:

easy-db-lab spark init

Prerequisites: easy-db-lab init and easy-db-lab up must have been run first.

Spark Init Configuration Options

Example with Custom Configuration

easy-db-lab spark init \
  --master.instance.type m5.2xlarge \
  --worker.instance.type m5.4xlarge \
  --worker.instance.count 5

Submitting Spark Jobs

Submit JAR-based Spark applications to your EMR cluster:

easy-db-lab spark submit \
  --jar /path/to/your-app.jar \
  --main-class com.example.YourMainClass \
  --conf spark.easydblab.keyspace=my_keyspace \
  --conf spark.easydblab.table=my_table \
  --wait

Submit Options

When --jar is a local path, it is automatically uploaded to the cluster's S3 bucket before submission. When it is an s3:// URI, it is used directly.

Using a JAR Already on S3

If your JAR is already on S3 (e.g., from a CI pipeline or a previous upload), pass the S3 URI directly:

easy-db-lab spark submit \
  --jar s3://my-bucket/jars/your-app.jar \
  --main-class com.example.YourMainClass \
  --conf spark.easydblab.keyspace=my_keyspace \
  --wait

This skips the upload step entirely, which is useful for large JARs or when resubmitting the same job.

Cancelling a Job

Cancel a running or pending Spark job without terminating the cluster:

easy-db-lab spark stop

Without --step-id, this cancels the most recent job. To cancel a specific job:

easy-db-lab spark stop --step-id <step-id>

The cancellation uses EMR's TERMINATE_PROCESS strategy (SIGKILL). The API is asynchronous — use spark status to confirm the job has been cancelled.

Checking Job Status

View Recent Jobs

List recent Spark jobs on the cluster:

easy-db-lab spark jobs

Options:

• --limit - Maximum number of jobs to display (default: 10)

Check Specific Job Status

easy-db-lab spark status --step-id <step-id>

Without --step-id, shows the status of the most recent job.

Options:

• --step-id - EMR step ID to check
• --logs - Download step logs (stdout, stderr)

Retrieving Logs

Download logs for a Spark job:

easy-db-lab spark logs --step-id <step-id>

Logs are automatically decompressed and include:

• stdout.gz - Standard output
• stderr.gz - Standard error
• controller.gz - EMR controller logs

Architecture

When Spark is enabled, easy-db-lab provisions:

• EMR Cluster: Managed Spark cluster with master and worker nodes
• S3 Integration: Logs stored at s3://<bucket>/spark/emr-logs/
• IAM Roles: Service and job flow roles for EMR operations
• Observability: Each EMR node runs an OTel Collector (host metrics, OTLP forwarding), OTel Java Agent (auto-instrumentation for logs/metrics/traces), and Pyroscope Java Agent (continuous CPU/allocation/lock profiling). All telemetry flows to the control node's observability stack.

Timeouts and Polling

• Job Polling Interval: 5 seconds
• Maximum Wait Time: 4 hours
• Cluster Creation Timeout: 30 minutes

Spark with Cassandra

A common use case is running Spark jobs that read from or write to Cassandra. Use the Spark Cassandra Connector:

import com.datastax.spark.connector._

val df = spark.read
  .format("org.apache.spark.sql.cassandra")
  .options(Map("table" -> "my_table", "keyspace" -> "my_keyspace"))
  .load()

Ensure your JAR includes the Spark Cassandra Connector dependency and configure the Cassandra host in your Spark application.

Bulk Write Implementations

easy-db-lab provides three different implementations for bulk writing data to Cassandra, each with different characteristics. All three use the same configuration properties (spark.easydblab.*) so you can easily compare performance by just swapping the JAR and main class.

Implementation Comparison

Direct Bulk Writer (Sidecar Transport)

Streams SSTables directly from Spark to Cassandra nodes via the Sidecar REST API on port 9043.

How it works:

1. Spark generates SSTables from source data
2. SSTables are streamed directly to Sidecar endpoints
3. Sidecar validates and imports SSTables into Cassandra

When to use:

• Direct network connectivity between Spark and Cassandra
• Lower latency requirements
• Smaller to medium datasets

Limitations:

• Requires network connectivity from EMR to Cassandra on port 9043
• Streaming backpressure if Sidecar can't keep up

S3 Bulk Writer (S3 Staging Transport)

Stages SSTables in S3, then notifies Cassandra Sidecar to download and import them.

How it works:

1. Spark generates SSTables and bundles them into ZIPs with manifests
2. Bundles are uploaded to S3 bucket (provided by easy-db-lab)
3. Spark pushes import notification to Sidecar REST API
4. Sidecar downloads bundles from S3, validates checksums, filters by token ranges
5. Sidecar imports SSTables into Cassandra

When to use:

• Large-scale bulk loads (terabytes)
• S3 provides durability and staging for retries
• Multi-DC Cassandra clusters

Prerequisites:

• S3 bucket (automatically provisioned by easy-db-lab as clusterState.dataBucket)
• EMR instance profile with S3 write permissions (automatically configured)
• Cassandra Sidecar running on port 9043
• For Cassandra 5.x: storage_compatibility_mode: NONE in cassandra.yaml (the bulk writer requires the new SSTable format)

Credentials:

• EMR uses instance profile (IMDS) to write to S3 - no manual credential configuration needed
• AWS region is auto-detected from EC2 metadata
• Sidecar uses its IAM role to read from S3

Benefits:

• S3 provides durability for large datasets
• Token range filtering ensures data goes to correct nodes
• Checksum validation guarantees data integrity

Standard Connector Writer

Uses the DataStax Spark Cassandra Connector to write data via CQL.

How it works:

1. Spark generates rows as DataFrames
2. Connector batches writes and sends via Cassandra native protocol (port 9042)
3. Cassandra processes writes through normal write path (memtables → SSTables)

When to use:

• Smaller datasets
• Need CDC, triggers, or other write-time features
• Existing Spark Cassandra Connector pipelines

Limitations:

• Significantly slower than bulk writers for large datasets (goes through full write path)
• Compaction overhead after writes complete
• More network round-trips

Spark Modules

The Spark job modules live in the spark-examples repository and share unified configuration via spark.easydblab.* properties. You can compare performance across implementations by swapping the JAR and main class while keeping the same --conf flags.

Module Overview

The Gradle paths above are within the spark-examples build, not this repo.

Getting the job JARs

The job modules live in their own repository, spark-examples, which publishes the shadow (fat) JARs as GitHub Release assets. Download the one you need and pass it to spark submit with --jar:

# Download a published job jar (check the releases page for the latest version)
curl -L -O https://github.com/rustyrazorblade/spark-examples/releases/download/v0.1.0/bulk-writer-sidecar.jar

Available jars: bulk-writer-sidecar.jar, bulk-writer-s3-iam-all.jar, connector-writer.jar, connector-read-write.jar. Releases: https://github.com/rustyrazorblade/spark-examples/releases

Usage

All modules use the same --conf properties for easy comparison.

Direct Bulk Writer (Sidecar)

Streams SSTables directly to Cassandra Sidecar endpoints:

easy-db-lab spark submit \
  --jar bulk-writer-sidecar.jar \
  --main-class com.rustyrazorblade.easydblab.spark.DirectBulkWriter \
  --conf spark.easydblab.contactPoints=host1:9043,host2:9043,host3:9043 \
  --conf spark.easydblab.keyspace=bulk_test \
  --conf spark.easydblab.localDc=us-west-2 \
  --conf spark.easydblab.rowCount=1000000 \
  --wait

Note: Contact points should include the Sidecar port (9043).

IAM S3 Bulk Writer

Stages SSTables in S3 using IAM instance profile credentials, then imports via Sidecar. Cluster topology is auto-discovered from the Cassandra driver — no manual DC configuration needed:

easy-db-lab spark submit \
  --jar bulk-writer-s3-iam-all.jar \
  --main-class com.rustyrazorblade.easydblab.spark.IamBulkWriter \
  --conf spark.easydblab.contactPoints=host1,host2,host3 \
  --conf spark.easydblab.localDc=us-west-2 \
  --conf spark.easydblab.s3.bucket=my-bucket \
  --conf spark.easydblab.rowCount=10000000 \
  --conf spark.easydblab.parallelism=20 \
  --wait

Credentials: Both the EMR executor and the Cassandra Sidecar authenticate independently via their attached IAM roles. No credentials are extracted or transmitted.

Standard Connector Writer

Standard CQL writes via Cassandra native protocol:

easy-db-lab spark submit \
  --jar connector-writer.jar \
  --main-class com.rustyrazorblade.easydblab.spark.StandardConnectorWriter \
  --conf spark.easydblab.contactPoints=host1,host2,host3 \
  --conf spark.easydblab.keyspace=bulk_test \
  --conf spark.easydblab.localDc=us-west-2 \
  --conf spark.easydblab.rowCount=1000000 \
  --wait

Note: Contact points for the connector use the Cassandra native protocol port (9042), not Sidecar.

Convenience Script

The bin/spark-bulk-write script handles JAR lookup, host resolution, and health checks:

# From a cluster directory
spark-bulk-write direct --rows 10000
spark-bulk-write s3 --rows 1000000 --parallelism 20
spark-bulk-write connector --keyspace myks --table mytable

Configuration Properties

All modules share these properties via spark.easydblab.*:

Table Schema

The test data generators produce this schema:

CREATE TABLE <keyspace>.<table> (
    partition_id bigint,
    sequence_id bigint,
    course blob,
    marks bigint,
    PRIMARY KEY ((partition_id), sequence_id)
);

Troubleshooting

S3 Bulk Writer Issues

"Required property not set: spark.easydblab.s3.bucket"

The S3 bulk writer requires an S3 bucket for staging SSTables. Get the bucket from cluster state:

jq -r '.dataBucket' state.json

Then add it to your submit command:

--conf spark.easydblab.s3.bucket=<bucket-from-state>

"Failed to resolve AWS credentials"

The EMR instance profile should provide credentials automatically via IMDS. If this fails:

1. Verify EMR cluster has EasyDBLabEMREC2Role instance profile attached
2. Check IAM role has s3:* permissions on the data bucket
3. Verify instance metadata service (IMDS) is accessible from EMR nodes

"Unable to detect AWS region"

Region is auto-detected from EC2 metadata. This should work automatically on EMR. If it fails, the EMR cluster may not have proper metadata access.

"S3 bucket name must be between 3 and 63 characters"

Bucket names must follow AWS S3 naming rules (3-63 characters, lowercase, DNS-compliant). Verify the bucket name in state.json.

Job succeeds but no data imported

Check Cassandra 5.x compatibility:

1. Verify storage_compatibility_mode: NONE in cassandra.yaml
2. Cassandra 5.x defaults to UPGRADING mode which uses legacy SSTable format
3. The bulk writer requires NONE to use the new SSTable format

To fix:

echo "storage_compatibility_mode: NONE" >> cassandra.patch.yaml
easy-db-lab cassandra update-config
easy-db-lab cassandra restart

Bundles uploaded but Sidecar didn't import

1. Check Sidecar is running: curl http://<cassandra-host>:9043/api/v1/health
2. Verify Sidecar has S3 read permissions (IAM role)
3. Check Sidecar logs for download or validation errors
4. Verify token ranges match between SSTables and Cassandra ring

Direct Bulk Writer Issues

"Connection refused" to port 9043

1. Verify Cassandra Sidecar is running on all nodes
2. Check security groups allow EMR → Cassandra on port 9043
3. Ensure contact points use correct IP addresses (private IPs if in same VPC)

Slow writes / backpressure

The direct transport streams data and can be throttled if Sidecar can't keep up. Consider:

1. Reduce spark.easydblab.parallelism to lower write rate
2. Use S3 staging transport for large datasets
3. Check Cassandra disk I/O and compaction status

Connector Writer Issues

"No route to host" or timeout

1. Check security groups allow EMR → Cassandra on port 9042
2. Verify contact points are reachable from EMR
3. Ensure Cassandra native protocol is enabled

Slow performance

The connector uses the standard write path (not bulk write). For large datasets, use a bulk writer instead.

Kits

A kit is a self-contained package of configuration and scripts that installs, starts, stops, and optionally backs up a workload on your cluster. Each kit defines its full lifecycle in a kit.yaml file using typed steps — no Kubernetes YAML wrangling required.

easy-db-lab ships with built-in kits (ClickHouse, Presto, Trino, TiDB, sysbench). You can also create your own kits for any workload you want to benchmark or test.

Discovering kits

List all available kits:

easy-db-lab kit list

Inspect a kit before installing it — see its args, endpoints, and available commands:

easy-db-lab kit info clickhouse

Installing a kit

easy-db-lab kit install clickhouse --clickhouse-version 25.4 --size 100Gi

Args vary by kit. Run kit info <name> to see what a kit accepts, or pass --help:

easy-db-lab kit install clickhouse --help

After install, the kit's files are written into a subdirectory of the cluster workspace. The kit's lifecycle commands are registered automatically.

Bench kits — benchmarking a database

Bench kits are a special class of kit that run against an already-running database kit. They require a --target flag pointing at the installed database kit you want to benchmark.

# Install sysbench targeting your running TiDB instance
easy-db-lab kit install sysbench --target tidb

# Run the prepare, start, and stop lifecycle as usual
easy-db-lab sysbench-tidb prepare
easy-db-lab sysbench-tidb start
easy-db-lab sysbench-tidb stop

The kit is installed into a directory named <bench-kit>-<target> (e.g. sysbench-tidb). This lets you run the same bench kit against multiple databases simultaneously and compare results:

easy-db-lab kit install sysbench --target tidb
easy-db-lab kit install sysbench --target my-custom-db

# Both run at the same time — compare results in Grafana
easy-db-lab sysbench-tidb start
easy-db-lab sysbench-my-custom-db start

The target must expose a wire protocol endpoint the bench tool can speak — sysbench supports MySQL and PostgreSQL. See Sysbench for the full lifecycle, flags, and metrics.

TARGET_* environment variables

When a bench kit starts, easy-db-lab reads the target database's endpoint configuration and injects it as environment variables into every phase script:

Which variables are populated depends on what endpoints the target kit declares. A kit that supports both JDBC and PostgreSQL wire protocol will populate both sets.

Running kit commands

Every installed kit gains a set of subcommands:

easy-db-lab clickhouse start       # deploy and start the workload
easy-db-lab clickhouse status      # show running state and connection endpoints
easy-db-lab clickhouse stop        # stop and remove the workload
easy-db-lab clickhouse backup --name my-backup   # back up data
easy-db-lab clickhouse restore --name my-backup  # restore from backup
easy-db-lab clickhouse uninstall   # stop and remove all kit resources

Installing a custom kit

Place your kit directory under the profile kits folder and it will appear in kit list and be installable by name like any built-in kit:

~/.easy-db-lab/profiles/default/kits/<kit-name>/

Custom kits in the profile directory take precedence over built-in kits with the same name.

mkdir -p ~/.easy-db-lab/profiles/default/kits/my-kit
cp -r /path/to/my-kit/* ~/.easy-db-lab/profiles/default/kits/my-kit/

# Now it appears in kit list and can be installed by name:
easy-db-lab kit install my-kit

Using kits from external projects

If you keep kit definitions alongside a private project (a POC, internal tooling, etc.), you can register that project's kits directory without copying files into your profile.

A typical project structure looks like this:

myapp/
├── src/
├── kits/
│   └── myapp-workload/
│       ├── kit.yaml
│       └── bin/
│           ├── start.sh
│           └── stop.sh
└── README.md

Clone your project and register the kits directory by name:

git clone https://github.com/myorg/myapp ~/myapp
easy-db-lab kit source add myapp ~/myapp/kits

The kits it contains now appear in kit list and can be installed by name:

easy-db-lab kit list
easy-db-lab kit install myapp-workload

Registered sources are persisted in ~/.easy-db-lab/profiles/<profile>/kit-sources.yaml and survive CLI restarts. When you kit install a kit from an external source, its files are copied into the cluster workspace exactly like any other kit — the installed kit is self-contained.

Managing registered sources

# List all registered sources (shows name and path, flags missing paths)
easy-db-lab kit source list

Output looks like:

Registered kit sources:
  myapp  /Users/jon/myapp/kits

If a registered path no longer exists on disk, [missing] appears next to it so you know which sources need attention.

# Remove a source by name
easy-db-lab kit source remove myproject

Updating a path (upsert behavior): Sources are identified by name. If you move or reclone a project to a different location, just re-add the source with the new path — no need to remove the old registration first:

# If you move or reclone the project, just update the path — no need to remove first
easy-db-lab kit source add myapp ~/new-location/myapp/kits
# Updated kit source 'myapp': /new-location/myapp/kits

Resolution priority

When multiple sources provide a kit with the same name, the first match wins:

1. Profile kits directory (~/.easy-db-lab/profiles/<profile>/kits/)
2. Registered additional sources (in registration order)
3. Built-in kits

For a full walkthrough of building and publishing your own kit, see the Kit Development guide.

ClickHouse

easy-db-lab supports deploying ClickHouse clusters on Kubernetes for analytics workloads alongside your Cassandra cluster.

Overview

ClickHouse is deployed as a StatefulSet on K3s with ClickHouse Keeper for distributed coordination. The deployment requires a minimum of 3 nodes.

Quick Start

Create a 6-node cluster and deploy ClickHouse with 2 shards:

# Initialize and start a 6-node cluster
easy-db-lab init my-cluster --db 6 --up

# Deploy ClickHouse (2 shards x 3 replicas)
easy-db-lab clickhouse start

Configuring ClickHouse

Use clickhouse init to configure ClickHouse settings before starting the cluster:

# Configure S3 cache size (default: 10Gi)
easy-db-lab clickhouse init --s3-cache 50Gi

# Disable write-through caching
easy-db-lab clickhouse init --s3-cache-on-write false

Configuration is saved to the cluster state and applied when you run clickhouse start.

Starting ClickHouse

To deploy ClickHouse on an existing cluster:

easy-db-lab clickhouse start

Options

Example with Custom Settings

# 6 nodes with 3 replicas per shard = 2 shards
easy-db-lab clickhouse start --replicas 6 --replicas-per-shard 3

# 9 nodes with 3 replicas per shard = 3 shards
easy-db-lab clickhouse start --replicas 9 --replicas-per-shard 3

Cluster Topology

ClickHouse is deployed with a sharded, replicated architecture. The total number of replicas must be divisible by --replicas-per-shard.

Shard and Replica Assignment

The cluster named easy_db_lab is automatically configured based on your replica count:

Pod-to-Node Pinning

Each ClickHouse pod is pinned to a specific database node using Local PersistentVolumes with node affinity:

• clickhouse-0 always runs on db0
• clickhouse-1 always runs on db1
• clickhouse-N always runs on dbN

This guarantees:

1. Consistent shard assignment - A pod's shard is calculated from its ordinal: shard = (ordinal / replicas_per_shard) + 1
2. Data locality - Data stored on a node stays with that node across pod restarts
3. Predictable performance - No data movement when pods restart

Shard Calculation Example

With 6 replicas and 3 replicas per shard:

Checking Status

To check the status of your ClickHouse cluster:

easy-db-lab clickhouse status

This displays:

• Pod status and health
• Access URLs for the Play UI and HTTP interface
• Native protocol connection details

Accessing ClickHouse

After deployment, ClickHouse is accessible via:

The MySQL and PostgreSQL interfaces are protocol-compatible, not dialect-compatible: queries sent over them are parsed as ClickHouse SQL. They are handy for connecting standard clients and drivers, but tools that emit MySQL- or PostgreSQL-specific DDL will not work unmodified.

Creating Tables

ClickHouse supports distributed, replicated tables that span multiple shards. The recommended pattern uses ReplicatedMergeTree for local replicated storage and Distributed for querying across shards.

Distributed Replicated Tables

Create a local replicated table on all nodes, then a distributed table for queries:

-- Step 1: Create local replicated table on all nodes
CREATE TABLE events_local ON CLUSTER easy_db_lab (
    id UInt64,
    timestamp DateTime,
    event_type String,
    data String
) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/events', '{replica}')
ORDER BY (timestamp, id)
SETTINGS storage_policy = 's3_main';

-- Step 2: Create distributed table for querying across all shards
CREATE TABLE events ON CLUSTER easy_db_lab AS events_local
ENGINE = Distributed(easy_db_lab, default, events_local, rand());

Key points:

• ON CLUSTER easy_db_lab runs the DDL on all nodes
• {shard} and {replica} are ClickHouse macros automatically set per node
• ReplicatedMergeTree replicates data within a shard using ClickHouse Keeper
• Distributed routes queries and inserts across shards
• rand() distributes inserts randomly; use a column for deterministic sharding

Querying and Inserting

-- Insert through distributed table (auto-sharded)
INSERT INTO events VALUES (1, now(), 'click', '{"page": "/home"}');

-- Query across all shards
SELECT count(*) FROM events WHERE event_type = 'click';

-- Query a specific shard (via local table)
SELECT count(*) FROM events_local WHERE event_type = 'click';

Table Engine Comparison

Storage Policies

ClickHouse is configured with two storage policies. You select the policy when creating a table using the SETTINGS storage_policy clause.

Policy Comparison

Local Storage (local)

The default policy stores data on local NVMe disks attached to the database nodes. This provides the best performance for latency-sensitive workloads.

CREATE TABLE my_table (...)
ENGINE = MergeTree()
ORDER BY id
SETTINGS storage_policy = 'local';

If you omit the storage_policy setting, tables use local storage by default.

When to use local storage:

• Performance benchmarking where latency matters
• Temporary or experimental datasets
• Workloads with predictable data sizes that fit on local disks
• When you don't need data to persist after cluster teardown

S3 Storage (s3_main)

The S3 policy stores data in your configured S3 bucket with a local cache for frequently accessed data. The cache size defaults to 10Gi and can be configured with clickhouse init --s3-cache. Write-through caching is enabled by default (--s3-cache-on-write true), which caches data during writes so subsequent reads can be served from cache immediately. This is ideal for large datasets where storage cost matters more than latency.

Prerequisite: Your cluster must be initialized with an S3 bucket. Set this during init:

easy-db-lab init my-cluster --s3-bucket my-clickhouse-data

Then create tables with S3 storage:

CREATE TABLE my_table (...)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/default/my_table', '{replica}')
ORDER BY id
SETTINGS storage_policy = 's3_main';

When to use S3 storage:

• Large analytical datasets (terabytes+)
• Data that should persist across cluster restarts
• Cost-sensitive workloads where storage cost > compute cost
• Sharing data between multiple clusters

How the cache works:

• Hot (frequently accessed) data is cached locally for fast reads
• Cold data is fetched from S3 on demand
• Cache is automatically managed by ClickHouse
• First query on cold data will be slower; subsequent queries use cache

S3 Tiered Storage (s3_tier)

The S3 tiered policy provides automatic data movement from local disks to S3 based on disk space availability. This policy starts with local storage and automatically moves data to S3 when local disk space runs low, providing the best of both worlds: fast local performance for hot data and unlimited S3 capacity for cold data.

Prerequisite: Your cluster must be initialized with an S3 bucket. Set this during init:

easy-db-lab init my-cluster --s3-bucket my-clickhouse-data

Configure the tiering behavior before starting ClickHouse:

# Move data to S3 when local disk free space falls below 20% (default)
easy-db-lab clickhouse init --s3-tier-move-factor 0.2

# More aggressive tiering - move when free space < 50%
easy-db-lab clickhouse init --s3-tier-move-factor 0.5

Then create tables with S3 tiered storage:

CREATE TABLE my_table (...)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/default/my_table', '{replica}')
ORDER BY id
SETTINGS storage_policy = 's3_tier';

When to use S3 tiered storage:

• Workloads with mixed hot/cold data access patterns
• Growing datasets that may outgrow local disk capacity
• Want automatic cost optimization without manual intervention
• Need local performance for recent data with S3 capacity for historical data

How automatic tiering works:

• New data is written to local disks first (fast writes)
• When local disk free space falls below the configured threshold (default: 20%), ClickHouse automatically moves the oldest data to S3
• Data on S3 is still queryable but with higher latency
• The local cache (configured with --s3-cache) helps performance for frequently accessed S3 data
• Manual moves are also possible: ALTER TABLE my_table MOVE PARTITION tuple() TO DISK 's3'

Backup and Restore

easy-db-lab provides backup and restore commands for ClickHouse workloads. Backups are stored in your account-level S3 bucket, outside the per-cluster prefix, so they survive cluster teardown and can be restored into a new cluster.

ClickHouse's native BACKUP/RESTORE SQL is used. The backup destination is a named disk (s3_backup) configured in the ClickHouseInstallation CR, which points to:

s3://<account-bucket>/clickhouse-backups/<backup-name>/

The S3 disk uses IAM instance profile credentials — no AWS keys are stored anywhere in the cluster configuration. The required S3 permissions on the account-level bucket are handled automatically by the easy-db-lab IAM setup — no manual configuration is needed for standard clusters.

Creating a Backup

easy-db-lab clickhouse backup <backup-name>

This discovers the primary ClickHouse pod and runs:

BACKUP DATABASE default ON CLUSTER clickhouse TO Disk('s3_backup', '<backup-name>/');

Restoring a Backup

easy-db-lab clickhouse restore <backup-name>

This discovers the primary ClickHouse pod and runs:

RESTORE DATABASE default ON CLUSTER clickhouse FROM Disk('s3_backup', '<backup-name>/');

Note: The restore command does not drop existing tables first. If tables with conflicting names exist, the restore will fail. Drop or truncate the conflicting tables before restoring.

Common Workflows

Snapshot before a destructive operation:

# Take a named snapshot before running a migration
easy-db-lab clickhouse backup pre-migration-snapshot

# Run the migration
# ...

# If something goes wrong, restore
easy-db-lab clickhouse restore pre-migration-snapshot

Persist data across cluster rebuilds:

# Back up before tearing down
easy-db-lab clickhouse backup my-dataset
easy-db-lab down

# Create a new cluster and restore
easy-db-lab up
easy-db-lab clickhouse install --size 100Gi
easy-db-lab clickhouse start
easy-db-lab clickhouse restore my-dataset

Stopping ClickHouse

To remove the ClickHouse cluster:

easy-db-lab clickhouse stop

This removes all ClickHouse pods, services, and associated resources from Kubernetes.

Monitoring

ClickHouse metrics are automatically integrated with the observability stack:

• Grafana Dashboard: Pre-configured dashboard for ClickHouse metrics
• Metrics Port: 9363 for Prometheus-compatible metrics
• Logs Dashboard: Dedicated dashboard for ClickHouse logs

Architecture

The ClickHouse deployment includes:

• ClickHouse Server: StatefulSet with configurable replicas
• ClickHouse Keeper: 3-node cluster for distributed coordination (ZooKeeper-compatible)
• Services: Headless services for internal communication
• ConfigMaps: Server and Keeper configuration
• Local PersistentVolumes: One PV per node for data locality

Storage Architecture

ClickHouse uses Local PersistentVolumes to guarantee pod-to-node pinning:

1. During cluster creation, each db node is labeled with its ordinal (easydblab.com/node-ordinal=0, etc.)
2. Local PVs are created with node affinity matching these ordinals
3. PVs are pre-bound to specific PVCs (e.g., data-clickhouse-0 binds to the PV on db0)
4. The StatefulSet's volumeClaimTemplate requests storage from these pre-bound PVs

This ensures clickhouse-X always runs on dbX, providing:

• Consistent shard assignments across restarts
• Data locality (no network storage overhead)
• Predictable failover behavior

Ports

Backup & Restore

Install Kafka

The kit install kafka command sets up Apache Kafka in KRaft mode (no ZooKeeper) using the Strimzi operator. Installation provisions local persistent volumes on db nodes and installs the Strimzi operator via Helm. easy-db-lab kafka start then applies the Kafka custom resource and waits for the cluster to become Ready; easy-db-lab kafka stop tears it back down.

Prerequisites

• Cluster is up (easy-db-lab up)
• At least one db node is provisioned
• Environment is sourced: source env.sh

Quick Start

easy-db-lab kit install kafka
easy-db-lab kafka start

Install Flags

Managing Kafka

start

Deploys the Kafka cluster and waits for it to become Ready:

1. Applies the JMX metrics ConfigMap
2. Applies the Kafka and KafkaNodePool custom resources
3. Waits for the Kafka cluster to report Ready (up to 300s)
4. Applies NodePort services for the kafka-exporter and JMX exporter

Grafana dashboards in kafka/dashboards/ are installed automatically after a successful start.

stop

Removes the Kafka and KafkaNodePool resources and NodePort services. Persistent volumes are retained — topic data survives a stop/start cycle.

uninstall

Removes all Kafka resources including PVCs and PVs, then uninstalls the Strimzi operator.

Bootstrap Addresses

Internal (in-cluster pods, e.g. a Cassandra sidecar or another kit):

kafka.default.svc.cluster.local:9092

External (from outside the cluster via Tailscale):

<control-node-private-ip>:32100

The external address is printed in the kafka/README.md generated at install time.

Benchmarking

Producer performance test

easy-db-lab kafka producer-perf

Consumer performance test

easy-db-lab kafka consumer-perf

Topic management

easy-db-lab kafka create-topic --topic my-topic --partitions 3 --replication-factor 3

Metrics

Two Prometheus scrape jobs are registered automatically by kafka start:

Metric names are lowercase. See kafka/METRICS.md for the full catalog.

The Kafka Overview Grafana dashboard is installed automatically and shows broker health, throughput, consumer lag, and topic health panels.

CDC: Cassandra → Kafka

When using Kafka as a CDC target for Cassandra, configure the sidecar connector with the internal bootstrap address to avoid NodePort overhead:

kafka.default.svc.cluster.local:9092

Replication and Multi-Broker Setup

The default install uses a single combined broker/controller node. For replication testing, install with multiple brokers and set the replication factor when creating topics:

easy-db-lab kit install kafka --brokers 3
easy-db-lab kafka start
easy-db-lab kafka create-topic --topic my-topic --partitions 3 --replication-factor 3

--brokers must not exceed the number of db nodes in the cluster.

Cleaning Up

To remove Kafka and free disk space:

easy-db-lab kafka stop
easy-db-lab kit uninstall kafka

kit uninstall deletes the Kafka resources, PVCs/PVs, and the Strimzi operator.

Apache Ignite 3

easy-db-lab supports deploying Apache Ignite 3 clusters on Kubernetes for distributed SQL and in-memory computing workloads.

Overview

Apache Ignite 3 is a distributed database with ACID transactions, distributed SQL, and a pluggable storage engine. It runs as a StatefulSet on K3s with configurable storage profiles ranging from pure in-memory to fully disk-backed persistence.

Metrics are automatically pushed to the cluster's OTel Collector via OTLP and appear in Grafana.

Quick Start

# Initialize and start a 3-node cluster
easy-db-lab init my-cluster --db 3 --up

# Install and start Ignite 3 with default settings
easy-db-lab kit install ignite3
easy-db-lab ignite3 start

Configuration Options

Storage Profiles

Ignite 3 supports three storage engines, selectable at start time:

# Start with pure in-memory storage (fastest, no persistence)
easy-db-lab ignite3 start --storage aimem

# Start with disk-based storage for large datasets
easy-db-lab ignite3 start --storage rocksdb

SQL Queries

Run SQL directly against the cluster using the thin client JDBC driver:

# Execute a SQL statement
easy-db-lab ignite3 sql "CREATE TABLE t1 (id INT PRIMARY KEY, val VARCHAR)"
easy-db-lab ignite3 sql "INSERT INTO t1 VALUES (1, 'hello')"
easy-db-lab ignite3 sql "SELECT * FROM t1"

# Execute SQL from a file
easy-db-lab ignite3 sql --file query.sql

Lifecycle

# Start the cluster
easy-db-lab ignite3 start

# Stop (preserves PVCs and data for aipersist/rocksdb profiles)
easy-db-lab ignite3 stop

# Start again — existing data is available immediately
easy-db-lab ignite3 start

# Remove all resources including data
easy-db-lab kit uninstall ignite3

Endpoints

Metrics

Ignite 3 metrics are pushed to the cluster's OTel Collector via OTLP automatically at start time. No additional configuration is needed. Metrics appear in VictoriaMetrics and are accessible from Grafana.

Install Presto

The install presto command scaffolds a Presto deployment using the Presto Helm chart. It generates scripts and a values.yaml in a local presto/ directory. Once installed, use easy-db-lab presto start and easy-db-lab presto stop to manage it.

Presto is stateless — it does not use persistent volumes. Queries run in-memory on app (type=app) nodes.

Prerequisites

• Cluster is up (easy-db-lab up)
• App nodes are provisioned (at least one ServerType.Stress node)
• kubectl and helm are available in your PATH (or run via the easy-db-lab container)
• Environment is sourced: source env.sh

Quick Start

easy-db-lab kit install presto
easy-db-lab presto start

Flags

What Gets Generated

presto/
├── README.md                    # Usage instructions for this cluster
├── values.yaml                  # Helm values for the Presto chart
├── catalogs/
│   ├── cassandra.properties     # Cassandra connector config
│   └── clickhouse.properties    # ClickHouse JDBC connector config
└── bin/
    ├── start.sh                 # Deploy sequence
    ├── stop.sh                  # Teardown sequence
    └── update-catalogs.sh       # Re-renders Presto catalog config from running kits

Managing Presto

After installation, use the CLI to start and stop:

# Deploy Presto
easy-db-lab presto start

# Tear down Presto
easy-db-lab presto stop

start

helm upgrade --install presto prestodb/presto -f values.yaml
# update-catalogs.sh runs automatically after helm install
kubectl wait --for=condition=Ready pods -l app=presto,component=coordinator --timeout=180s

No platform create-pvs step — Presto is stateless.

stop

helm uninstall presto

Automatic Catalog Management

Presto has built-in support for two catalogs that wire up automatically:

• Cassandra — uses the Cassandra connector, pointing at the cluster's Cassandra nodes
• ClickHouse — uses the ClickHouse JDBC connector, pointing at the cluster's ClickHouse nodes

When either of those kits starts or stops, Presto detects the change via a post-workload-start / post-workload-stop hook and re-renders its catalog configuration automatically. No restart of Presto is needed.

# Start Cassandra — Presto wires up the cassandra catalog automatically
easy-db-lab cassandra start

# Start ClickHouse — Presto wires up the clickhouse catalog automatically
easy-db-lab kit install clickhouse
easy-db-lab clickhouse start

You do not need to restart Presto or manually update any config when adding or removing either kit.

Node Placement

Presto workers are scheduled on type=app nodes using nodeSelector: type: app. The coordinator runs on the control plane or app nodes depending on cluster size.

Connecting

After easy-db-lab presto start completes, the Presto coordinator is accessible within the cluster. Use kubectl port-forward or the SOCKS5 proxy to connect from your workstation:

kubectl port-forward svc/presto 8080:8080

Then connect with any Presto-compatible client at localhost:8080.

Adding Presto Catalogs for Custom Kits

If you install a custom kit via --from and want Presto to connect to it, drop a presto-catalog.properties file in the kit's installed directory:

my-kit/
├── bin/
│   ├── start.sh
│   └── stop.sh
└── presto-catalog.properties    # Presto picks this up automatically

The file follows the standard Presto connector properties format:

connector.name=jdbc
connection-url=jdbc:postgresql://localhost:5432/mydb
connection-user=presto
connection-password=

update-catalogs.sh scans all sibling kit directories for this file. When found, the catalog is added to Presto's configuration under a name matching the kit directory name. No restart of Presto is required — the script runs helm upgrade with the merged catalog values.

Install Trino

The install trino command scaffolds a Trino deployment using the Trino Helm chart. It generates scripts and a values.yaml in a local trino/ directory. Once installed, use easy-db-lab trino start and easy-db-lab trino stop to manage it.

Trino is stateless — it does not use persistent volumes. Queries run in-memory on app (type=app) nodes.

Prerequisites

• Cluster is up (easy-db-lab up)
• App nodes are provisioned (at least one ServerType.Stress node)
• kubectl and helm are available on the control node
• Environment is sourced: source env.sh

Quick Start

easy-db-lab kit install trino
easy-db-lab trino start

Flags

What Gets Generated

trino/
├── README.md                    # Usage instructions for this cluster
├── values.yaml                  # Helm values for the Trino chart
├── catalogs/
│   ├── cassandra.properties     # Cassandra connector config
│   └── clickhouse.properties    # ClickHouse JDBC connector config
└── bin/
    ├── start.sh                 # Deploy sequence
    ├── stop.sh                  # Teardown sequence
    ├── uninstall.sh             # Remove Helm release
    └── update-catalogs.sh       # Re-renders Trino catalog config from running kits

Managing Trino

After installation, use the CLI to start and stop:

# Deploy Trino
easy-db-lab trino start

# Stop Trino (scale to zero, no data loss)
easy-db-lab trino stop

Automatic Catalog Management

Trino has built-in support for two catalogs that wire up automatically:

• Cassandra — uses the Cassandra connector, pointing at the cluster's Cassandra nodes
• ClickHouse — uses the ClickHouse JDBC connector, pointing at the cluster's ClickHouse nodes

When either of those kits starts or stops, Trino detects the change via a post-workload-start / post-workload-stop hook and re-renders its catalog configuration automatically. No restart of Trino is needed.

# Start Cassandra — Trino wires up the cassandra catalog automatically
easy-db-lab cassandra start

# Start ClickHouse — Trino wires up the clickhouse catalog automatically
easy-db-lab kit install clickhouse
easy-db-lab clickhouse start

Node Placement

Trino workers are scheduled on type=app nodes using nodeSelector: type: app.

Connecting

After easy-db-lab trino start completes, the Trino coordinator is accessible within the cluster. Connect using the SOCKS5 proxy or a port-forward:

kubectl port-forward svc/trino 8080:8080

Then connect with any Trino-compatible client at jdbc:trino://localhost:8080.

You can also use the built-in SQL command:

easy-db-lab trino sql "SELECT count(*) FROM cassandra.mykeyspace.mytable"

Adding Trino Catalogs for Custom Kits

If you install a custom kit via --from and want Trino to connect to it, drop a trino-catalog.properties file in the kit's installed directory:

my-kit/
├── bin/
│   ├── start.sh
│   └── stop.sh
└── trino-catalog.properties    # Trino picks this up automatically

The file follows the standard Trino connector properties format:

connector.name=postgresql
connection-url=jdbc:postgresql://localhost:5432/mydb
connection-user=trino
connection-password=

update-catalogs.sh scans all sibling kit directories for this file. When found, the catalog is added to Trino's configuration under a name matching the kit directory name. No restart of Trino is required — the script runs helm upgrade with the merged catalog values.

Presto vs Trino

Trino is the open-source fork of Presto (previously known as PrestoSQL). Both are supported as independent kits. Choose Trino for the active open-source community and Presto for compatibility with Meta's Presto ecosystem.

TiDB

The tidb kit deploys a TiDB HTAP cluster using the TiDB Operator. TiDB combines a MySQL-compatible SQL layer with two storage engines: TiKV (row store, for OLTP) and TiFlash (columnar store, for analytics) — letting you run transactional and analytical queries against the same data.

Prerequisites

• Cluster is up (easy-db-lab up)
• At least 1 db node (runs TiKV and TiFlash)
• At least 1 app node (runs TiDB and PD)

Quick Start

easy-db-lab kit install tidb
easy-db-lab tidb start
easy-db-lab tidb sql "SELECT tidb_version()"

tidb start deploys the TiDB Operator-managed cluster and waits for each component (PD, TiKV, TiDB, TiFlash) to become Ready. TiFlash takes the longest — expect a few minutes on first start while images pull.

Flags

Cluster Layout

Connecting

TiDB speaks the MySQL wire protocol, exposed as NodePort 30400 on every cluster node:

mysql -h <node-ip> -P 30400 -u root

Or use the built-in SQL command, which resolves the endpoint for you:

easy-db-lab tidb sql "SHOW DATABASES"

Using TiFlash

Tables are not automatically replicated to TiFlash. Enable replication per table:

ALTER TABLE my_table SET TIFLASH REPLICA 1;

Once the replica is in place, TiDB's optimizer routes analytical queries to TiFlash automatically. To force it for a specific query:

SELECT /*+ read_from_storage(tiflash[my_table]) */ count(*) FROM my_table;

Lifecycle

easy-db-lab tidb start       # deploy the TiDB cluster
easy-db-lab tidb status      # show running state and endpoints
easy-db-lab tidb stop        # tear down the TiDB cluster
easy-db-lab tidb uninstall   # remove the TiDB Operator (requires stop first)

uninstall refuses to run while the TiDB cluster is still up — run tidb stop first.

Monitoring

The kit registers four Prometheus scrape jobs with the cluster's observability stack automatically — no configuration needed:

Metrics are available in Grafana and VictoriaMetrics as soon as the kit starts.

TiDB also exports traces to Tempo (via the OTel Collector's Jaeger receiver, since TiDB v8.x has no native OTLP support). Search for them in Grafana with service.name=TiDB — the tag is case-sensitive.

Benchmarking

TiDB declares the sql capability, so bench kits can target it directly. See Sysbench:

easy-db-lab kit install sysbench --target tidb

Sysbench

The sysbench kit runs sysbench OLTP benchmarks against a running database kit. It is a bench kit: it does not deploy a database itself, but targets one you've already started, connecting over the MySQL or PostgreSQL wire protocol. Benchmark pods run inside the Kubernetes cluster, and per-interval results are pushed to VictoriaMetrics so you can watch throughput and latency live in Grafana.

Prerequisites

• Cluster is up (easy-db-lab up)
• A database kit with the sql capability is installed and running (e.g. TiDB)
• The target kit exposes a MySQL or PostgreSQL wire protocol endpoint — sysbench connects over the wire protocol, not JDBC

Quick Start

# Start a database to benchmark
easy-db-lab kit install tidb
easy-db-lab tidb start

# Install sysbench pointed at it
easy-db-lab kit install sysbench --target tidb

# Load data, run the benchmark, clean up
easy-db-lab sysbench-tidb prepare
easy-db-lab sysbench-tidb start
easy-db-lab sysbench-tidb stop

The kit installs as sysbench-<target> — the instance name and the CLI subcommand both include the target, so multiple sysbench instances can run against different databases at the same time. See Bench kits for how cross-kit targeting works.

Flags

--target is baked in at install time — to point sysbench at a different database, install another instance. Every other flag is passed per invocation, so you can vary them run to run without reinstalling:

easy-db-lab sysbench-tidb start --threads 32 --duration 300

Lifecycle

prepare

easy-db-lab sysbench-tidb prepare

Creates the sbtest database on the target if it doesn't exist, then loads the test tables (--tables tables with --scale thousand rows each). Run this once before the first benchmark run.

start

easy-db-lab sysbench-tidb start

Runs the benchmark for --duration seconds, streaming sysbench's interval output to your terminal. Each 10-second interval report (TPS, QPS, p99 latency, errors/s) is also pushed to VictoriaMetrics.

When the run finishes, the final sysbench summary (SQL statistics, throughput, latency percentiles, and errors) is written to last-run.txt in the kit's workspace directory (e.g. sysbench-tidb/last-run.txt), prefixed with the run's parameters. Each run overwrites the file, so a completed run's numbers survive after the terminal output scrolls away.

stop

easy-db-lab sysbench-tidb stop

Kills any running benchmark pod and runs sysbench cleanup, dropping the test tables from the target database.

Rate limiting and overload testing

By default (--rate=0) sysbench is thread-bound: each of the --threads worker threads issues transactions as fast as the target will answer them, so throughput settles at whatever the database can sustain. Setting --rate to a non-zero value switches sysbench to a fixed target rate — it generates events on a schedule of that many transactions per second and hands them to the worker threads, regardless of how fast the target is actually responding.

That distinction matters when the requested rate exceeds what the target can sustain. The generated events queue up faster than the workers can drain them, sysbench's internal event queue fills, and the run hard-aborts with:

FATAL: event queue is full

The abort is fast — under 15 seconds into the run in the case that prompted this section, regardless of the --duration you asked for. A --rate set well above capacity does not produce a sustained high-latency window; it produces a run that dies almost immediately with no useful results.

An aborted run is easy to miss after the fact, because it does not look like a failure downstream. last-run.txt holds only the seeded parameter header with no SQL statistics block, and the run's p50/p95/p99 series on the Grafana dashboard flatline at 0 — which reads as a suspiciously excellent result rather than a crash. If latency drops to zero and the summary is truncated, check the pod output for the FATAL line.

For overload and latency testing, drive the target past its limit with concurrency instead of with a target rate: leave --rate=0 and raise --threads until latency climbs. A thread-bound run applies backpressure naturally — slower responses mean fewer transactions issued — so it degrades into a high-latency window instead of overflowing the event queue. It is not immune to aborting for other reasons: sysbench still exits on unhandled SQL errors, which a heavily overloaded target is more likely to return. If you do want a fixed rate, first measure the target's sustainable throughput with a thread-bound run, then set --rate at or just above that measured number rather than far above it.

Comparing Databases

Because each install is a separate named instance, you can benchmark several databases simultaneously and compare them side by side in Grafana:

easy-db-lab kit install sysbench --target tidb
easy-db-lab kit install sysbench --target my-custom-db

easy-db-lab sysbench-tidb prepare && easy-db-lab sysbench-tidb start
easy-db-lab sysbench-my-custom-db prepare && easy-db-lab sysbench-my-custom-db start

Of the built-in kits, TiDB and ClickHouse expose MySQL and PostgreSQL wire endpoints. Any custom kit that declares a mysql or postgresql endpoint and the sql capability in its kit.yaml can be targeted the same way.

Note that ClickHouse's wire interfaces parse queries as ClickHouse SQL, and sysbench's built-in oltp_* workloads issue MySQL-specific DDL during prepare — running them against ClickHouse unmodified will fail at table creation. Benchmarking ClickHouse with sysbench requires a custom Lua workload with ClickHouse-compatible schemas.

Metrics & Dashboard

The kit ships a Sysbench Benchmark Grafana dashboard, installed automatically. During a run, these metrics are pushed to VictoriaMetrics, labelled by instance (kit):

The kit label carries the instance name (e.g. sysbench-tidb), so runs against different targets plot as separate series on the same panel.

sysbench's interval reports only emit the single configured percentile (p99), so p50/p95 cannot be sampled per interval. The kit runs sysbench with --histogram and parses the final latency histogram to derive whole-run p50/p95/p99, pushed once when the run completes.

Install PostgreSQL

The postgres kit deploys PostgreSQL on K8s db nodes via the CloudNativePG (CNPG) operator. Data is persisted on PersistentVolumes — stopping and restarting preserves your dataset.

Prerequisites

• Cluster is up (easy-db-lab up) with at least one db node

Quick Start

easy-db-lab kit install postgres
easy-db-lab postgres start

Flags

Managing PostgreSQL

# Start PostgreSQL
easy-db-lab postgres start

# Stop PostgreSQL (data is preserved)
easy-db-lab postgres stop

# Remove operator and delete all PersistentVolumes
easy-db-lab postgres uninstall

start

1. Creates PersistentVolumes on db nodes via platform-pvs
2. Applies the CNPG Cluster custom resource
3. Waits for all pods to reach Ready
4. Applies the NodePort service for external access

stop

Deletes the CNPG Cluster CR and the NodePort service. PersistentVolumes are retained — running postgres start again resumes from the existing dataset.

uninstall

Deletes PersistentVolumes and uninstalls the CNPG operator Helm release.

Connecting

The PostgreSQL primary is exposed as a NodePort on port 30432 of each db node.

# JDBC URL
jdbc:postgresql://<db-node-ip>:30432/postgres

# psql via SOCKS5 proxy or port-forward
kubectl port-forward svc/postgres-nodeport 5432:5432
psql -h localhost -U postgres postgres

The default user is postgres with password postgres (stored in the postgres-credentials K8s Secret).

Running SQL

Use the built-in sql capability to execute queries directly:

easy-db-lab postgres sql "SELECT version()"
easy-db-lab postgres sql --file query.sql

Extensions

PostgreSQL extensions that require custom container images are activated at install time via --extension on kit install postgres. Each extension installs as a separate named instance (e.g. postgres-duckdb) that starts and stops independently.

Listing available extensions

easy-db-lab postgres extensions

Outputs a table of alias names, image templates, shared_preload_libraries, and CREATE EXTENSION statements.

Built-in aliases

Example

# Install with an extension
easy-db-lab kit install postgres --extension duckdb

# Start, stop, and use the named instance
easy-db-lab postgres-duckdb start
easy-db-lab postgres-duckdb sql "SELECT duckdb_version()"
easy-db-lab postgres-duckdb stop

Versioning

Built-in alias images use __PG_MAJOR__ as a placeholder for the PostgreSQL major version (e.g. 17). This is substituted automatically from the --version flag set at install time.

Presto Integration

When both postgres and presto are running, Presto automatically exposes a postgres catalog using the PostgreSQL JDBC connector pointed at the CNPG primary service. No manual configuration is needed.

easy-db-lab kit install postgres
easy-db-lab postgres start

easy-db-lab kit install presto
easy-db-lab presto start

# postgres catalog is available automatically
easy-db-lab presto sql "SHOW CATALOGS"

Platform Substrate

The platform substrate is the set of Kubernetes primitives that easy-db-lab provisions on every cluster. It provides a stable foundation so any kit can be deployed without bespoke manifest code.

Two-Layer Model

┌─────────────────────────────────────┐
│         Kit Layer              │
│  (ClickHouse, Presto, custom, …)    │
│  deployed via install + helm/kubectl│
└────────────────┬────────────────────┘
                 │ uses
┌────────────────▼────────────────────┐
│        Platform Substrate           │
│  StorageClasses · Node Labels · PVs │
│  provisioned at cluster `up` time   │
└─────────────────────────────────────┘

The platform substrate is provisioned automatically when you run easy-db-lab up. You do not need to configure it manually.

StorageClasses

Two StorageClasses are created at up time:

local-storage-wfc is used for StatefulSet kits. Kubernetes must know which node a pod schedules on before binding the volume — WaitForFirstConsumer enforces this ordering.

Node Labels

All cluster nodes are labeled at up time:

Use nodeSelector: type: db (or app) in pod specs to constrain placement. The ordinal label is used by the PV pre-binding mechanism so each StatefulSet replica lands on the right node.

Persistent Volumes

Per-kit PVs are created lazily at install time, not at cluster-up time. Run platform create-pvs before starting a stateful kit:

easy-db-lab platform create-pvs --kit clickhouse --size 100Gi

This creates one PV per db node with:

• Path: /mnt/db1/<kit> on each host
• StorageClass: local-storage-wfc
• Node affinity: easydblab.com/node-ordinal=N for deterministic binding
• ClaimRef: pre-bound to <volumeClaimTemplateName>-<kit>-N

The command is safe to re-run. If a PV exists with a stale claimRef (the PVC was deleted), the UID is cleared and the PV is returned to Available.

platform Commands

platform create-pvs

easy-db-lab platform create-pvs --kit <name> --size <Gi> [--node-type db|app] [--pvc-name <name>]

Creates one PV per node of the specified type. Defaults to db nodes. For stateless kits that need app-node storage, use --node-type app. --pvc-name sets the volumeClaimTemplate name to pre-bind against (default: data).

platform info

easy-db-lab platform info

Displays StorageClasses, available PV counts per node pool, node selector labels, and the ordinal label key. Use this to verify substrate readiness before deploying a kit.

Custom Templates

The install command can render templates from a custom directory:

easy-db-lab kit install --from ./my-kit/ --kit my-kit --size 50Gi

Template Variable Contract

All templates receive these standard variables from cluster state:

Unresolved __VAR__ placeholders emit a warning but do not fail the render.

Template Directory Layout

my-kit/
├── kit.yaml
├── README.md.template
├── values.yaml.template
└── bin/
    ├── start.sh.template
    └── stop.sh.template

Files without .template suffix are copied verbatim.

Profile Templates

Place templates in ~/.easy-db-lab/profiles/<profile>/kits/<name>/ to make them discoverable via kit list. Additional template directories can be registered with kit source add. Resolution priority: profile templates override additional sources, which override built-in templates of the same name.

Port Exposure Model

Kits use standard pod networking (not hostNetwork). Client and metrics ports are surfaced on each EC2 instance's network interface in one of two ways:

• NodePort services — stateful db kits (ClickHouse, TiDB) expose their ports through a NodePort service, remapping native ports into the NodePort range (30000–32767). Example: ClickHouse HTTP 8123 → 30123.
• hostPort patches — helm-based app kits (Presto, Trino) patch hostPort mappings onto the coordinator pod at start time, keeping the native port (e.g. 8080 → 8080).

Why ports must reach the host

The OTel collector DaemonSet runs with hostNetwork: true so it can scrape both host processes (Cassandra/MAAC at localhost:9000) and kit metrics endpoints. It scrapes each kit's declared metrics port at localhost:<port>, so that port must be reachable on every node's host network — which both NodePort (listens on all nodes) and hostPort provide. This also avoids conflicts with host processes: a NodePort-range port can never collide with a database listening on its native port on the host.

Port Assignments

When adding a new kit, choose ports that do not conflict with any host process or existing kit in the table above. Each kit's ports are declared in its kit.yaml (metrics and endpoints sections).

Kit Observability

Each kit declares its metrics targets in kit.yaml. metrics is a list — kits with multiple components declare one entry per scrape target, each with a unique job name:

metrics:
  - type: scrape     # Prometheus endpoint — OTel DaemonSet scrapes it at localhost:<port>
    port: 31080
    path: /metrics
    job: tidb-sql
  - type: scrape     # pod service discovery — each pod scraped directly, per-pod `instance`
    job: tikv
    pod-selector: "app.kubernetes.io/component=tikv,app.kubernetes.io/instance=tidb"
    port: 20180      # container metrics port, not a NodePort
    path: /metrics

If job is omitted, the kit name is used.

Static NodePort vs. pod discovery. By default a scrape target is a static localhost:<port> NodePort — every collector scrapes it and instance is the collector's hostname. When a target sets pod-selector (a comma-separated K8s label selector), the OTel collector instead uses Prometheus pod service discovery (kubernetes_sd_configs, role: pod): each collector scrapes only the matching pods co-located on its own node, and instance becomes the pod name. Use this when a component has multiple pods behind one service and you need per-pod attribution — a NodePort load-balances scrapes across all pods, so a single store/pod cannot be distinguished. TiKV uses this so each of the 3 stores reports under its own instance (e.g. tidb-tikv-0).

Three modes are supported:

Metrics Registration Lifecycle

When a kit with type: scrape targets starts successfully, easy-db-lab:

1. Creates one ConfigMap easydblab-metrics-<job> per scrape target in the default namespace, labeled easydblab.com/workload-metrics=true and easydblab.com/kit=<kit>, containing the job name, port, and path.
2. Regenerates the OTel collector ConfigMap to include a Prometheus scrape job per target.
3. Applies the updated OTel ConfigMap so the running collector picks it up.

When the kit stops:

1. Deletes all of the kit's metrics ConfigMaps via the easydblab.com/kit label selector.
2. Regenerates and applies the OTel collector ConfigMap without the kit's scrape jobs.

This is fully automatic — no manual OTel configuration is required when starting or stopping kits.

Verifying the Substrate

# Check StorageClasses and PV availability
easy-db-lab platform info

# List all nodes and their labels
kubectl get nodes --show-labels

# List PVs for a kit
kubectl get pv | grep clickhouse

Monitoring

Grafana Dashboards

Grafana is deployed automatically as part of the observability stack (k8 apply). It is accessible on port 3000 of the control node.

Cluster Identification

When running multiple environments side by side, Grafana displays the cluster name in several places to help you identify which environment you're looking at:

• Browser tab - Shows the cluster name instead of "Grafana"
• Dashboard titles - Each dashboard title is prefixed with the cluster name
• Sidebar org name - The organization name in the sidebar shows the cluster name
• Home dashboard - The System Overview dashboard is set as the home page instead of the default Grafana welcome page

System Dashboard

Shows CPU, memory, disk I/O, network I/O, and load average for all cluster nodes via OpenTelemetry metrics.

AWS CloudWatch Overview

A combined dashboard showing S3, EBS, and EC2 metrics via CloudWatch. Available after running easy-db-lab up.

S3 metrics:

• Throughput: BytesDownloaded, BytesUploaded
• Request Counts: GetRequests, PutRequests
• Latency: FirstByteLatency (p99), TotalRequestLatency (p99)

EBS volume metrics:

• IOPS: VolumeReadOps, VolumeWriteOps (mirrored read/write chart)
• Throughput: VolumeReadBytes, VolumeWriteBytes (mirrored read/write chart)
• Queue Length: VolumeQueueLength
• Burst Balance: BurstBalance (percentage)

EC2 status checks:

• Status Check Failures: StatusCheckFailed_Instance, StatusCheckFailed_System (red threshold at >= 1)

Use the dropdowns at the top to select S3 bucket, EC2 instances, and EBS volumes.

How it works:

• S3 request metrics are automatically enabled for the cluster's prefix in the account S3 bucket during easy-db-lab up
• EBS and EC2 metrics are published automatically by AWS for all instances and volumes
• Grafana queries CloudWatch using the EC2 instance's IAM role (no credentials needed)
• During easy-db-lab down, the S3 metrics configuration is automatically removed to stop CloudWatch billing

Note: S3 request metrics take approximately 15 minutes to appear in CloudWatch after being enabled. EBS and EC2 metrics are available immediately.

EMR Overview

Shows Spark/EMR node metrics via OpenTelemetry. Available when an EMR cluster is provisioned. Each EMR node runs an OTel Collector that collects host metrics and receives JVM telemetry from the OTel and Pyroscope Java agents.

Host Metrics:

• CPU Usage: Per-node CPU utilization percentage
• Memory Usage: Used and cached memory per node
• Disk I/O: Read/write throughput per node (mirrored chart)
• Network I/O: Receive/transmit throughput per node (mirrored chart)
• Load Average: 1m and 5m load per node
• Filesystem Usage: Root filesystem utilization percentage

Spark JVM Metrics:

• JVM Heap Memory: Used and committed heap per node/pool
• GC Duration Rate: Garbage collection duration rate per collector
• JVM Threads: Thread count per node
• JVM Classes Loaded: Class count per node

Use the Hostname dropdown to filter by specific EMR nodes.

OpenSearch Overview

Shows OpenSearch domain metrics via CloudWatch. Available when an OpenSearch domain is provisioned.

Metrics displayed:

• Cluster Health: ClusterStatus (green/yellow/red), FreeStorageSpace
• CPU / Memory: CPUUtilization, JVMMemoryPressure
• Search Performance: SearchLatency (p99), SearchRate
• Indexing Performance: IndexingLatency (p99), IndexingRate
• HTTP Responses: 2xx, 3xx, 4xx, 5xx (color-coded)
• Storage: ClusterUsedSpace

Use the Domain dropdown to select which OpenSearch domain to view.

Cassandra Condensed

A single-pane-of-glass summary of the most important Cassandra metrics, powered by the MAAC (Management API for Apache Cassandra) agent. Shows:

• Cluster Overview: Nodes up/down, compaction rates, CQL request throughput, dropped messages, connected clients, timeouts, hints, data size, GC time
• Condensed Metrics: Request throughput, coordinator latency percentiles, memtable space, compaction activity, table-level latency, streaming bandwidth

Requires the MAAC agent to be loaded (Cassandra 4.0, 4.1, or 5.0). Metrics are exposed on port 9000 and scraped by the OTel collector.

Cassandra Overview

A comprehensive deep-dive into Cassandra cluster health, also powered by the MAAC agent. Shows:

• Request Throughput: Read/write distribution, latency percentiles (P98-P999), error throughput
• Node Status: Per-node up/down status (polystat panel), node count, status history
• Data Status: Disk space usage, data size, SSTable count, pending compactions
• Internals: Thread pool pending/blocked/active tasks, dropped messages, hinted handoff
• Hardware: CPU, memory, disk I/O, network I/O, load average
• JVM/GC: Application throughput, GC time, heap utilization

eBPF Observability

The cluster deploys eBPF-based agents on all nodes for deep system observability:

Beyla (L7 Network Metrics)

Grafana Beyla uses eBPF to automatically instrument network traffic and provide RED metrics (Rate, Errors, Duration) for:

• Cassandra CQL protocol (port 9042) and inter-node communication (port 7000)
• ClickHouse HTTP (port 8123) and native (port 9000) protocols

Metrics are scraped by the OTel collector and stored in VictoriaMetrics.

ebpf_exporter (Low-Level Metrics)

Cloudflare's ebpf_exporter provides kernel-level metrics via eBPF:

• TCP retransmits — count of retransmitted TCP segments
• Block I/O latency — histogram of block device I/O operation latency
• VFS latency — histogram of filesystem read/write operation latency

These metrics are scraped by the OTel collector and stored in VictoriaMetrics.

See Profiling for continuous profiling with Pyroscope.

Profiling

Continuous profiling is provided by Grafana Pyroscope, deployed automatically as part of the observability stack.

Architecture

Profiling data is collected from multiple sources and sent to the Pyroscope server on the control node (port 4040):

• Pyroscope Java agent (Cassandra) — Runs as a -javaagent inside the Cassandra JVM. Uses async-profiler to collect CPU, allocation, lock contention, and wall-clock profiles with full method-level resolution.
• Pyroscope Java agent (Stress jobs) — Runs as a -javaagent inside cassandra-easy-stress K8s Jobs. Collects the same profile types as Cassandra (CPU, allocation, lock). The agent JAR is mounted from the host via a hostPath volume.
• Pyroscope Java agent (Presto) — Runs as a -javaagent inside both the Presto coordinator and worker JVMs. Injected via JAVA_TOOL_OPTIONS during the presto start phase. Profiles appear under service_name=presto with component=coordinator or component=worker labels.
• Pyroscope Java agent (Spark/EMR) — Runs as a -javaagent on Spark driver and executor JVMs. Installed via EMR bootstrap action to /opt/pyroscope/pyroscope.jar. Collects CPU, allocation (512k threshold), and lock (10ms threshold) profiles in JFR format. Profiles appear under service_name=spark-<job-name>.
• Grafana Alloy eBPF profiler — Runs as a DaemonSet on all nodes via Grafana Alloy. Profiles all processes (Cassandra, ClickHouse, TiDB/TiKV/PD, Presto, stress jobs) at the system level using eBPF. Provides CPU flame graphs including kernel stack frames. Pod processes are attributed per pod/container/service_name; see the eBPF Agent section below.

Accessing Profiles

Profiling Dashboard

A dedicated Profiling dashboard is available in Grafana with flame graph panels for each profile type:

1. Open Grafana (port 3000)
2. Navigate to Dashboards and select the Profiling dashboard
3. Use the Service dropdown to select a service (e.g., cassandra, cassandra-easy-stress, clickhouse-server)
4. Use the Hostname dropdown to filter by specific nodes
5. Select a time range to view profiles for that period

The dashboard includes panels for:

• CPU Flame Graph — CPU time spent in each method
• Memory Allocation Flame Graph — Heap allocation hotspots
• Lock Contention Flame Graph — Time spent waiting for monitors
• Mutex Contention Flame Graph — Mutex delay analysis

Grafana Explore

For ad-hoc profile exploration:

1. Open Grafana (port 3000) and navigate to Explore
2. Select the Pyroscope datasource
3. Choose a profile type (e.g., process_cpu, memory, mutex)
4. Filter by labels:
   • service_name — process or application name
   • hostname — node hostname
   • cluster — cluster name

Profile Types

Java Agent (Cassandra, Stress Jobs)

eBPF Agent (All Processes)

The eBPF agent profiles all processes on every node, including ClickHouse and other kit databases (TiDB, TiKV, PD). Since these are written in C++/Go, only CPU profiles are available (no allocation or lock profiles).

Processes that run inside Kubernetes pods are attributed to their pod: they carry namespace, pod, container, and a service_name derived as <namespace>/<container> (for example tidb-cluster/tikv). This makes each kit component individually selectable in the Pyroscope UI instead of collapsing into a single unspecified service. Host processes that don't run in a pod (for example the Cassandra systemd service) are still profiled, just without pod labels.

Stress Job Profiling

Stress jobs are automatically profiled via the Pyroscope Java agent. No additional configuration is needed — when you start a stress job, the agent is mounted from the host node and configured to send profiles to the Pyroscope server.

Profiles appear under service_name=cassandra-easy-stress with labels for cluster and job_name.

Wall-Clock vs CPU Profiling

By default, the Cassandra Java agent profiles CPU time. You can switch to wall-clock profiling to find I/O bottlenecks and blocking operations.

To enable wall-clock profiling:

1. SSH to each Cassandra node
2. Add PYROSCOPE_PROFILER_EVENT=wall to /etc/default/cassandra
3. Restart Cassandra

To switch back to CPU profiling, either remove the line or set PYROSCOPE_PROFILER_EVENT=cpu.

Configuration

Cassandra Java Agent

The Pyroscope Java agent is configured via JVM system properties in cassandra.in.sh. It activates when the PYROSCOPE_SERVER_ADDRESS environment variable is set (configured by easy-db-lab at cluster startup).

The agent JAR is installed at /usr/local/pyroscope/pyroscope.jar.

eBPF Agent

The eBPF profiler runs as a privileged Grafana Alloy DaemonSet (pyroscope-ebpf) and profiles all processes on each node. Configuration is in the pyroscope-ebpf-config ConfigMap (Alloy River format). It uses discovery.kubernetes to discover the pods on each node, discovery.process (joined to those pods by container id) to discover host processes, and pyroscope.ebpf to collect CPU profiles. The DaemonSet runs under the pyroscope-ebpf ServiceAccount, whose ClusterRole grants read access to pods so samples can be attributed to a pod/container/service_name.

Pyroscope Server

The Pyroscope server runs on the control node with data stored in S3 (s3://<account-bucket>/clusters/<name>-<id>/pyroscope/). Configuration is in the pyroscope-config ConfigMap.

Data Flow

Cassandra JVM ──(Java agent)──────► Pyroscope Server (:4040)
                                         ▲
Stress Jobs ──(Java agent)──────────────┘
                                         ▲
Presto JVMs ──(Java agent)─────────────┘
                                         ▲
Spark JVMs ──(Java agent)──────────────┘
                                         ▲
All Processes ──(eBPF agent)────────────┘
                                         │
                                         ▼
                                    S3 storage
                                  Grafana (:3000)
                             Pyroscope datasource
                            + Profiling dashboard

Victoria Metrics

Victoria Metrics is a time-series database that stores metrics from all nodes in your easy-db-lab cluster. It receives metrics via OTLP from the OpenTelemetry Collector.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                     All Nodes (DaemonSet)                    │
├─────────────────────────────────────────────────────────────┤
│   System metrics (CPU, memory, disk, network)               │
│   Cassandra metrics (via JMX)                               │
│   Application metrics                                        │
└──────────────────────────┬──────────────────────────────────┘
                           │
                           ▼
              ┌────────────────────────┐
              │   OTel Collector       │
              │   (DaemonSet)          │
              └───────────┬────────────┘
                          │
┌─────────────────────────┼─────────────────────────┐
│   Control Node          │                          │
├─────────────────────────┼─────────────────────────┤
│                         ▼                          │
│              ┌──────────────────┐                  │
│              │ Victoria Metrics │                  │
│              │    (:8428)       │                  │
│              └────────┬─────────┘                  │
└───────────────────────┼────────────────────────────┘
                        │
                        ▼
              ┌──────────────────┐
              │     Grafana      │
              │    (:3000)       │
              └──────────────────┘

Configuration

Victoria Metrics runs on the control node as a Kubernetes deployment:

• Port: 8428 (HTTP API)
• Storage: Persistent at /mnt/db1/victoriametrics
• Retention: 7 days (configurable via -retentionPeriod flag)

Accessing Metrics

Grafana

1. Access Grafana at http://control0:3000 (via SOCKS proxy)
2. Victoria Metrics is pre-configured as the Prometheus datasource
3. System dashboards show node metrics

Direct API

Query metrics directly using the Prometheus-compatible API:

source env.sh

# Get all metric names
with-proxy curl "http://control0:8428/api/v1/label/__name__/values"

# Query specific metric
with-proxy curl "http://control0:8428/api/v1/query?query=up"

# Query with time range
with-proxy curl "http://control0:8428/api/v1/query_range?query=node_cpu_seconds_total&start=$(date -d '1 hour ago' +%s)&end=$(date +%s)&step=60"

Common Queries

# CPU usage by node
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage percentage
100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

# Disk usage
100 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100)

# Network received bytes
rate(node_network_receive_bytes_total[5m])

Backup

Backup Victoria Metrics data to S3:

# Backup to cluster's default S3 bucket
easy-db-lab metrics backup

# Backup to a custom S3 location
easy-db-lab metrics backup --dest s3://my-backup-bucket/victoriametrics

By default, backups are stored at: s3://{cluster-bucket}/victoriametrics/{timestamp}/

Use --dest to override the destination bucket and path

Features

• Uses native vmbackup tool with snapshot support
• Non-disruptive; metrics collection continues during backup
• Direct S3 upload (no intermediate storage needed)
• Incremental backup support for faster subsequent backups

Listing Backups

List available VictoriaMetrics backups in S3:

easy-db-lab metrics ls

This displays a summary table of all backups grouped by timestamp, showing the number of files and total size for each.

Importing Metrics to an External Instance

Stream metrics from the running cluster's VictoriaMetrics to an external VictoriaMetrics instance via the native export/import API:

# Import all metrics
easy-db-lab metrics import --target http://victoria:8428

# Import only specific metrics
easy-db-lab metrics import --target http://victoria:8428 --match '{job="cassandra"}'

This is useful for exporting metrics at the end of test runs when running easy-db-lab from a Docker container. Unlike binary backups, this approach streams data via HTTP and can target any reachable VictoriaMetrics instance.

Options

Troubleshooting

No metrics appearing

1. Verify Victoria Metrics pod is running:

   kubectl get pods -l app.kubernetes.io/name=victoriametrics
   kubectl logs -l app.kubernetes.io/name=victoriametrics
   

2. Check OTel Collector is forwarding metrics:

   kubectl get pods -l app=otel-collector
   kubectl logs -l app=otel-collector
   

3. Verify the cluster-config ConfigMap exists:

   kubectl get configmap cluster-config -o yaml
   

Connection errors

If you see connection errors when querying metrics:

1. Ensure the cluster is running: easy-db-lab status
2. The proxy is started automatically when needed
3. Check that control node is accessible: ssh control0 hostname

High memory usage

Victoria Metrics is configured with memory limits. If you see OOM kills:

1. Check current memory usage:

   kubectl top pod -l app.kubernetes.io/name=victoriametrics
   

2. Consider adjusting the memory limits in the deployment manifest

Backup failures

If backup fails:

1. Check the backup job logs:

   kubectl logs -l app.kubernetes.io/name=victoriametrics-backup
   

2. Verify S3 bucket permissions (IAM role should have S3 access)

3. Ensure there's sufficient disk space on the control node

Victoria Logs

Victoria Logs is a centralized log aggregation system that collects logs from all nodes in your easy-db-lab cluster. It provides a unified way to search and analyze logs from Cassandra, ClickHouse, and system services.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                     All Nodes (DaemonSet)                    │
├─────────────────────────────────────────────────────────────┤
│   /var/log/*              journald                          │
│   /mnt/db1/cassandra/logs/*.log                             │
│   /mnt/db1/clickhouse/logs/*.log                            │
└──────────────────────────┬──────────────────────────────────┘
                           │
                           ▼
              ┌────────────────────────┐
              │  OTel Collector        │
              │  (DaemonSet)           │
              │  filelog + journald    │
              └───────────┬────────────┘
                          │
┌─────────────────────────┼─────────────────────────┐
│   Control Node          │                          │
├─────────────────────────┼─────────────────────────┤
│                         ▼                          │
│              ┌──────────────────┐                  │
│              │  Victoria Logs   │                  │
│              │    (:9428)       │                  │
│              └────────┬─────────┘                  │
└───────────────────────┼────────────────────────────┘
                        │
                        ▼
              ┌──────────────────┐
              │  easy-db-lab     │
              │  logs query      │
              └──────────────────┘

Components

Victoria Logs Server

Victoria Logs runs on the control node as a Kubernetes deployment:

• Port: 9428 (HTTP API)
• Storage: Local ephemeral storage
• Retention: 7 days (configurable)
• Location: Control node only (node-role.kubernetes.io/control-plane)

OTel Collector

The OpenTelemetry Collector collects logs from all sources and forwards them to Victoria Logs.

The OTel Collector runs as a DaemonSet on every node (Cassandra, stress, control) to collect:

Log Sources

Each log entry is tagged with a source field:

Querying Logs

Using the CLI

The easy-db-lab logs query command provides a unified interface:

# Query all logs from the last hour
easy-db-lab logs query

# Filter by source
easy-db-lab logs query --source cassandra
easy-db-lab logs query --source clickhouse
easy-db-lab logs query --source systemd

# Filter by host
easy-db-lab logs query --source cassandra --host db0

# Filter by systemd unit
easy-db-lab logs query --source systemd --unit docker.service

# Search for text
easy-db-lab logs query --grep "OutOfMemory"
easy-db-lab logs query --grep "ERROR"

# Time range and limit
easy-db-lab logs query --since 30m --limit 500
easy-db-lab logs query --since 1d

# Raw LogsQL query
easy-db-lab logs query -q 'source:cassandra AND host:db0'

Query Options

Using the HTTP API

Victoria Logs exposes a REST API on port 9428. Access it through the SOCKS proxy:

source env.sh
with-proxy curl "http://control0:9428/select/logsql/query?query=source:cassandra&time=1h&limit=100"

Using Grafana

Victoria Logs is configured as a datasource in Grafana. You can use it in two ways:

Log Investigation Dashboard

The Log Investigation dashboard is designed for interactive log analysis during investigations. Access it at Grafana → Dashboards → Log Investigation.

Filter variables (dropdowns at the top):

Panels:

• Log Volume — time-series bar chart showing log count over time, broken down by source. Helps identify spikes and anomalies at a glance.
• Logs — scrollable log viewer with timestamps, source labels, and expandable log details. Click any log entry to see all available fields.

Tips:

• Use the ad-hoc Filters variable to filter by host, unit, component, or any other field without needing a dedicated dropdown.
• The dashboard auto-refreshes every 10 seconds by default. Adjust or disable via the refresh picker in the top-right corner.
• Combine multiple filters to narrow down — e.g., set Node Role to db, Source to cassandra, Level to Error to see only Cassandra errors on database nodes.
• To search for exec job logs, set Source to tool-runner and use the Search box for the job name.

Explore Mode

For ad-hoc queries beyond what the dashboard provides:

1. Access Grafana at http://control0:3000 (via SOCKS proxy)
2. Navigate to Explore
3. Select "VictoriaLogs" datasource
4. Use LogsQL syntax for queries

LogsQL Query Syntax

Victoria Logs uses LogsQL for querying. Basic syntax:

# Simple field match
source:cassandra

# Multiple conditions (AND)
source:cassandra AND host:db0

# Text search
"OutOfMemory"

# Combine field match with text search
source:cassandra AND "Exception"

# Time filter (in addition to --since)
_time:1h

For full LogsQL documentation, see the Victoria Logs documentation.

Deployment

Victoria Logs and the OTel Collector are automatically deployed when you run:

easy-db-lab k8 apply

This deploys:

• Victoria Logs server on the control node
• OTel Collector DaemonSet on all nodes
• Grafana datasource configuration

Verifying the Setup

Check that all components are running:

source env.sh
kubectl get pods -l app.kubernetes.io/name=victorialogs
kubectl get pods -l app.kubernetes.io/name=otel-collector

Test connectivity:

# Check Victoria Logs health
with-proxy curl http://control0:9428/health

# Query recent logs
easy-db-lab logs query --limit 10

Troubleshooting

No logs appearing

1. Verify OTel Collector pods are running:

   kubectl get pods -l app.kubernetes.io/name=otel-collector
   kubectl logs -l app.kubernetes.io/name=otel-collector
   

2. Check Victoria Logs is healthy:

   with-proxy curl http://control0:9428/health
   

3. Verify the cluster-config ConfigMap exists:

   kubectl get configmap cluster-config -o yaml
   

Connection errors

The logs query command uses the internal SOCKS5 proxy to connect to Victoria Logs. If you see connection errors:

1. Ensure the cluster is running: easy-db-lab status
2. The proxy is started automatically when needed
3. Check that control node is accessible: ssh control0 hostname

Listing Backups

List available VictoriaLogs backups in S3:

easy-db-lab logs ls

This displays a summary table of all backups grouped by timestamp, showing the number of files and total size for each.

Importing Logs to an External Instance

Stream logs from the running cluster's VictoriaLogs to an external VictoriaLogs instance via the jsonline API:

# Import all logs
easy-db-lab logs import --target http://victorialogs:9428

# Import only specific logs
easy-db-lab logs import --target http://victorialogs:9428 --query 'source:cassandra'

This is useful for exporting logs at the end of test runs when running easy-db-lab from a Docker container. Unlike binary backups, this approach streams data via HTTP and can target any reachable VictoriaLogs instance.

Options

Backup

Victoria Logs data can be backed up to S3 for disaster recovery using consistent snapshots.

Creating a Backup

# Backup to cluster's default S3 bucket
easy-db-lab logs backup

# Backup to a custom S3 location
easy-db-lab logs backup --dest s3://my-backup-bucket/victorialogs

By default, backups are stored at: s3://{cluster-bucket}/victorialogs/{timestamp}/

Use --dest to override the destination bucket and path.

How It Works

The backup uses VictoriaLogs' snapshot API to create consistent, point-in-time copies:

1. Create snapshots — calls the VictoriaLogs snapshot API to create read-only snapshots of all active log partitions
2. Sync to S3 — uploads each snapshot directory to S3 using aws s3 sync
3. Cleanup — deletes the snapshots from disk to free space (runs even if the sync step fails)

Using snapshots ensures data consistency, since VictoriaLogs may be actively writing to its data directory during the backup.

What Gets Backed Up

• All log partitions (organized by date)
• Complete log history up to retention period (7 days default)

Notes

• The process is non-disruptive; log ingestion continues during backup
• Snapshot cleanup always runs, even if the S3 upload fails, to avoid filling disk
• Persistent storage at /mnt/db1/victorialogs ensures logs survive pod restarts

Kubernetes

easy-db-lab uses K3s to provide a lightweight Kubernetes cluster for deploying supporting services like ClickHouse, monitoring, and stress testing workloads.

Overview

K3s is automatically installed on all nodes during provisioning:

• Control node: Runs the K3s server (Kubernetes control plane)
• Cassandra nodes: Run as K3s agents with label type=db
• Stress nodes: Run as K3s agents with label type=app

Accessing the Cluster

kubectl

After running source env.sh, kubectl is automatically configured:

source env.sh
kubectl get nodes
kubectl get pods -A

The kubeconfig is downloaded to your working directory and kubectl is configured to use the SOCKS5 proxy for connectivity.

k9s

k9s provides a terminal-based UI for Kubernetes:

source env.sh
k9s

k9s is pre-configured to use the correct kubeconfig and proxy settings.

Port Forwarding

easy-db-lab uses a SOCKS5 proxy for accessing the private Kubernetes cluster.

Starting the Proxy

The proxy starts automatically when you source the environment:

source env.sh

Manual Proxy Control

# Start the SOCKS5 proxy
start-socks5

# Check proxy status
socks5-status

# Stop the proxy
stop-socks5

Running Commands Through the Proxy

Commands like kubectl and k9s automatically use the proxy. For other commands:

# Route any command through the proxy
with-proxy curl http://10.0.1.50:8080/api

Pushing Docker Images with Jib

easy-db-lab includes a private Docker registry accessible via HTTPS. You can push custom images using Jib.

Gradle Configuration

Add Jib to your build.gradle.kts:

plugins {
    id("com.google.cloud.tools.jib") version "3.4.0"
}

jib {
    from {
        image = "eclipse-temurin:21-jre"
    }
    to {
        // Use the control node's registry
        image = "control0:5000/my-app"
        tags = setOf("latest", project.version.toString())
    }
    container {
        mainClass = "com.example.MainKt"
    }
}

Pushing to the Registry

# Build and push to the cluster registry
./gradlew jib

# Or build locally first
./gradlew jibDockerBuild

Using Images in Kubernetes

Reference your pushed images in Kubernetes manifests:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: my-app
    image: control0:5000/my-app:latest

Node Labels

Nodes are automatically labeled for workload scheduling:

Using Node Selectors

Schedule pods on specific node types:

apiVersion: v1
kind: Pod
metadata:
  name: stress-worker
spec:
  nodeSelector:
    type: app
  containers:
  - name: worker
    image: my-stress-tool:latest

Useful Commands

# List all nodes
kubectl get nodes

# List pods in all namespaces
kubectl get pods -A

# Watch pod status
kubectl get pods -w

# View logs
kubectl logs <pod-name>

# Execute command in pod
kubectl exec -it <pod-name> -- /bin/bash

# Port forward a service locally
kubectl port-forward svc/my-service 8080:80

Architecture

Networking

• K3s server runs on the control node
• All nodes communicate over the private VPC network
• External access is via SOCKS5 proxy through the control node

Storage

• Local path provisioner for persistent volumes
• Data stored on node-local NVMe drives at /mnt/db1/

Kubeconfig

The kubeconfig file is:

• Downloaded automatically during cluster setup
• Stored as kubeconfig in your working directory
• Backed up to S3 for recovery

Network Connectivity

This guide covers how to connect to your easy-db-lab cluster from your local machine.

Overview

easy-db-lab clusters run in a private AWS VPC. By default, the VPC uses 10.0.0.0/16, but you can customize this:

easy-db-lab init --cidr 10.14.0.0/20 ...

There are two methods to access your cluster:

Tailscale VPN (Recommended)

Tailscale provides a persistent VPN connection to your cluster. Once connected, you can access cluster resources directly—no proxy configuration needed.

Why Tailscale?

• Native access - Use any tool (browsers, kubectl, ssh) without proxy configuration
• Persistent - Connection survives terminal sessions
• Team sharing - Share cluster access with teammates
• Reliable - No SSH tunnels to maintain or reconnect

Setup (One-Time)

Step 1: Configure Tailscale ACL

Go to Tailscale ACL Editor and add:

{
  "tagOwners": {
    "tag:easy-db-lab": ["autogroup:admin"]
  },
  "autoApprovers": {
    "routes": {
      "10.0.0.0/8": ["tag:easy-db-lab"]
    }
  }
}

The autoApprovers section automatically approves subnet routes, so you don't need to manually approve each cluster.

Step 2: Create OAuth Client

1. Go to Tailscale OAuth Settings
2. Click Generate OAuth Client
3. Configure:
   • Description: easy-db-lab
   • Scopes: Select Devices: Write
   • Tags: Add tag:easy-db-lab
4. Click Generate and save the Client ID and Client Secret

Step 3: Configure easy-db-lab

easy-db-lab setup-profile

Enter your Tailscale OAuth credentials when prompted.

Usage

Tailscale starts automatically with easy-db-lab up. Once connected:

# Direct access to private IPs
ssh ubuntu@10.0.1.50
curl http://10.0.1.50:9428/health
kubectl get pods

# Web UIs work directly in your browser
# http://10.0.1.50:3000 (Grafana)

Manual Control

easy-db-lab tailscale start
easy-db-lab tailscale status
easy-db-lab tailscale stop

Troubleshooting Tailscale

"requested tags are invalid or not permitted" - Add the tag to your ACL (Step 1).

Can't reach private IPs - Check subnet route is approved in Tailscale admin, or add autoApprovers to your ACL.

Using a custom tag:

easy-db-lab tailscale start --tag tag:my-custom-tag

SOCKS Proxy (Alternative)

If you don't want to set up Tailscale, the SOCKS proxy provides connectivity via an SSH tunnel through the control node.

┌─────────────────┐     SSH Tunnel      ┌──────────────┐
│  Your Machine   │ ──────────────────► │ Control Node │
│  localhost:1080 │                     │  (control0)  │
└────────┬────────┘                     └──────┬───────┘
         │                                     │
    SOCKS5 Proxy                         Private VPC
         │                                     │
         ▼                                     ▼
   kubectl, curl                          VPC network

Quick Start

source env.sh
kubectl get pods
curl http://control0:9428/health

The proxy starts automatically when you load the environment.

Proxied Commands

These commands are automatically configured to use the proxy after source env.sh:

Manual Proxy Usage

For other commands, use the with-proxy wrapper:

with-proxy wget http://10.0.1.50:8080/api
with-proxy http http://control0:3000/api/health

Kit commands over SOCKS

Kit lifecycle commands work transparently on SOCKS-only clusters — no extra flags or setup — whether or not Tailscale is enabled.

kit <name> start / stop and other lifecycle phases. These run kubectl and helm on your machine to apply manifests, wait on pods, and read pod state. On a SOCKS-only cluster the private Kubernetes API is reachable only through the tunnel, so easy-db-lab hands those local kubectl/helm invocations a throwaway kubeconfig carrying a proxy-url: socks5://127.0.0.1:<port> on the cluster entry. That routes only kubectl/helm through the tunnel — aws, curl, and anything else a kit step runs stay direct. The proxied kubeconfig is derived per command and deleted when the command finishes; the workspace kubeconfig is never modified.

easy-db-lab postgres start

sql. The sql command opens a short-lived in-process loopback bridge that forwards the JDBC connection through the existing tunnel to the database's private IP, then tears it down when the query finishes. This works for raw-TCP drivers (PostgreSQL, MySQL) as well as HTTP-based ones (Trino, ClickHouse):

easy-db-lab postgres sql "SELECT 1"

On Tailscale-enabled clusters the same commands connect directly to the private IP with no proxy, so behavior is identical either way. In neither path are the JVM-global socksProxyHost / socksProxyPort properties touched — routing is scoped per client.

Browser Access

Configure your browser's SOCKS5 proxy:

Then access cluster services:

• Grafana: http://control0:3000
• Victoria Metrics: http://control0:8428
• Victoria Logs: http://control0:9428

Proxy Management

start-socks5          # Start proxy
start-socks5 1081     # Start on different port
socks5-status         # Check status
stop-socks5           # Stop proxy

Host Key Verification

The sshConfig generated for your cluster sets UserKnownHostsFile=/dev/null alongside StrictHostKeyChecking=no. ssh — and therefore the SOCKS tunnel, which is launched with ssh -N -D against that config — never reads or writes your ~/.ssh/known_hosts for cluster nodes.

This matters because AWS recycles public IPs across ephemeral cluster lifetimes. Without this setting, a recycled IP that previously belonged to a different cluster (with a different host key) would make ssh hard-fail with REMOTE HOST IDENTIFICATION HAS CHANGED — StrictHostKeyChecking=no only auto-adds unknown hosts, it doesn't override a changed key for a host already recorded. Every cluster is short-lived and gets fresh host keys on every provision, so there is nothing to verify against across runs.

If you connected to easy-db-lab clusters before this change, their host keys may still be in your ~/.ssh/known_hosts. They're no longer read by the tool, so you can prune them any time — look for entries matching your cluster's Hostname lines in the generated sshConfig.

Tunnel Failures

If the SOCKS tunnel can't be established, the command that needed it fails immediately with a non-zero exit code rather than silently continuing against a proxy port nothing is listening on. The error names the SOCKS proxy as the failing component and points at socks5-proxy.log in your cluster workspace directory — that file holds the ssh -v transcript from the tunnel attempt and is the fastest way to find the real cause (a host-key mismatch, a security group blocking port 22, the control node not yet accepting SSH, and so on).

easy-db-lab status is the one exception: it still reports everything it can reach over SSH and the AWS SDK even when the tunnel is down, marking only the sections that require the private Kubernetes API (stress jobs, ClickHouse) as unavailable. See the status command reference for details.

Troubleshooting SOCKS Proxy

"Connection refused" errors:

socks5-status              # Check if running
start-socks5               # Start if needed
ssh control0 hostname      # Verify SSH works

Proxy not working after network change:

stop-socks5
source env.sh

Port already in use:

lsof -i :1080         # Check what's using it
start-socks5 1081     # Use different port

Commands timing out:

1. Check cluster status: easy-db-lab status
2. Verify SSH works: ssh control0 hostname
3. Restart proxy: stop-socks5 && start-socks5

easy-db-lab command fails with a SOCKS proxy error: As of this change, easy-db-lab commands that need the tunnel (up, kit commands, Grafana config updates, etc.) abort immediately if the tunnel can't be established, instead of silently running against a dead proxy port. Check socks5-proxy.log in your cluster workspace directory for the ssh -v transcript — it shows the actual reason the tunnel failed. See Host Key Verification above for the most common cause on a newly-provisioned cluster.

Comparison

Shell Aliases

After running source env.sh, you get access to several helpful aliases and functions for managing your cluster.

SSH Aliases

SSH aliases for all Cassandra nodes are automatically created as c0-cN. The ssh command is not required. For example:

c0 nodetool status

This runs nodetool status on the first Cassandra node.

Cluster Management Functions

Examples

Run a command on all nodes

c-all "df -h"

Check cluster status

c-status

Collect artifacts for performance testing

c-collect-artifacts my-test-run

This is useful when doing performance testing to capture the state of the system at a given moment.

Graceful Rolling Restarts

For true rolling restarts, we recommend using cstar instead of c-restart.

Server

easy-db-lab includes a server mode that provides AI assistant integration via MCP (Model Context Protocol), REST status endpoints, and live metrics streaming. This enables Claude to directly interact with your clusters, and provides programmatic access to cluster status.

The server exposes tools for all supported databases — Cassandra, ClickHouse, OpenSearch, and Spark — as well as cluster lifecycle management and observability.

Starting the Server

To start the server, run:

easy-db-lab server

By default, the server picks an available port. To specify a port:

easy-db-lab server --port 8888

The server automatically generates a .mcp.json configuration file in the current directory with the connection details.

Adding to Claude Code

Once the server is running, start Claude Code from the same directory:

claude

Claude Code automatically detects and uses the .mcp.json file generated by the server.

Available Tools

The server exposes commands annotated with @McpCommand as MCP tools to Claude. Tool names use underscores and are derived from the command's package namespace.

Cluster Lifecycle

Cassandra Management

Cassandra Stress Testing

ClickHouse

OpenSearch

Spark

Kubernetes

Utilities

Tool Naming Convention

MCP tool names are derived from the command's package location:

• Top-level commands: status, hosts, ip, clean, init, up
• Cassandra commands: cassandra_ prefix (e.g., cassandra_start, cassandra_use)
• Nested commands: cassandra_stress_ prefix (e.g., cassandra_stress_start)
• Hyphens become underscores: update-config → cassandra_update_config

Benefits of Server Integration

Example Workflow

1. Start the server in one terminal:

   easy-db-lab server
   

2. In another terminal, start Claude Code from the same directory:

   claude
   

   Claude Code automatically detects the .mcp.json file generated by the server.

3. Ask Claude to help manage your cluster:

   • "Initialize a new 5-node cluster with i4i.xlarge instances"
   • "Check the status of all nodes"
   • "Select Cassandra version 5.0 and start it"
   • "Start a KeyValue stress test for 1 hour"
   • "Deploy ClickHouse and check its status"
   • "Create an OpenSearch domain and monitor its progress"
   • "Submit a Spark job to the EMR cluster"

Live Metrics Streaming

When Redis is configured via the EASY_DB_LAB_REDIS_URL environment variable, the server publishes live cluster metrics to the Redis pub/sub channel every 5 seconds. Metrics are queried from VictoriaMetrics using the same PromQL expressions as the Grafana dashboards.

Enabling

export EASY_DB_LAB_REDIS_URL=redis://localhost:6379/easydblab-events
easy-db-lab server

Metrics events are published to the same channel as command events. Consumers filter by the event.type field.

Event Types

Only metrics for running services are published. If the cluster is running ClickHouse instead of Cassandra, no Cassandra metrics events are emitted.

Metrics.System

Published every 5 seconds with per-node CPU, memory, disk I/O, and filesystem metrics:

{
  "timestamp": "2026-03-08T14:22:05.123Z",
  "commandName": "server",
  "event": {
    "type": "Metrics.System",
    "nodes": {
      "db-0": {
        "cpuUsagePct": 34.2,
        "memoryUsedBytes": 17179869184,
        "diskReadBytesPerSec": 52428800.0,
        "diskWriteBytesPerSec": 104857600.0,
        "filesystemUsedPct": 45.2
      },
      "db-1": {
        "cpuUsagePct": 28.7,
        "memoryUsedBytes": 16106127360,
        "diskReadBytesPerSec": 41943040.0,
        "diskWriteBytesPerSec": 83886080.0,
        "filesystemUsedPct": 42.8
      }
    }
  }
}

Metrics.Cassandra

Published every 5 seconds when the cluster is running Cassandra:

{
  "timestamp": "2026-03-08T14:22:05.187Z",
  "commandName": "server",
  "event": {
    "type": "Metrics.Cassandra",
    "readP99Ms": 1.247,
    "writeP99Ms": 0.832,
    "readOpsPerSec": 15234.5,
    "writeOpsPerSec": 12087.3,
    "compactionPending": 3,
    "compactionCompletedPerSec": 1.5,
    "compactionBytesWrittenPerSec": 52428800.0
  }
}

Field Reference

System — per node:

Cassandra — cluster-wide:

Auto-Shutdown on Infrastructure Removal

When running the server in unattended or automated scenarios, you can enable automatic shutdown if the cluster's AWS infrastructure is torn down:

easy-db-lab server --auto-shutdown

When --auto-shutdown is set, the server checks whether the cluster VPC still exists on each status refresh cycle (controlled by --refresh). If the VPC is no longer found, the server emits a shutdown event and exits cleanly with code 0.

This is useful when:

• Running the server alongside an automated test workflow that tears down infrastructure when done
• Leaving the server running overnight and wanting it to stop automatically after easy-db-lab down

Note: The check is skipped if no cluster state exists or the VPC name cannot be determined. AWS API errors during the check are logged and ignored — only a confirmed "VPC not found" result triggers shutdown.

Notes

• The server requires Docker to be installed
• Your AWS profile must be configured (easy-db-lab setup-profile)
• The server runs in the foreground and logs to stdout
• Use Ctrl+C to stop the server

Command Reference

Complete reference for all easy-db-lab commands.

Global Options

Setup Commands

setup-profile

Set up user profile interactively.

easy-db-lab setup-profile

Aliases: setup

Guides you through:

• Email and AWS credentials collection
• AWS credential validation
• Key pair generation
• IAM role creation
• Packer VPC infrastructure setup
• AMI validation/building

show-iam-policies

Display IAM policies with your account ID populated.

easy-db-lab show-iam-policies [policy-name]

Aliases: sip

build-image

Build both base and Cassandra AMI images.

easy-db-lab build-image [options]

Cluster Lifecycle Commands

init

Initialize a directory for easy-db-lab.

easy-db-lab init [cluster-name] [options]

The database and application node groups are configured through a namespaced --db.* / --app.* scheme. Every pre-existing flag continues to work as an alias carrying its established default. When both a namespaced option and its legacy alias are supplied for the same setting, the namespaced option always wins, regardless of the order they appear on the command line.

Architecture is no longer a flag. Each node group's CPU architecture is derived automatically from that group's resolved instance type at init time (via the EC2 DescribeInstanceTypes SupportedArchitectures field) and persisted per group in cluster state. A cluster whose database and application groups have different architectures is provisioned correctly, each group booting from the AMI for its own architecture. An instance type whose architecture cannot be determined fails at init, before any instance is created.

up

Provision AWS infrastructure.

easy-db-lab up [options]

Creates: VPC, EC2 instances, K3s cluster. Configures the account S3 bucket for this cluster.

up fails fast. If any provisioning step fails — EC2 setup, K3s, node labeling, the local-storage/local-storage-wfc StorageClasses, the observability stack, Tailscale, and so on — the command exits non-zero and stops rather than continuing with a partially-provisioned cluster. EC2 instances that were already launched are left running; there is no automatic rollback. Reclaim them with easy-db-lab down, fix the underlying issue, and re-run up.

down

Shut down AWS infrastructure.

easy-db-lab down [vpc-id] [options]

clean

Clean up generated files from the current directory.

easy-db-lab clean

hosts

List all hosts in the cluster.

easy-db-lab hosts

status

Display full environment status.

easy-db-lab status

status is the one command that degrades instead of failing outright when the SOCKS proxy tunnel can't be established. It still reports EC2, VPC, security groups, Spark/EMR, OpenSearch, S3, kits, observability URLs, and database versions — the last read directly over SSH, which never uses the tunnel. Only the sections that require the private Kubernetes API (stress jobs, ClickHouse) are marked unavailable, each stating the proxy failure as the reason. status still exits non-zero when degraded, so a partial report is never mistaken for a healthy cluster by a script. See Network Connectivity for how to diagnose a tunnel that won't come up.

Cassandra Commands

All Cassandra commands are available under the cassandra subcommand group.

cassandra use

Select a Cassandra version.

easy-db-lab cassandra use <version> [options]

Versions: 3.0, 3.11, 4.0, 4.1, 5.0, 5.0-HEAD, 6.0-HEAD, trunk

cassandra write-config

Generate a new configuration patch file.

easy-db-lab cassandra write-config [filename] [options]

Aliases: wc

cassandra update-config

Apply configuration patch to all nodes.

easy-db-lab cassandra update-config [options]

Aliases: uc

cassandra download-config

Download configuration files from nodes.

easy-db-lab cassandra download-config [options]

Aliases: dc

cassandra start

Start Cassandra on all nodes.

easy-db-lab cassandra start [options]

Use --sidecar-image to test a fork or specific version:

easy-db-lab cassandra start --sidecar-image ghcr.io/myfork/cassandra-sidecar:my-branch

cassandra stop

Stop Cassandra on all nodes.

easy-db-lab cassandra stop [options]

cassandra restart

Restart Cassandra on all nodes.

easy-db-lab cassandra restart [options]

cassandra list

List available Cassandra versions.

easy-db-lab cassandra list

Aliases: ls

Cassandra Stress Commands

Stress testing commands under cassandra stress.

cassandra stress start

Start a stress job on Kubernetes.

easy-db-lab cassandra stress start [options]

Aliases: run

cassandra stress stop

Stop and delete stress jobs.

easy-db-lab cassandra stress stop [options]

cassandra stress status

Check status of stress jobs.

easy-db-lab cassandra stress status

cassandra stress logs

View logs from stress jobs.

easy-db-lab cassandra stress logs [options]

cassandra stress list

List available workloads.

easy-db-lab cassandra stress list

cassandra stress fields

List available field generators.

easy-db-lab cassandra stress fields

cassandra stress info

Show information about a workload.

easy-db-lab cassandra stress info <workload>

Utility Commands

exec

Execute commands on remote hosts via systemd-run. Tool output is captured by the systemd journal and shipped to VictoriaLogs via a dedicated journald OTel collector, with accurate timestamps for cross-service log correlation.

exec run

Run a command on remote hosts (foreground by default).

# Foreground (blocks until complete, shows output)
easy-db-lab exec run -t cassandra -- ls /mnt/db1

# Background (returns immediately, tool keeps running)
easy-db-lab exec run --bg -t cassandra -- inotifywait -m /mnt/db1/data

# Background with custom name
easy-db-lab exec run --bg --name watch-imports -t cassandra -- inotifywait -m /mnt/db1/data

exec list

List running background tools on remote hosts.

easy-db-lab exec list
easy-db-lab exec list -t cassandra

exec stop

Stop a named background tool.

easy-db-lab exec stop watch-imports
easy-db-lab exec stop watch-imports -t cassandra

ip

Get IP address for a host by alias.

easy-db-lab ip <alias>

version

Display the easy-db-lab version.

easy-db-lab version

repl

Start interactive REPL.

easy-db-lab repl

server

Start the server for Claude Code integration, REST status endpoints, and live metrics.

easy-db-lab server

See Server for details.

Kubernetes Commands

k8 apply

Apply observability stack to K8s cluster.

easy-db-lab k8 apply

Dashboard Commands

dashboards generate

Extract all Grafana dashboard manifests (core and ClickHouse) from JAR resources to the local k8s/ directory. Useful for rapid dashboard iteration without re-running init.

easy-db-lab dashboards generate

dashboards upload

Apply all Grafana dashboard manifests and the datasource ConfigMap to the K8s cluster. Extracts dashboards, creates the grafana-datasources ConfigMap with runtime configuration, and applies everything.

easy-db-lab dashboards upload

ClickHouse Commands

clickhouse start

Deploy ClickHouse cluster to K8s.

easy-db-lab clickhouse start [options]

clickhouse stop

Stop and remove ClickHouse cluster.

easy-db-lab clickhouse stop

clickhouse status

Check ClickHouse cluster status.

easy-db-lab clickhouse status

Spark Commands

spark submit

Submit Spark job to EMR cluster.

easy-db-lab spark submit [options]

spark status

Check status of a Spark job.

easy-db-lab spark status [options]

spark jobs

List recent Spark jobs on the cluster.

easy-db-lab spark jobs

spark logs

Download EMR logs from S3.

easy-db-lab spark logs [options]

OpenSearch Commands

opensearch start

Create an AWS OpenSearch domain.

easy-db-lab opensearch start [options]

opensearch stop

Delete the OpenSearch domain.

easy-db-lab opensearch stop

opensearch status

Check OpenSearch domain status.

easy-db-lab opensearch status

AWS Commands

aws vpcs

List all easy-db-lab VPCs.

easy-db-lab aws vpcs

Port Reference

This page documents the ports used by easy-db-lab and the services it provisions.

Cassandra Ports

Observability Ports (Control Node)

Cassandra Agent Ports

Observability Ports (All Nodes — DaemonSets)

Server

SSH

SSH access is configured automatically through the sshConfig file generated by source env.sh.

OpenTelemetry Instrumentation

easy-db-lab includes optional OpenTelemetry (OTel) instrumentation for distributed tracing and metrics. When enabled, traces and metrics are exported to an OTLP-compatible collector.

CLI Tool Instrumentation

The easy-db-lab CLI tool runs with the OpenTelemetry Java Agent, which automatically instruments:

• AWS SDK calls - EC2, S3, IAM, EMR, STS, OpenSearch operations
• HTTP clients - OkHttp and other HTTP libraries
• JDBC/Cassandra driver - Database operations
• JVM metrics - Memory, threads, garbage collection

Enabling Instrumentation

Set the OTEL_EXPORTER_OTLP_ENDPOINT environment variable to your OTLP collector endpoint:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
easy-db-lab up

When this environment variable is:

• Set: Traces and metrics are exported via gRPC to the specified endpoint
• Not set: The agent is still loaded but no telemetry is exported (minimal overhead)

The agent uses automatic instrumentation only - there is no custom manual instrumentation in the CLI tool code.

Cluster Node Instrumentation

The following instrumentation applies to cluster nodes (Cassandra, stress, Spark) and is separate from the CLI tool:

Node Role Labeling

The OTel Collector on cluster nodes uses the k8sattributes processor to read the K8s node label type and set it as the node_role resource attribute. This label is used by Grafana dashboards (e.g., System Overview) for hostname and service filtering.

The k8sattributes processor runs in the metrics/local and logs/local pipelines only. Remote metrics arriving via OTLP (e.g., from Spark nodes) already carry node_role and are not modified.

The processor requires RBAC access to the K8s API. The OTel Collector DaemonSet runs with a dedicated ServiceAccount (otel-collector) that has read-only access to pods and nodes.

Stress Job Metrics

When running cassandra-easy-stress as K8s Jobs, metrics are automatically collected via an OTel collector sidecar container. The sidecar scrapes the stress process's Prometheus endpoint (localhost:9500) and forwards metrics via OTLP to the node's OTel DaemonSet, which then exports them to VictoriaMetrics.

The Prometheus scrape job is named cassandra-easy-stress. The following labels are available in Grafana:

Short-lived stress commands (list, info, fields) do not include the sidecar since they complete quickly and don't produce meaningful metrics.

Spark JVM Instrumentation

EMR Spark jobs are auto-instrumented with the OpenTelemetry Java Agent (v2.25.0) and Pyroscope Java Agent (v2.3.0), both installed via an EMR bootstrap action. The OTel agent is activated through spark.driver.extraJavaOptions and spark.executor.extraJavaOptions.

Each EMR node also runs an OTel Collector as a systemd service, collecting host metrics (CPU, memory, disk, network) and receiving OTLP from the Java agents. The collector forwards all telemetry to the control node's OTel Collector via OTLP gRPC.

Key configuration:

• OTel Agent JAR: Downloaded by bootstrap action to /opt/otel/opentelemetry-javaagent.jar
• Pyroscope Agent JAR: Downloaded by bootstrap action to /opt/pyroscope/pyroscope.jar
• OTel Collector: Installed at /opt/otel/otelcol-contrib, runs as otel-collector.service
• Export protocol: OTLP/gRPC to localhost:4317 (local collector), which forwards to control node
• Logs exporter: OTLP (captures JVM log output)
• Service name: spark-<job-name> (set per job)
• Profiling: CPU, allocation (512k threshold), lock (10ms threshold) profiles in JFR format sent to Pyroscope server

Cassandra Sidecar Instrumentation

The Cassandra Sidecar process is instrumented with the OpenTelemetry Java Agent and Pyroscope Java Agent, matching the pattern used for Cassandra itself. Both agents are loaded via -javaagent flags set in /etc/default/cassandra-sidecar, which is written by the setup-instances command.

Key configuration:

• OTel Agent JAR: Installed by Packer to /usr/local/otel/opentelemetry-javaagent.jar
• Pyroscope Agent JAR: Installed by Packer to /usr/local/pyroscope/pyroscope.jar
• Service name: cassandra-sidecar (both OTel and Pyroscope)
• Export endpoint: localhost:4317 (local OTel Collector DaemonSet)
• Profiling: CPU, allocation (512k threshold), lock (10ms threshold) profiles sent to Pyroscope server
• Activation: Gated on /etc/default/cassandra-sidecar — the systemd EnvironmentFile=- directive makes it optional, so the sidecar starts normally without instrumentation if the file doesn't exist

Tool Runner Log Collection

Commands run via exec run are executed through systemd-run, which captures stdout and stderr to log files under /var/log/easydblab/tools/. The OTel Collector's filelog/tools receiver watches this directory and ships log entries to VictoriaLogs with the attribute source: tool-runner.

This provides automatic log capture for ad-hoc debugging tools (e.g., inotifywait, tcpdump, strace) run during investigations. Logs are queryable in VictoriaLogs and preserved in S3 backups via logs backup.

Key details:

• Log directory: /var/log/easydblab/tools/
• Source attribute: tool-runner (for filtering in VictoriaLogs queries)
• Foreground commands: Output displayed after completion, also logged to file
• Background commands (--bg): Output logged to file only, tool runs as a systemd transient unit

YACE CloudWatch Scrape

YACE (Yet Another CloudWatch Exporter) runs on the control node and scrapes AWS CloudWatch metrics for services used by the cluster. It uses tag-based auto-discovery with the easy_cass_lab=1 tag to find relevant resources.

YACE scrapes metrics for:

• S3 — bucket request/byte counts
• EBS — volume read/write ops and latency
• EC2 — instance CPU, network, disk
• OpenSearch — domain health, indexing, search metrics

EMR metrics are collected directly via OTel Collectors on Spark nodes (see Spark JVM Instrumentation above).

YACE exposes scraped metrics as Prometheus-compatible metrics on port 5001, which are then scraped by the OTel Collector and forwarded to VictoriaMetrics. This replaces the previous CloudWatch datasource in Grafana with a Prometheus-based approach, giving dashboards access to CloudWatch metrics through VictoriaMetrics queries.

Resource Attributes

Traces from the CLI tool and cluster nodes include the following resource attributes:

• service.name: Service identifier (e.g., easy-db-lab, cassandra-sidecar, spark-<job-name>)
• service.version: Application version (CLI tool only)
• host.name: Hostname

Configuration

The following environment variables are supported:

Additional standard OTel environment variables are supported by the agent. See the OpenTelemetry Java Agent documentation for details.

Example: Using with Jaeger

Start Jaeger with OTLP support:

docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  jaegertracing/all-in-one:latest

Export traces to Jaeger:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
easy-db-lab up

View traces at http://localhost:16686

Example: Using with Grafana Tempo

If you have Grafana Tempo running with OTLP gRPC ingestion:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo:4317
easy-db-lab up

Troubleshooting

No Traces Appearing

1. Verify the endpoint is correct and reachable
2. Check that the collector accepts gRPC OTLP (port 4317 is standard)
3. Look for OpenTelemetry agent logs on startup (use -Dotel.javaagent.debug=true to enable debug logging)

High Latency

Traces are batched before export (default 1 second delay). This is normal and reduces overhead.

Spark Observability Debugging

Diagnostic commands for troubleshooting Spark observability on EMR nodes. These require SSH access to the EMR master node (ssh hadoop@<master-public-dns>).

OTel Collector

# Check collector is running
sudo systemctl status otel-collector

# View collector config (verify control node IP)
cat /opt/otel/config.yaml

# Test connectivity to control node collector
curl -s -o /dev/null -w '%{http_code}' http://<control-ip>:4318

Spark Configuration

# Verify -javaagent flags and OTel env vars are present
cat /etc/spark/conf/spark-defaults.conf

# Verify agent JARs exist
ls -la /opt/otel/opentelemetry-javaagent.jar
ls -la /opt/pyroscope/pyroscope.jar

Runtime Verification (while a job is running)

# Confirm agents are attached to Spark JVMs
ps aux | grep javaagent

Pyroscope API (from any node that can reach control0)

# List all label names
curl \
  -H "Content-Type: application/json" \
  -d '{
      "end": '$(date +%s)000',
      "start": '$(expr $(date +%s) - 3600)000'
    }' \
  http://localhost:4040/querier.v1.QuerierService/LabelNames

# List values for a specific label
curl \
  -H "Content-Type: application/json" \
  -d '{
      "end": '$(date +%s)000',
      "name": "hostname",
      "start": '$(expr $(date +%s) - 3600)000'
    }' \
  http://localhost:4040/querier.v1.QuerierService/LabelValues

# Diff two profiles (compare workloads)
# POST to /querier.v1.QuerierService/Diff with left/right profile selectors
# See: left.labelSelector, right.labelSelector, profileTypeID, start/end

Grafana Explore Queries

# All metrics from Spark nodes
{node_role="spark"}

# JVM metrics only
{node_role="spark", __name__=~"jvm_.*"}

# List distinct JVM metric names
group({node_role="spark", __name__=~"jvm_.*"}) by (__name__)

# Filesystem usage (raw)
system_filesystem_usage_bytes{state="used", node_role="spark", mountpoint="/"}

JFR Format Reference

The Java Flight Recorder format is used by JVM-based profilers and supported by the Pyroscope Java integration.

When JFR format is used, query parameters behave differently:

• format should be set to jfr
• name contains the prefix of the application name. Since a single request may contain multiple profile types, the final application name is created by concatenating this prefix and the profile type. For example, if you send cpu profiling data and set name to my-app{}, it will appear in Pyroscope as my-app.cpu{}
• units is ignored — actual units depend on the profile types in the data
• aggregationType is ignored — actual aggregation type depends on the profile types in the data

Supported JFR Profile Types

• cpu — samples from runnable threads only
• itimer — similar to cpu profiling
• wall — samples from any thread regardless of state
• alloc_in_new_tlab_objects — number of new TLAB objects created
• alloc_in_new_tlab_bytes — size in bytes of new TLAB objects created
• alloc_outside_tlab_objects — number of new allocated objects outside any TLAB
• alloc_outside_tlab_bytes — size in bytes of new allocated objects outside any TLAB

JFR with Dynamic Labels

To ingest JFR data with dynamic labels:

1. Use multipart/form-data Content-Type
2. Send JFR data in a form file field called jfr
3. Send LabelsSnapshot protobuf message in a form file field called labels

message Context {
    // string_id -> string_id
    map<int64, int64> labels = 1;
}
message LabelsSnapshot {
    // context_id -> Context
    map<int64, Context> contexts = 1;
    // string_id -> string
    map<int64, string> strings = 2;
}

Where context_id is a parameter set in async-profiler.

Ingestion Examples

Simple profile upload:

printf "foo;bar 100\n foo;baz 200" | curl \
  -X POST \
  --data-binary @- \
  'http://localhost:4040/ingest?name=curl-test-app&from=1615709120&until=1615709130'

JFR profile with labels:

curl -X POST \
  -F jfr=@profile.jfr \
  -F labels=@labels.pb \
  "http://localhost:4040/ingest?name=curl-test-app&units=samples&aggregationType=sum&sampleRate=100&from=1655834200&until=1655834210&spyName=javaspy&format=jfr"

Future: Ad-hoc Profiling with async-profiler

async-profiler can capture JFR profiles on demand and upload them to Pyroscope with labels. This enables targeted profiling of specific Spark jobs or Cassandra operations to inspect exactly what is happening at the JVM level.

Common Issues

• No JVM metrics: Check ps aux | grep javaagent — if -javaagent flags are missing, spark.driver.extraJavaOptions may be overridden at job submission time (replaces spark-defaults.conf entirely).
• Collector retry errors at startup: Normal if the control node collector isn't ready yet. Should stabilize within a minute.
• Spark profiles missing hostname label: PYROSCOPE_LABELS env var must be set via spark-env classification with hostname=$(hostname -s).

Development Overview

Hello there. If you're reading this, you've probably decided to contribute to easy-db-lab or use the tools for your own work. Very cool.

Prerequisites

Install these locally before building:

• Java 21 or newer (Temurin) via SDKMAN — Java 21 is the project default, but the build also works on newer JDKs (e.g. 25). The build compiles with whatever JDK you have installed and always emits Java 21 bytecode, so artifacts still run on older JVMs. One caveat: static analysis (detekt, and therefore the full check task) must run on JDK 21 — detekt 1.23.8 cannot run under a JDK 25 runtime. Building, running, and testing the application all work on JDK 25; CI runs check on JDK 21.
• Kotlin and Gradle (the Gradle wrapper ./gradlew is committed)
• Docker — for TestContainers-backed integration tests
• mdbook + mdbook-admonish — for previewing documentation

The example Spark jobs and their Cassandra Analytics build live in the separate spark-examples repository, so this repo needs only a single JDK (21 or newer).

Local Configuration (.env)

Both bin/easy-db-lab and bin/end-to-end-test automatically load a .env file from the project root if one exists. This is the recommended way to set per-developer configuration without modifying committed scripts.

Setup

cp .env.example .env
# Edit .env with your values

.env is listed in .gitignore and will never be committed.

Supported Variables

Example .env:

AWS_PROFILE=sandbox-admin
# SIDECAR_IMAGE=102382809497.dkr.ecr.us-west-2.amazonaws.com/rustyrazorblade/cassandra-sidecar
# EASY_DB_LAB_INSTANCE_TYPE=c5d.4xlarge

Variables already exported in your shell always take precedence over .env.

Building the Project

With the required tools installed:

./gradlew assemble
./gradlew test

Documentation Preview

Preview documentation locally with live reload:

cd docs
mdbook serve

Then open http://localhost:3000 in your browser.

Project Structure

easy-db-lab is broken into several subprojects:

• Docker containers (prefixed with docker-)
• Documentation (the manual you're reading now)
• Utility code for downloading artifacts

Architecture

The project follows a layered architecture:

Commands (PicoCLI) → Services → External Systems (K8s, AWS, Filesystem)

Layer Responsibilities

• Commands (commands/): Lightweight PicoCLI execution units
• Services (services/, providers/): Business logic layer

For more details, see the project's CLAUDE.md file.

Docker Development

Building Docker Containers

Each container is versioned and can be built locally using the following:

./gradlew :PROJECT-NAME:buildDocker

Where PROJECT-NAME is one of the subproject directories you see in the top level.

Setup

We recommend updating your local Docker service to use 8GB of memory. This is necessary when running dashboard previews locally. The preview is configured to run multiple Cassandra containers at once.

Available Docker Projects

Check the root project directory for subprojects prefixed with docker- to see available containerized components.

Local Testing

To test containers locally:

1. Build the container:

   ./gradlew :docker-cassandra:buildDocker
   

2. Run the container:

   docker run -it <image-name>
   

Memory Requirements

Publishing

Pre-Release Checklist

1. First check CI to ensure the build is clean and green
2. Ensure the following environment variables are set:
   • DOCKER_USERNAME
   • DOCKER_PASSWORD
   • DOCKER_EMAIL

Publishing Steps

Build and Upload

./gradlew buildAll uploadAll

Post-Release

After publishing, bump the version in build.gradle.kts.

Container Publishing

Containers are automatically published to GitHub Container Registry (ghcr.io) when:

• A version tag (v*) is pushed
• PR Checks pass on main branch

See .github/workflows/publish-container.yml for details.

Documentation

Documentation is automatically built and deployed via GitHub Actions when changes are pushed to the docs/ directory on the main branch.

Building a Cassandra Ref On Demand

The Build Cassandra Ref workflow turns an arbitrary Apache Cassandra git ref (branch, tag, or commit SHA) into two reusable artifacts in a single CI run:

1. A GHCR Docker image that is a drop-in replacement for the Docker Official cassandra image — same docker-entrypoint.sh contract, CASSANDRA_* environment variables, exposed ports (7000/7001/7199/9042/9160), and /var/lib/cassandra data volume.
2. A binary tarball attached to a per-build GitHub release, consumable by the lab install path (cassandra_versions.yaml + install_cassandra.sh).

This is the self-service alternative to a manual local build or waiting on the nightly tarball matrix. It does not change cluster provisioning or the AMI/Packer pipeline — the produced artifacts are inert until a consumer references them.

Triggering the workflow

Run it from the Actions tab (Build Cassandra Ref → Run workflow) or via the CLI:

gh workflow run build-cassandra-ref.yml \
  -f ref=cassandra-5.0

Inputs

When jdk / base_image are left blank they are auto-mapped from the resolved Cassandra version: 4.x → JDK 11 / eclipse-temurin:11-jre, 5.0 → JDK 17 / eclipse-temurin:17-jre, 5.1+/trunk → JDK 21 / eclipse-temurin:21-jre. Supply an override when a branch needs a different JRE than its version maps to (for example when trunk's floor moves).

For 4.0/4.1 refs the build also passes -Duse.jdk11=true automatically: those branches default their ant build to JDK 8 and need that flag to compile under the auto-mapped JDK 11 (the same flag the Packer install path uses for 4.x). This version → JDK / ant-flags / base-image / tag mapping lives in .github/cassandra-image/resolve-build-plan.sh and is unit-tested by resolve-build-plan.test.sh (run via ./gradlew testCassandraBuildPlan).

How it works

Three sequential jobs make "build fails → nothing published" structural:

1. resolve — validates ref with git ls-remote (fails fast, naming the bad ref, if it does not exist), pins it to a full + short commit SHA, reads base.version from build.xml, and computes the build JDK, runtime base image, image tags, tarball name, and release tag. The ref-resolution logic (ls-remote / raw-SHA fallback / fail-fast naming the bad ref) lives in .github/cassandra-image/resolve-ref.sh and is unit-tested by resolve-ref.test.sh (run via ./gradlew testCassandraResolveRef) with the git ls-remote call stubbed, so the fail-fast path is covered without a live run.
2. build — checks out the exact resolved SHA, sets up the build JDK, runs ant artifacts on the runner, and uploads the tarball as an intra-workflow artifact.
3. publish (needs: [resolve, build]) — creates the per-build release with the tarball attached, builds the image from .github/cassandra-image/ with the tarball + resolved base image, pushes the tags to GHCR, runs a CQL smoke test against the pushed image, and writes the run summary.

Authentication to GHCR uses the repository's GITHUB_TOKEN (packages: write + contents: write) — no secrets are entered at trigger time.

Where the artifacts land

• Image: ghcr.io/<owner>/<repo>/cassandra, tagged with both an immutable sha-<short> tag and a moving sanitized-ref tag. When the build is driven by the nightly matrix (build-cassandra-set.yml), each image also gets a stable version-label tag (5.0-HEAD, 6.0-HEAD, trunk) that matches the tarball naming, giving a stable pull URL that always points at the latest build for that line. The latest tag is reserved for the main CLI image and is never produced here.
• Tarball: apache-cassandra-<version>-<short-sha>-bin.tar.gz, attached to a GitHub release tagged cassandra-<version>-<short-sha>.

The short SHA in every tag and asset name guarantees that distinct refs never overwrite each other's artifacts. Accumulated per-build releases are pruned manually.

The run summary surfaces the pullable image reference, the tarball download URL, and the resolved version + commit SHA.

Consuming the artifacts

Docker image

Drop it into a compose file in place of cassandra:<n> using the same CASSANDRA_* environment variables:

docker pull ghcr.io/<owner>/<repo>/cassandra:sha-<short>

Tarball (lab install path)

Pin the release's tarball URL in packer/cassandra/cassandra_versions.yaml:

- version: "my-branch"
  url: https://github.com/<owner>/<repo>/releases/download/cassandra-<version>-<short-sha>/apache-cassandra-<version>-<short-sha>-bin.tar.gz
  java: "17"
  python: "3.11.9"

install_cassandra.sh downloads the URL and expects it to unpack into a single top-level *cassandra* directory, which the ant artifacts tarball satisfies.

Image assembly

The image is assembled from repo-owned files in .github/cassandra-image/:

• Dockerfile — reproduces the Docker Official cassandra layout but injects the branch-built tarball (passed as the TARBALL build arg) onto the BASE_IMAGE JRE base instead of downloading and GPG-verifying a released tarball.
• docker-entrypoint.sh — a vendored, byte-identical copy of the official entrypoint. If upstream changes it, re-vendor in a deliberate PR; the CQL smoke test guards against contract drift.

Testing Guidelines

This document outlines the testing standards and practices for the easy-db-lab project.

Core Testing Principles

1. Use BaseKoinTest for Dependency Injection

All tests should extend BaseKoinTest to take advantage of automatic dependency injection setup and teardown.

class MyCommandTest : BaseKoinTest() {
    // Your test code here
}

BaseKoinTest provides:

• Automatic Koin lifecycle management
• Core modules that are always mocked (AWS, SSH, OutputHandler)
• Ability to add test-specific modules via additionalTestModules()

2. Use AssertJ for Assertions

Tests should use AssertJ assertions, not JUnit assertions. AssertJ provides more readable and powerful assertion methods.

// Good - AssertJ style
import org.assertj.core.api.Assertions.assertThat

assertThat(result).isNotNull()
assertThat(result.value).isEqualTo("expected")
assertThat(list).hasSize(3).contains("item1", "item2")

// Avoid - JUnit style
import org.junit.jupiter.api.Assertions.assertEquals

assertEquals("expected", result.value)

3. Create Custom Assertions for Non-Trivial Classes

When testing non-trivial classes, create custom AssertJ assertions to implement Domain-Driven Design in tests. This decouples business logic from implementation details and makes tests more maintainable during refactoring.

Custom Assertions Pattern

Custom assertions provide a fluent, domain-specific language for testing that improves readability and maintainability.

Example: Custom Assertion for a Domain Class

Here's a complete example showing how to create and use custom assertions:

// Domain class to be tested
data class CassandraNode(
    val nodeId: String,
    val datacenter: String,
    val rack: String,
    val status: NodeStatus,
    val tokens: Int
)

enum class NodeStatus {
    UP, DOWN, JOINING, LEAVING
}

// Custom assertion class
import org.assertj.core.api.AbstractAssert

class CassandraNodeAssert(actual: CassandraNode?) :
    AbstractAssert<CassandraNodeAssert, CassandraNode>(actual, CassandraNodeAssert::class.java) {

    companion object {
        fun assertThat(actual: CassandraNode?): CassandraNodeAssert {
            return CassandraNodeAssert(actual)
        }
    }

    fun hasNodeId(nodeId: String): CassandraNodeAssert {
        isNotNull
        if (actual.nodeId != nodeId) {
            failWithMessage("Expected node ID to be <%s> but was <%s>", nodeId, actual.nodeId)
        }
        return this
    }

    fun isInDatacenter(datacenter: String): CassandraNodeAssert {
        isNotNull
        if (actual.datacenter != datacenter) {
            failWithMessage("Expected datacenter to be <%s> but was <%s>", datacenter, actual.datacenter)
        }
        return this
    }

    fun hasStatus(status: NodeStatus): CassandraNodeAssert {
        isNotNull
        if (actual.status != status) {
            failWithMessage("Expected status to be <%s> but was <%s>", status, actual.status)
        }
        return this
    }

    fun isUp(): CassandraNodeAssert {
        return hasStatus(NodeStatus.UP)
    }

    fun isDown(): CassandraNodeAssert {
        return hasStatus(NodeStatus.DOWN)
    }

    fun hasTokenCount(tokens: Int): CassandraNodeAssert {
        isNotNull
        if (actual.tokens != tokens) {
            failWithMessage("Expected token count to be <%s> but was <%s>", tokens, actual.tokens)
        }
        return this
    }
}

// Usage in tests
import CassandraNodeAssert.Companion.assertThat

@Test
fun `test cassandra node configuration`() {
    val node = CassandraNode(
        nodeId = "node1",
        datacenter = "dc1",
        rack = "rack1",
        status = NodeStatus.UP,
        tokens = 256
    )

    // Fluent assertions with domain language
    assertThat(node)
        .hasNodeId("node1")
        .isInDatacenter("dc1")
        .isUp()
        .hasTokenCount(256)
}

Project-Wide Assertions Helper

Create a central assertions class to provide access to all custom assertions:

// MyProjectAssertions.kt
object MyProjectAssertions {

    // Cassandra domain assertions
    fun assertThat(actual: CassandraNode?): CassandraNodeAssert {
        return CassandraNodeAssert(actual)
    }

    fun assertThat(actual: Host?): HostAssert {
        return HostAssert(actual)
    }

    fun assertThat(actual: TFState?): TFStateAssert {
        return TFStateAssert(actual)
    }

    // Add more domain assertions as needed
}

Then import statically in tests:

import com.rustyrazorblade.easydblab.assertions.MyProjectAssertions.assertThat

@Test
fun `test complex scenario`() {
    val node = createTestNode()
    val host = createTestHost()

    // All domain assertions available through single import
    assertThat(node).isUp()
    assertThat(host).hasPrivateIp("10.0.0.1")
}

Benefits of Custom Assertions

1. Domain-Driven Design: Tests use business language, not implementation details
2. Refactoring Safety: Changes to class internals don't break test logic
3. Readability: Tests read like specifications
4. Reusability: Common assertions are centralized
5. Maintainability: Single place to update assertion logic
6. Type Safety: Compile-time checking of assertion methods

When to Create Custom Assertions

Create custom assertions for:

• Domain entities (e.g., Host, TFState, CassandraNode)
• Complex value objects with multiple properties
• Classes that appear in multiple test scenarios
• Any class where you find yourself writing repetitive assertion code

Testing Best Practices

1. Test Names: Use descriptive names with backticks

   @Test
   fun `should start cassandra node when status is DOWN`() { }
   

2. Test Structure: Follow Arrange-Act-Assert pattern

   @Test
   fun `test node startup`() {
       // Arrange
       val node = createTestNode(status = NodeStatus.DOWN)
   
       // Act
       val result = nodeManager.startNode(node)
   
       // Assert
       assertThat(result).isUp()
   }
   

3. Mock External Dependencies: Always mock AWS, SSH, and other external services

   class MyTest : BaseKoinTest() {
       override fun additionalTestModules() = listOf(
           module {
               single { mockRemoteOperationsService() }
           }
       )
   }
   

4. Test Edge Cases: Include tests for error conditions and boundary cases

5. Keep Tests Focused: Each test should verify one specific behavior

Testing Interactive Commands with TestPrompter

Commands that require user input (like setup-profile) can be tested deterministically using TestPrompter. This test utility replaces the real Prompter interface and returns predefined responses.

Basic Usage

class MyCommandTest : BaseKoinTest() {
    private lateinit var testPrompter: TestPrompter

    override fun additionalTestModules() = listOf(
        module {
            single<Prompter> { testPrompter }
        }
    )

    @BeforeEach
    fun setup() {
        // Configure responses - keys can be exact matches or partial matches
        testPrompter = TestPrompter(
            mapOf(
                "email" to "test@example.com",
                "region" to "us-west-2",
                "AWS Access Key" to "AKIAIOSFODNN7EXAMPLE",
            )
        )
    }

    @Test
    fun `should collect user credentials`() {
        // Run command that prompts for input
        val command = SetupProfile()
        command.call()

        // Verify prompts were called
        assertThat(testPrompter.wasPromptedFor("email")).isTrue()
        assertThat(testPrompter.wasPromptedFor("region")).isTrue()
    }
}

Response Matching

TestPrompter supports two matching strategies:

1. Exact match: The question text matches a key exactly
2. Partial match: The question text contains the key (case-insensitive)

val prompter = TestPrompter(
    mapOf(
        // Exact match - only matches "email" exactly
        "email" to "test@example.com",

        // Partial match - matches any question containing "AWS Profile"
        "AWS Profile" to "my-profile",
    )
)

Sequential Responses for Retry Testing

For testing retry logic (e.g., credential validation failures), use addSequentialResponses():

@Test
fun `should retry on invalid credentials`() {
    testPrompter = TestPrompter()

    // First call returns invalid credentials, second returns valid ones
    testPrompter.addSequentialResponses(
        "AWS Access Key",
        "invalid-key",      // First attempt
        "AKIAVALIDKEY123"   // Second attempt (after retry)
    )

    testPrompter.addSequentialResponses(
        "AWS Secret",
        "invalid-secret",
        "valid-secret-key"
    )

    val command = SetupProfile()
    command.call()

    // Verify the command handled retry correctly
    val callLog = testPrompter.getCallLog()
    val accessKeyCalls = callLog.filter { it.question.contains("Access Key") }
    assertThat(accessKeyCalls).hasSize(2)
}

Verifying Prompt Behavior

TestPrompter records all prompt calls for verification:

@Test
fun `should not prompt for credentials when using AWS profile`() {
    testPrompter = TestPrompter(
        mapOf(
            "AWS Profile" to "my-profile",  // Non-empty = use profile auth
        )
    )

    val command = SetupProfile()
    command.call()

    // Verify credential prompts were skipped
    assertThat(testPrompter.wasPromptedFor("Access Key")).isFalse()
    assertThat(testPrompter.wasPromptedFor("Secret")).isFalse()

    // Check detailed call log
    val callLog = testPrompter.getCallLog()
    assertThat(callLog).anyMatch { it.question.contains("email") }
}

TestPrompter API Reference

PromptCall Data Class

Each recorded call contains:

• question: The prompt question text
• default: The default value offered
• secret: Whether input was masked (for passwords)
• returnedValue: The value that was returned

Additional Resources

• AssertJ Documentation
• AssertJ Custom Assertions Guide
• Koin Testing Documentation

End-to-End Testing

easy-db-lab includes a comprehensive end-to-end test suite that validates the entire workflow from provisioning to teardown.

Running the Test

The end-to-end test is located at bin/end-to-end-test:

./bin/end-to-end-test --cassandra

Command-Line Options

Feature Flags

Testing and Inspection

Examples

# List all available test steps
./bin/end-to-end-test --list-steps

# Run full test with all features
./bin/end-to-end-test --all

# Run with Cassandra and pause before teardown
./bin/end-to-end-test --cassandra --wait

# Run with breakpoints at steps 5 and 15
./bin/end-to-end-test --cassandra --break 5,15

# Build custom AMI images and run test
./bin/end-to-end-test --build --cassandra

Test Steps

The test executes approximately 27 steps covering:

Infrastructure

1. Build project
2. Check version command
3. Build packer images (optional)
4. Set IAM policies
5. Initialize cluster
6. Setup kubectl
7. Wait for K3s ready
8. Verify K3s cluster

Registry and Storage

9. Test registry push/pull
10. List hosts
11. Verify S3 backup

Cassandra

12. Setup Cassandra
13. Verify Cassandra backup
14. Verify restore
15. Cassandra start/stop cycle
16. Test SSH and nodetool
17. Check Sidecar
18. Test exec command
19. Run stress test
20. Run stress K8s test

Optional Services

21. Submit Spark job (if --spark)
22. Check Spark status (if --spark)
23. Start ClickHouse (if --clickhouse)
24. Test ClickHouse (if --clickhouse)
25. Stop ClickHouse (if --clickhouse)
26. Start OpenSearch (if --opensearch)
27. Test OpenSearch (if --opensearch)
28. Stop OpenSearch (if --opensearch)

Observability and Cleanup

29. Test observability stack
30. Teardown cluster

Error Handling

When a test step fails, an interactive menu appears:

1. Retry from failed step - Resume from the point of failure
2. Start a shell session - Opens a shell with:
   • easy-db-lab commands available
   • rebuild - Rebuild just the project
   • rerun - Rebuild and resume from failed step
3. Tear down environment - Run easy-db-lab down --yes
4. Exit - Exit the script

AWS Requirements

The test requires:

• AWS profile with sufficient permissions
• VPC and subnet configuration
• S3 bucket for backups and logs

Default Configuration

• Instance count: 3 nodes
• Instance type: c5.2xlarge
• Cassandra version: 5.0 (when enabled)
• Spark workers: 2 (when enabled)

CI Integration

The end-to-end test is designed to run in CI environments:

• Supports non-interactive mode
• Returns appropriate exit codes
• Provides detailed logging
• Cleans up resources on failure

Spark Development

The example Spark jobs — the bulk writers and the Spark Cassandra Connector read/write examples — have moved to their own repository: spark-examples (locally ../spark-examples).

That repo is where you:

• build and test the job modules (common, bulk-writer-sidecar, bulk-writer-s3-iam, connector-writer, connector-read-write),
• build Apache Cassandra Analytics (the bulk-writer modules depend on it), and
• publish the shadow (fat) job jars.

This repository keeps the spark CLI commands that provision EMR and run jobs: easy-db-lab spark init/submit/status/logs/jobs/down. They submit a pre-built job jar to EMR — download the published jar from the spark-examples releases, then point spark submit --jar <path> at it. See the Spark user guide for provisioning EMR, submitting jobs, and debugging failed steps.

Log Infrastructure

This page documents the centralized logging infrastructure in easy-db-lab, including OTel for log collection and Victoria Logs for storage and querying.

Architecture Overview

┌──────────────────────────────────────────────────────────────────┐
│                          All Nodes                                │
├──────────────────────────────────────────────────────────────────┤
│  /var/log/system logs     │  /mnt/db1/container-logs/ (NVMe)     │
│  /mnt/db1/cassandra/logs/ │    K8s pod stdout/stderr             │
│  journald                 │    (symlinked from /var/log/pods)    │
│                           │                                       │
└──────────────┬────────────────────────────┬──────────────────────┘
               │                            │
               ▼                            ▼
              ┌────────────────────────────────────────┐
              │  OTel Collector (DaemonSet)             │      ┌──────────────────┐
              │  filelog/system  filelog/containers      │◀─────│  EMR Spark JVMs  │
              │  filelog/cassandra                       │ OTLP │  (OTel Java Agent│
              │  + OTLP receiver                        │      │   v2.25.0)       │
              └───────────────────┬─────────────────────┘      └──────────────────┘
                                  │
┌─────────────────────────────────┼────────────────────────────┐
│   Control Node                  │                             │
├─────────────────────────────────┼────────────────────────────┤
│                                 ▼                             │
│                    ┌──────────────────┐                       │
│                    │  Victoria Logs   │                       │
│                    │    (:9428)       │                       │
│                    └────────┬─────────┘                      │
└─────────────────────────────┼──────────────────────────────────┘
                              │
                              ▼
                    ┌──────────────────┐
                    │  easy-db-lab     │
                    │  logs query      │
                    └──────────────────┘

Components

OTel Collector DaemonSet

The OpenTelemetry Collector runs on all nodes as a DaemonSet, collecting logs from four separate pipelines:

• logs/local — host file-based logs:
  • System logs: /var/log/**/*.log, /var/log/messages, /var/log/syslog (excludes container log paths)
  • Tool runner logs: /var/log/easydblab/tools/*.log
  • Cassandra logs: /mnt/db1/cassandra/logs/*.log
• logs/containers — K8s pod stdout/stderr from all running pods, enriched with Kubernetes metadata (pod name, namespace, container name, kit label). Automatically covers any K8s-native kit without per-kit configuration. Logs are stored on NVMe at /mnt/db1/container-logs/ (symlinked from /var/log/pods) to keep the boot volume free.
• logs/otlp — logs pushed via OTLP from remote applications (e.g. EMR Spark JVMs)
• systemd journal — collected via a separate Fluent Bit DaemonSet (fluent-bit-journald)

All pipelines forward to Victoria Logs on the control node.

Spark OTel Java Agent (EMR)

When EMR Spark jobs are running, the Spark driver and executor JVMs are instrumented with the OpenTelemetry Java Agent (v2.25.0) via an EMR bootstrap action. The agent auto-instruments the JVMs and exports logs via OTLP to the control node's OTel Collector.

Logs appear in VictoriaLogs with a service.name attribute like spark-<job-name>, making it easy to filter logs for specific Spark jobs.

The data flow is: Spark JVM → OTel Java Agent → OTLP → OTel Collector (control node) → VictoriaLogs.

Victoria Logs

Victoria Logs runs on the control node and provides:

• Log storage with efficient compression
• LogsQL query language
• HTTP API for querying (port 9428)

Querying Logs

Using the CLI

# Query all logs from last hour
easy-db-lab logs query

# Filter by source
easy-db-lab logs query --source cassandra
easy-db-lab logs query --source systemd

# Filter by host
easy-db-lab logs query --source cassandra --host db0

# Filter by systemd unit
easy-db-lab logs query --source systemd --unit docker.service

# Search for text
easy-db-lab logs query --grep "OutOfMemory"

# Time range and limit
easy-db-lab logs query --since 30m --limit 500

# Raw Victoria Logs query (LogsQL syntax)
easy-db-lab logs query -q 'source:cassandra AND host:db0'

Log Stream Fields

Common fields (all sources):

K8s container log fields (from logs/containers pipeline):

Source-specific fields:

Troubleshooting

No logs appearing

1. Check Victoria Logs is running:

   kubectl get pods | grep victoria
   

2. Check OTel Collector is running:

   kubectl get pods | grep otel
   

3. Verify the cluster-config ConfigMap exists:

   kubectl get configmap cluster-config -o yaml
   

Connection errors

The logs query command uses the internal SOCKS5 proxy to connect to Victoria Logs. If you see connection errors:

1. Ensure the cluster is running: easy-db-lab status
2. The proxy is started automatically when needed
3. Check that control node is accessible: ssh control0 hostname

Ports

Pyroscope Configuration Parameters

Reference for Pyroscope server configuration. Source: Grafana Pyroscope docs.

How Configuration Works

Pyroscope is configured via a YAML file (-config.file flag) or CLI flags. CLI flags take precedence over YAML values. Environment variables can be used with -config.expand-env=true using ${VAR} or ${VAR:-default} syntax.

View current config at the /config HTTP API endpoint.

Key Configuration Sections

Top-Level

# Modules to load. 'all' enables single-binary mode.
[target: <string> | default = "all"]

api:
  [base-url: <string> | default = ""]

Server

HTTP on port 4040 (default), gRPC on port 9095 (default).

server:
  [http_listen_address: <string> | default = ""]
  [http_listen_port: <int> | default = 4040]
  [grpc_listen_port: <int> | default = 9095]
  [graceful_shutdown_timeout: <duration> | default = 30s]
  [http_server_read_timeout: <duration> | default = 30s]
  [http_server_write_timeout: <duration> | default = 30s]
  [http_server_idle_timeout: <duration> | default = 2m]
  [log_format: <string> | default = "logfmt"]  # logfmt or json
  [log_level: <string> | default = "info"]      # debug, info, warn, error
  [grpc_server_max_recv_msg_size: <int> | default = 4194304]
  [grpc_server_max_send_msg_size: <int> | default = 4194304]
  [grpc_server_max_concurrent_streams: <int> | default = 100]

PyroscopeDB (Local Storage)

pyroscopedb:
  # Directory for local storage
  [data_path: <string> | default = "./data"]
  # Max block duration
  [max_block_duration: <duration> | default = 1h]
  # Row group target size (uncompressed)
  [row_group_target_size: <int> | default = 1342177280]
  # Partition label for symbols
  [symbols_partition_label: <string> | default = ""]
  # Disk retention: minimum free disk (GiB)
  [min_free_disk_gb: <int> | default = 10]
  # Disk retention: minimum free percentage
  [min_disk_available_percentage: <float> | default = 0.05]
  # How often to enforce retention
  [enforcement_interval: <duration> | default = 5m]
  # Disable retention enforcement
  [disable_enforcement: <boolean> | default = false]

Storage (Object Storage Backend)

Supported backends: s3, gcs, azure, swift, filesystem, cos.

storage:
  [backend: <string> | default = ""]
  [prefix: <string> | default = ""]

  s3:
    [endpoint: <string> | default = ""]
    [region: <string> | default = ""]
    [bucket_name: <string> | default = ""]
    [secret_access_key: <string> | default = ""]
    [access_key_id: <string> | default = ""]
    [insecure: <boolean> | default = false]
    [signature_version: <string> | default = "v4"]
    [bucket_lookup_type: <string> | default = "auto"]
    # NOTE: native_aws_auth_enabled exists on main but NOT in v1.18.0.
    # In v1.18.0, leave access_key_id/secret_access_key empty to use
    # the default AWS SDK credential chain (env vars, IMDS).
    sse:
      [type: <string> | default = ""]           # SSE-KMS or SSE-S3
      [kms_key_id: <string> | default = ""]
      [kms_encryption_context: <string> | default = ""]

  gcs:
    [bucket_name: <string> | default = ""]
    [service_account: <string> | default = ""]

  azure:
    [account_name: <string> | default = ""]
    [account_key: <string> | default = ""]
    [container_name: <string> | default = ""]

  filesystem:
    [dir: <string> | default = "./data-shared"]

Distributor

distributor:
  [pushtimeout: <duration> | default = 5s]
  ring:
    kvstore:
      [store: <string> | default = "memberlist"]  # consul, etcd, inmemory, memberlist, multi

Ingester

ingester:
  lifecycler:
    ring:
      kvstore:
        [store: <string> | default = "consul"]
      [heartbeat_timeout: <duration> | default = 1m]
      [replication_factor: <int> | default = 1]
    [num_tokens: <int> | default = 128]
    [heartbeat_period: <duration> | default = 5s]

Querier

querier:
  # Time after which queries go to storage instead of ingesters
  [query_store_after: <duration> | default = 4h]

Compactor

compactor:
  [block_ranges: <list of durations> | default = 1h0m0s,2h0m0s,8h0m0s]
  [data_dir: <string> | default = "./data-compactor"]
  [compaction_interval: <duration> | default = 30m]
  [compaction_concurrency: <int> | default = 1]
  [deletion_delay: <duration> | default = 12h]
  [downsampler_enabled: <boolean> | default = false]

Limits (Per-Tenant)

limits:
  # Ingestion rate limit (MB/s)
  [ingestion_rate_mb: <float> | default = 4]
  [ingestion_burst_size_mb: <float> | default = 2]
  # Label constraints
  [max_label_name_length: <int> | default = 1024]
  [max_label_value_length: <int> | default = 2048]
  [max_label_names_per_series: <int> | default = 30]
  # Profile constraints
  [max_profile_size_bytes: <int> | default = 4194304]
  [max_profile_stacktrace_samples: <int> | default = 16000]
  [max_profile_stacktrace_depth: <int> | default = 1000]
  # Series limits
  [max_global_series_per_tenant: <int> | default = 5000]
  # Query limits
  [max_query_lookback: <duration> | default = 1w]
  [max_query_length: <duration> | default = 1d]
  [max_flamegraph_nodes_default: <int> | default = 8192]
  [max_flamegraph_nodes_max: <int> | default = 1048576]
  # Retention
  [compactor_blocks_retention_period: <duration> | default = 0s]
  # Ingestion time bounds
  [reject_older_than: <duration> | default = 1h]
  [reject_newer_than: <duration> | default = 10m]
  # Relabeling
  [ingestion_relabeling_rules: <list of Configs> | default = []]
  [sample_type_relabeling_rules: <list of Configs> | default = []]

Self-Profiling

self_profiling:
  # Disable push profiling in single-binary mode
  [disable_push: <boolean> | default = false]
  [mutex_profile_fraction: <int> | default = 5]
  [block_profile_rate: <int> | default = 5]

Memberlist (Gossip)

memberlist:
  [bind_port: <int> | default = 7946]
  [join_members: <list of strings> | default = []]
  [gossip_interval: <duration> | default = 200ms]
  [gossip_nodes: <int> | default = 3]
  [leave_timeout: <duration> | default = 20s]

Tracing

tracing:
  [enabled: <boolean> | default = true]

Multi-Tenancy

# Require X-Scope-OrgId header; false = use "anonymous" tenant
[multitenancy_enabled: <boolean> | default = false]

Embedded Grafana

embedded_grafana:
  [data_path: <string> | default = "./data/__embedded_grafana/"]
  [listen_port: <int> | default = 4041]
  [pyroscope_url: <string> | default = "http://localhost:4040"]

Port Summary

Relevant to Our Deployment

Our Pyroscope deployment (configuration/pyroscope/PyroscopeManifestBuilder.kt) uses:

• S3 backend — IAM role auth via IMDS (no explicit credentials; v1.18.0 lacks native_aws_auth_enabled, SDK defaults to credential chain)
• Single-binary mode (target: all)
• Port 4040 for HTTP API
• Flat storage prefix — pyroscope.{name}-{id} (Pyroscope rejects / in storage.prefix)
• Config values substituted at build time via TemplateService (__KEY__ placeholders)
• Profiles received from: Java agent (Cassandra, Spark), eBPF agent (all nodes), stress jobs

Kit Development Guide

This guide covers how to build a kit for easy-db-lab — from the kit.yaml structure through lifecycle phases, metrics collection, hooks, and Grafana dashboard provisioning.

What is a Kit?

A kit is a self-contained package of configuration and scripts that installs, starts, stops, and optionally backs up a piece of software on the cluster. Examples: ClickHouse, Presto, OpenSearch.

Each kit lives under src/main/resources/com/rustyrazorblade/easydblab/kits/<name>/.

After easy-db-lab kit install <name> runs, the kit directory is copied to the cluster's working directory. The <name> start, <name> stop, etc. subcommands then drive it.

Directory Layout

install/<name>/
├── kit.yaml              # Required: kit definition
├── bin/                     # Optional: legacy shell scripts (start.sh, stop.sh, ...)
├── dashboards/              # Optional: Grafana dashboard JSON files
├── <name>.yaml.template     # Optional: K8s manifest templates for typed steps
└── METRICS.md               # Optional but recommended: documents exposed metrics

kit.yaml Reference

name: myworkload
description: Short description shown in help text
version: "1.0.0"
collision-check: false   # true = refuse to install if already present

metrics:
  type: scrape           # see Metrics section
  port: 9090

runtime:
  type: helm             # see Runtime section
  release: myworkload
  namespace: default

endpoints:
  - name: "HTTP UI"
    node-type: app       # "app" or "db"
    port: 8080
    type: http           # http | https | jdbc | native | cql | postgresql | mysql
    scheme: ""           # optional: used for JDBC URLs (jdbc type only)
    path: ""             # optional: appended to URL (http/https/jdbc only)
    database: ""         # optional: logical database name (postgresql and mysql types)

args:
  - flag: --workers
    variable: WORKERS
    description: "Number of workers"
    type: int            # string | int | float | boolean | kit-ref
    capability: sql      # optional: for kit-ref, declares required capability
    required: false
    default: "${APP_NODE_COUNT}"

hooks:
  post-workload-start:
    script: bin/update-catalogs.sh
    workloads: []       # optional: only fire when these kits start
  post-workload-stop:
    script: bin/update-catalogs.sh

install:  []   # steps to run on `easy-db-lab kit install <name>`
start:    []   # steps to run on `easy-db-lab <name> start`
stop:     []   # steps to run on `easy-db-lab <name> stop`
uninstall: []  # steps to run on `easy-db-lab <name> uninstall`
backup:   []   # steps to run on `easy-db-lab <name> backup <backup-name>`
restore:  []   # steps to run on `easy-db-lab <name> restore <backup-name>`

Lifecycle Phases

If no typed steps are defined for a phase and a matching script exists in bin/ (e.g. bin/start.sh), the script is executed instead.

Step Types

All phases use the same set of typed steps.

helm-repo

Adds a Helm chart repository.

- type: helm-repo
  name: altinity
  url: https://docs.altinity.com/clickhouse-operator/

helm

Installs or upgrades a Helm chart.

- type: helm
  chart: altinity/altinity-clickhouse-operator
  release: clickhouse-operator
  namespace: kube-system
  version: "1.2.3"          # optional: pin chart version
  values:                    # optional: inline values
    replicaCount: "3"
  values-file: values.yaml  # optional: path relative to kit dir

helm-uninstall

Uninstalls a Helm release.

- type: helm-uninstall
  release: clickhouse-operator
  namespace: kube-system

manifest

Applies a K8s manifest template. The template file must exist in the kit's resource directory. Template variables (see below) are substituted before applying.

- type: manifest
  template: clickhouseinstallation.yaml

manifest-url

Fetches and applies a manifest from a URL.

- type: manifest-url
  url: https://example.com/operator.yaml

kustomize

Applies a kustomize configuration from a URL.

- type: kustomize
  url: https://github.com/example/repo/config/default

namespace

Creates a Kubernetes namespace (no-op if it already exists).

- type: namespace
  name: monitoring

wait

Waits for a K8s resource to reach a condition.

- type: wait
  kind: Deployment
  name: my-operator
  namespace: kube-system
  condition: Available   # default: Available
  timeout: 300s          # default: 300s

delete

Deletes a K8s resource.

- type: delete
  kind: ClickHouseInstallation
  name: clickhouse
  namespace: default
  ignore-not-found: true   # default: true

platform-pvs

Creates persistent volumes on cluster nodes using the platform substrate.

- type: platform-pvs
  node-type: db    # default: db
  count: 3         # optional: defaults to node count

configmap

Creates or updates a K8s ConfigMap.

- type: configmap
  name: my-config
  namespace: default
  data:
    key: value

label

Applies labels to cluster nodes.

- type: label
  node-type: db
  labels:
    kit: presto

exec

Runs a command inside a running pod.

- type: exec
  pod: my-pod-name
  namespace: default
  command: ["clickhouse-client", "--query", "SELECT 1"]

shell

Runs an inline shell script. The script runs locally (not on the remote node) with cluster variables injected as environment variables.

- type: shell
  script: |
    kubectl wait --for=condition=Ready pods \
      -l app=myworkload \
      --timeout=300s

Environment Variables

All scripts and shell steps receive the following environment variables: