End-to-End Testing

easy-db-lab includes a comprehensive end-to-end test suite that validates the entire workflow from provisioning to teardown.

Running the Test

The end-to-end test is located at bin/end-to-end-test:

./bin/end-to-end-test --cassandra

Command-Line Options

Feature Flags

Flag Description
--cassandra Enable Cassandra-specific tests
--spark Enable Spark EMR provisioning and tests
--clickhouse Enable ClickHouse deployment and tests
--opensearch Enable OpenSearch deployment and tests
--all Enable all optional features
--ebs Enable EBS volumes (gp3, 256GB)
--build Build Packer images (default: skip)

Testing and Inspection

Flag Description
--list-steps, -l List all test steps without running
--break <steps> Set breakpoints at specific steps (comma-separated)
--wait Run all steps except teardown, then wait for confirmation

Examples

# List all available test steps
./bin/end-to-end-test --list-steps

# Run full test with all features
./bin/end-to-end-test --all

# Run with Cassandra and pause before teardown
./bin/end-to-end-test --cassandra --wait

# Run with breakpoints at steps 5 and 15
./bin/end-to-end-test --cassandra --break 5,15

# Build custom AMI images and run test
./bin/end-to-end-test --build --cassandra

Test Steps

The test executes approximately 27 steps covering:

Infrastructure

  1. Build project
  2. Check version command
  3. Build packer images (optional)
  4. Set IAM policies
  5. Initialize cluster
  6. Setup kubectl
  7. Wait for K3s ready
  8. Verify K3s cluster

Registry and Storage

  1. Test registry push/pull
  2. List hosts
  3. Verify S3 backup

Cassandra

  1. Setup Cassandra
  2. Verify Cassandra backup
  3. Verify restore
  4. Cassandra start/stop cycle
  5. Test SSH and nodetool
  6. Check Sidecar
  7. Test exec command
  8. Run stress test
  9. Run stress K8s test

Optional Services

  1. Submit Spark job (if --spark)
  2. Check Spark status (if --spark)
  3. Start ClickHouse (if --clickhouse)
  4. Test ClickHouse (if --clickhouse)
  5. Stop ClickHouse (if --clickhouse)
  6. Start OpenSearch (if --opensearch)
  7. Test OpenSearch (if --opensearch)
  8. Stop OpenSearch (if --opensearch)

Observability and Cleanup

  1. Test observability stack
  2. Teardown cluster

Error Handling

When a test step fails, an interactive menu appears:

  1. Retry from failed step - Resume from the point of failure
  2. Start a shell session - Opens a shell with:
  3. easy-db-lab commands available
  4. rebuild - Rebuild just the project
  5. rerun - Rebuild and resume from failed step
  6. Tear down environment - Run easy-db-lab down --yes
  7. Exit - Exit the script

AWS Requirements

The test requires:

Default Configuration

CI Integration

The end-to-end test is designed to run in CI environments: