Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

End-to-End Testing

easy-db-lab includes a comprehensive end-to-end test suite that validates the entire workflow from provisioning to teardown.

Running the Test

The end-to-end test is located at bin/end-to-end-test:

./bin/end-to-end-test --cassandra

Command-Line Options

Feature Flags

FlagDescription
--cassandraEnable Cassandra-specific tests
--sparkEnable Spark EMR provisioning and tests
--clickhouseEnable ClickHouse deployment and tests
--opensearchEnable OpenSearch deployment and tests
--allEnable all optional features
--ebsEnable EBS volumes (gp3, 256GB)
--buildBuild Packer images (default: skip)

Testing and Inspection

FlagDescription
--list-steps, -lList all test steps without running
--break <steps>Set breakpoints at specific steps (comma-separated)
--waitRun all steps except teardown, then wait for confirmation

Examples

# List all available test steps
./bin/end-to-end-test --list-steps

# Run full test with all features
./bin/end-to-end-test --all

# Run with Cassandra and pause before teardown
./bin/end-to-end-test --cassandra --wait

# Run with breakpoints at steps 5 and 15
./bin/end-to-end-test --cassandra --break 5,15

# Build custom AMI images and run test
./bin/end-to-end-test --build --cassandra

Test Steps

The test executes approximately 27 steps covering:

Infrastructure

  1. Build project
  2. Check version command
  3. Build packer images (optional)
  4. Set IAM policies
  5. Initialize cluster
  6. Setup kubectl
  7. Wait for K3s ready
  8. Verify K3s cluster

Registry and Storage

  1. Test registry push/pull
  2. List hosts
  3. Verify S3 backup

Cassandra

  1. Setup Cassandra
  2. Verify Cassandra backup
  3. Verify restore
  4. Cassandra start/stop cycle
  5. Test SSH and nodetool
  6. Check Sidecar
  7. Test exec command
  8. Run stress test
  9. Run stress K8s test

Optional Services

  1. Submit Spark job (if --spark)
  2. Check Spark status (if --spark)
  3. Start ClickHouse (if --clickhouse)
  4. Test ClickHouse (if --clickhouse)
  5. Stop ClickHouse (if --clickhouse)
  6. Start OpenSearch (if --opensearch)
  7. Test OpenSearch (if --opensearch)
  8. Stop OpenSearch (if --opensearch)

Observability and Cleanup

  1. Test observability stack
  2. Teardown cluster

Error Handling

When a test step fails, an interactive menu appears:

  1. Retry from failed step - Resume from the point of failure
  2. Start a shell session - Opens a shell with:
    • easy-db-lab commands available
    • rebuild - Rebuild just the project
    • rerun - Rebuild and resume from failed step
  3. Tear down environment - Run easy-db-lab down --yes
  4. Exit - Exit the script

AWS Requirements

The test requires:

  • AWS profile with sufficient permissions
  • VPC and subnet configuration
  • S3 bucket for backups and logs

Default Configuration

  • Instance count: 3 nodes
  • Instance type: c5.2xlarge
  • Cassandra version: 5.0 (when enabled)
  • Spark workers: 2 (when enabled)

CI Integration

The end-to-end test is designed to run in CI environments:

  • Supports non-interactive mode
  • Returns appropriate exit codes
  • Provides detailed logging
  • Cleans up resources on failure