Skip to content

QueryService

API Version: e6data.io/v1alpha1 Kind: QueryService Short Names: qs, cluster


1. Purpose

QueryService deploys and manages the query execution components of an e6data analytics cluster. This includes:

  • Planner: Query parsing, optimization, and execution planning
  • Queue: Query queuing, scheduling, and cluster coordination
  • Executor: Distributed query execution workers
  • Transpiler (optional): SQL dialect translation sidecar
  • Gateway (optional): External API gateway for programmatic access

Note: Traffic routing is now handled by TrafficInfra which deploys Envoy + xDS for dynamic load balancing. HAProxy has been removed from QueryService.

Create a QueryService after NamespaceConfig and MetadataServices are running. Each QueryService represents a logical analytics cluster that users connect to for running queries.

Note: Infrastructure settings (cloud, tolerations, node selectors, image pull secrets) are now managed by NamespaceConfig. QueryService inherits these settings automatically.


2. High-level Behavior

When you create a QueryService CR, the operator:

  1. Inherits infrastructure settings from NamespaceConfig in the same namespace
  2. Discovers MetadataServices in the same namespace for metadata endpoints
  3. Deploys all components with blue-green deployment strategy
  4. Creates Services for internal and external access
  5. Configures autoscaling if enabled (executor scaling)
  6. Integrates with Pool for burst capacity (if poolRef specified)
  7. Manages auto-suspension/resume if configured

Prerequisites

  • NamespaceConfig must exist in the same namespace (provides cloud, storage backend, scheduling config)
  • MetadataServices must exist in the same namespace (provides storage/schema endpoints)

Child Resources Created

Resource Type Name Pattern Purpose
Deployment {name}-planner-{blue\|green} Query planner pods
Deployment {name}-queue-{blue\|green} Queue/coordinator pods
Deployment {name}-executor-{blue\|green} Query executor pods
Deployment {name}-executor-pool-{blue\|green} Pool executor pods (if poolRef)
Deployment {name}-gateway Gateway pods (if enabled)
ConfigMap {name}-{component}-config-{blue\|green} Component configurations
Service {name}-planner-{blue\|green} Headless planner services (for xDS discovery)
Service {name}-queue-{blue\|green} Queue services
Service {name}-gateway Gateway service (if enabled)

Note: Planner services are now headless (ClusterIP: None) with a GRPC port (9001) for xDS endpoint discovery by TrafficInfra.

External Dependencies

  • MetadataServices: Must exist in same namespace (provides storage/schema endpoints)
  • TrafficInfra (required for external access): Provides Envoy + xDS for traffic routing
  • Pool (optional): For burst capacity beyond regular nodes
  • Object Storage: Access to data lake for query execution

3. Spec Reference

3.1 Top-level Fields

Field Type Required Default Description
alias string Yes - Cluster alias (human-readable identifier)
workspace string Yes - Workspace name (must match MetadataServices)
clusterUUID string No CR name Unique cluster identifier
baseDomain string No - Base domain for endpoints
planner PlannerSpec Yes - Planner configuration
queue QueueSpec Yes - Queue configuration
executor ExecutorSpec Yes - Executor configuration
transpiler TranspilerSpec No Disabled Transpiler sidecar
gateway GatewaySpec No Disabled Gateway deployment
autoSuspension AutoSuspensionSpec No Disabled Auto-suspend configuration
autoResume AutoResumeSpec No Disabled Auto-resume configuration
queryHistory QueryHistorySpec No Disabled Query event logging
podAnnotations map[string]string No {} Annotations for all pods

Inherited from NamespaceConfig

The following fields are inherited from NamespaceConfig and no longer specified in QueryService:

Field Description
cloud Cloud provider (AWS, GCP, AZURE)
storageBackend Object storage path (s3a://, gs://, abfs://)
s3Endpoint Custom S3 endpoint for S3-compatible storage
imageRepository Container registry path
imagePullSecrets Secrets for private registries
tolerations Pod tolerations for scheduling
nodeSelector Node labels for pod placement
affinity Advanced scheduling rules
karpenterNodePool Karpenter NodePool name
serviceAccount ServiceAccount for pods (via serviceAccounts.data)
defaultPoolRef Default pool for burst capacity

3.2 Planner (PlannerSpec)

Field Type Required Default Description
replicas int32 No 1 Number of planner pods
image ImageSpec Yes - Planner image configuration
resources ResourceSpec Yes - CPU/Memory limits
ports.web int32 No 10001 Web/query port
ports.grpc int32 No 9001 GRPC port (for Envoy xDS routing)
ports.metrics int32 No 8081 Metrics port
queryTimeoutSeconds int64 No 900 Query timeout (15 min default)
jwksEnabled bool No false Enable JWKS authentication
configVariables map[string]string No {} config.properties entries (see ConfigVariables Reference)
environmentVariables map[string]string No {} Container env vars (see EnvironmentVariables Reference)
extraConfig map[string]string No {} Additional config properties

3.3 Queue (QueueSpec)

Field Type Required Default Description
image ImageSpec Yes - Queue image configuration
resources ResourceSpec Yes - CPU/Memory limits
ports.cache int32 No 9007 Cache port
ports.clusterManager int32 No 9004 Cluster manager port
ports.thrift int32 No 9003 Thrift RPC port
ports.web int32 No 10003 HTTP API port
ports.metrics int32 No 8081 Metrics port
configVariables map[string]string No {} config.properties entries (see ConfigVariables Reference)
environmentVariables map[string]string No {} Container env vars (see EnvironmentVariables Reference)

3.4 Executor (ExecutorSpec)

Field Type Required Default Description
replicas int32 No 1 Number of executor pods
image ExecutorImageSpec Yes - Executor image (tag required)
resources ResourceSpec Yes - CPU/Memory limits
ports.thrift int32 No 9002 Thrift port
ports.web int32 No 10002 Health check port
ports.metrics int32 No 8081 Metrics port
health HealthConfig No See defaults Health probe config
configVariables map[string]string No {} config.properties entries (see ConfigVariables Reference)
environmentVariables map[string]string No {} Container env vars (see EnvironmentVariables Reference)
affinity Affinity No - Pod affinity rules
autoscaling AutoscalingSpec No Disabled Executor autoscaling
poolRef PoolReference No - Pool for burst capacity

3.5 Transpiler (TranspilerSpec)

Field Type Required Default Description
enabled bool No false Enable transpiler sidecar
image ImageSpec No - Transpiler image
resources ResourceSpec No - CPU/Memory limits
ports.http int32 No 8100 HTTP port
environmentVariables map[string]string No {} Container env vars

Note: Transpiler runs as a sidecar in the planner pod, not as a separate deployment.

3.6 Gateway (GatewaySpec)

Field Type Required Default Description
enabled bool No false Enable gateway deployment
replicas int32 No 1 Number of gateway pods
image ImageSpec No - Gateway image
resources ResourceSpec No - CPU/Memory limits
ports.http int32 No 8083 HTTP port
ports.metrics int32 No 8081 Metrics port
configVariables map[string]string No {} config.properties entries (see ConfigVariables Reference)
environmentVariables map[string]string No {} Container env vars (see EnvironmentVariables Reference)
service.type string No NodePort Service type
service.port int32 No 8083 Service port

Note: Gateway is a standalone deployment (no blue-green). ENGINE_HOST/ENGINE_PORT are auto-configured.

3.7 AutoSuspension

Field Type Required Default Description
enabled bool No false Enable auto-suspension
maxIdleDurationMinutes int32 Conditional - Idle timeout before suspend

3.8 AutoResume

Field Type Required Default Description
enabled bool No false Enable auto-resume

When enabled, the cluster automatically resumes on incoming requests via the operator API.

3.9 QueryHistory

Field Type Required Default Description
enabled bool No true Enable query history collection
image QueryHistoryImageSpec No timberio/vector:0.41.0-alpine Vector sidecar image
s3Prefix string No query-history S3 prefix for logs
s3Partition string No hour Partition by hour or day
httpPort int32 No 2020 Vector HTTP source port
timeoutSeconds int32 No 5 Publish timeout
messageQueueCapacity int32 No 10000 Internal buffer size
numPublishingThreads int32 No 10 Concurrent publishers
maxPayloadBatchSize int32 No 100 Events per batch
batchTimeoutSecs int32 No 30 S3 batch timeout
batchMaxBytes int64 No 10485760 S3 batch size limit
compression string No gzip Output compression
resources ResourceSpec No - Vector sidecar resources
greptimeRef QueryHistoryGreptimeRef No - GreptimeDB for real-time queries

3.10 Autoscaling

Field Type Required Default Description
enabled bool No false Enable autoscaling
minExecutors int32 No Current replicas Minimum executors
maxExecutors int32 Conditional - Maximum executors
clusterManagementBaseURL string No Auto-generated Operator API URL
windowBased.enabled bool No false Enable window-based scaling
windowBased.slidingWindowDuration int32 Conditional - Window duration (seconds)

3.11 PoolReference

Field Type Required Default Description
name string Yes - Pool name
namespace string No Same namespace Pool namespace

3.12 ConfigVariables Reference

ConfigVariables are written to a config.properties file mounted in the container. These are application-level settings that configure the e6data services.

Auto-Populated Variables (Do Not Override)

The operator automatically sets these values. Specifying them in configVariables will be ignored:

Variable Auto-Value Description
CLOUD Auto-detected Cloud provider (AWS/GCP/AZURE)
ALIAS spec.alias Cluster alias
WORKSPACE spec.workspace Workspace name
CLUSTER_UUID spec.clusterUUID Cluster identifier

Common Planner ConfigVariables

Variable Type Default Description
ENABLE_COST_BASED_OPTIMIZATION bool true Enable cost-based query optimization
MAX_SPLIT_SIZE_BYTES int 134217728 Maximum split size (128MB)
ENABLE_DYNAMIC_FILTERING bool true Enable runtime filtering
ENABLE_QUERY_CACHING bool false Cache query results
MAX_CACHED_QUERY_AGE_SECONDS int 3600 Query cache TTL

Common Queue ConfigVariables

Variable Type Default Description
MAX_CONCURRENT_QUERIES int 100 Max concurrent queries
QUERY_QUEUE_SIZE int 1000 Query queue buffer size
ENABLE_QUERY_PRIORITIES bool true Enable query priority levels
DEFAULT_QUERY_PRIORITY int 5 Default priority (1-10)

Common Executor ConfigVariables

Variable Type Default Description
SPILL_ENABLED bool true Enable spilling to disk
SPILL_PATH string /nvme/spill Spill directory path
MAX_MEMORY_PER_QUERY_BYTES int Auto Memory limit per query
ENABLE_BLOOM_FILTERS bool true Use bloom filters

Common Gateway ConfigVariables

Variable Type Default Description
ENGINE_HOST string Auto Envoy proxy host (auto-configured)
ENGINE_PORT int Auto Envoy proxy port (auto-configured)
API_VERSION string v1 API version prefix

3.13 EnvironmentVariables Reference

EnvironmentVariables are set as container environment variables. These control runtime behavior.

Auto-Populated Variables (Do Not Override)

The operator automatically sets these values from pod metadata:

Variable Auto-Value Description
IS_KUBE "true" Indicates Kubernetes environment
POD_NAME Pod metadata Current pod name
POD_IP Pod status Current pod IP
NAMESPACE Pod metadata Current namespace
JAVA_TOOL_OPTIONS Auto-calculated JVM options (80% of memory)

Common EnvironmentVariables

Variable Type Default Description
E6_LOGGING_LEVEL string E6_INFO Log level: E6_DEBUG, E6_INFO, E6_WARN, E6_ERROR
LOG_FORMAT string json Log format: json, text
TZ string UTC Timezone
ENABLE_JMX bool true Enable JMX metrics
JMX_PORT int 9010 JMX port

Example: Custom Configuration

planner:
  configVariables:
    ENABLE_COST_BASED_OPTIMIZATION: "true"
    MAX_SPLIT_SIZE_BYTES: "268435456"  # 256MB
  environmentVariables:
    E6_LOGGING_LEVEL: "E6_DEBUG"
    TZ: "America/New_York"

queue:
  configVariables:
    MAX_CONCURRENT_QUERIES: "50"
    ENABLE_QUERY_PRIORITIES: "true"
  environmentVariables:
    E6_LOGGING_LEVEL: "E6_INFO"

executor:
  configVariables:
    SPILL_ENABLED: "true"
    ENABLE_BLOOM_FILTERS: "true"
  environmentVariables:
    E6_LOGGING_LEVEL: "E6_INFO"

4. Example Manifests

4.1 Minimal Example

apiVersion: e6data.io/v1alpha1
kind: QueryService
metadata:
  name: analytics-cluster
  namespace: workspace-analytics-prod
spec:
  alias: analytics
  workspace: analytics-prod

  planner:
    image:
      tag: "3.0.217"
    resources:
      memory: "4Gi"
      cpu: "2"

  queue:
    image:
      tag: "3.0.217"
    resources:
      memory: "4Gi"
      cpu: "2"

  executor:
    replicas: 2
    image:
      tag: "3.0.217"
    resources:
      memory: "30Gi"
      cpu: "15"

Note: For external access, create a TrafficInfra CR that references the planner services.

4.2 Production Example (Full Configuration)

apiVersion: e6data.io/v1alpha1
kind: QueryService
metadata:
  name: analytics-cluster
  namespace: workspace-analytics-prod
  labels:
    e6data.io/workspace: analytics-prod
    e6data.io/environment: production
    e6data.io/pool: burst-pool  # For pool selector matching
spec:
  alias: analytics
  workspace: analytics-prod
  clusterUUID: analytics-cluster-prod-001
  baseDomain: e6data.example.com

  # Planner - Query Optimization
  planner:
    replicas: 2
    image:
      tag: "3.0.217"
    resources:
      memory: "8Gi"
      cpu: "4"
    queryTimeoutSeconds: 1800  # 30 minutes
    jwksEnabled: true
    configVariables:
      ENABLE_COST_BASED_OPTIMIZATION: "true"

  # Queue - Query Scheduling
  queue:
    replicas: 2
    image:
      tag: "3.0.217"
    resources:
      memory: "8Gi"
      cpu: "4"
    configVariables:
      MAX_CONCURRENT_QUERIES: "50"

  # Executor - Query Workers
  executor:
    replicas: 4
    image:
      tag: "3.0.217"
    resources:
      memory: "60Gi"
      cpu: "30"
    health:
      initialDelaySeconds: 30
      periodSeconds: 10
      failureThreshold: 30

    # Autoscaling configuration
    autoscaling:
      enabled: true
      minExecutors: 2
      maxExecutors: 20
      windowBased:
        enabled: true
        slidingWindowDuration: 300  # 5 minutes

    # Pool for burst capacity
    poolRef:
      name: burst-pool
      namespace: e6-pools

  # Transpiler - SQL Dialect Translation (optional)
  transpiler:
    enabled: true
    image:
      tag: "1.0.0"
    resources:
      memory: "1Gi"
      cpu: "500m"

  # Gateway - External API (optional)
  gateway:
    enabled: true
    replicas: 2
    image:
      tag: "1.0.0"
    resources:
      memory: "1Gi"
      cpu: "500m"
    service:
      type: LoadBalancer
      port: 8083

  # Auto-suspension for cost savings
  autoSuspension:
    enabled: true
    maxIdleDurationMinutes: 30

  autoResume:
    enabled: true

  # Query history logging
  queryHistory:
    enabled: true
    s3Prefix: "query-history/analytics-cluster"
    s3Partition: hour
    compression: gzip
    greptimeRef:
      name: greptime-cluster
      database: public
      table: query_history

  # Scheduling
  tolerations:
    - key: "e6data-workspace-name"
      operator: "Equal"
      value: "analytics-prod"
      effect: "NoSchedule"

  karpenterNodePool: query-services

  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8081"

4.3 Development/Testing Example

apiVersion: e6data.io/v1alpha1
kind: QueryService
metadata:
  name: dev-cluster
  namespace: workspace-dev
spec:
  alias: dev
  workspace: dev

  planner:
    replicas: 1
    image:
      tag: "3.0.217"
    resources:
      memory: "2Gi"
      cpu: "1"
    queryTimeoutSeconds: 300  # 5 minutes for dev

  queue:
    replicas: 1
    image:
      tag: "3.0.217"
    resources:
      memory: "2Gi"
      cpu: "1"

  executor:
    replicas: 1
    image:
      tag: "3.0.217"
    resources:
      memory: "8Gi"
      cpu: "4"

  # Auto-suspend after 10 minutes idle
  autoSuspension:
    enabled: true
    maxIdleDurationMinutes: 10

  autoResume:
    enabled: true

5. Status & Lifecycle

5.1 Status Fields

Field Type Description
phase string Current lifecycle phase
message string Human-readable status message
ready bool true when all components ready
activeStrategy string Current active deployment (blue or green)
activeReleaseVersion string Currently running version
pendingStrategy string Deployment being prepared
deploymentPhase string Blue-green phase: Stable, Deploying, Switching, Draining, Cleanup
plannerDeployment DeploymentStatus Planner status
queueDeployment DeploymentStatus Queue status
executorDeployment DeploymentStatus Executor status
poolExecutorDeployment DeploymentStatus Pool executor status
transpilerDeployment DeploymentStatus Transpiler status
gatewayDeployment DeploymentStatus Gateway status
plannerService string Planner service endpoint
queueService string Queue service endpoint
gatewayService string Gateway service endpoint
poolName string Attached pool name
poolNamespace string Attached pool namespace
regularExecutorReplicas int32 Executors on regular nodes
poolExecutorReplicas int32 Executors on pool nodes
scalingHistory []ScalingEvent Last 20 scaling operations
suspensionHistory []SuspensionEvent Last 20 suspend/resume events
releaseHistory []QueryReleaseRecord Last 10 releases

5.2 Phase Values

Phase Description
Waiting Waiting for dependencies (MetadataServices)
Pending Resource created, waiting to reconcile
Creating Initial deployment in progress
Deploying Blue-green deployment in progress
Ready All components healthy
Running Alias for Ready
Updating Configuration update in progress
Failed Deployment failed
Degraded Partially healthy

5.3 Scaling Events

The operator tracks scaling operations:

scalingHistory:
  - timestamp: "2024-01-15T10:30:00Z"
    component: executor
    oldReplicas: 4
    newReplicas: 8
    trigger: autoscaling-api
    strategy: blue

5.4 Suspension Events

suspensionHistory:
  - timestamp: "2024-01-15T22:00:00Z"
    action: suspend
    trigger: auto-suspension-api
    strategy: blue
    componentsSuspended: [executor, planner, queue]
    preSuspensionReplicas:
      executorReplicas: 4
      plannerReplicas: 2
      queueReplicas: 1

Dependencies

CRD Relationship
NamespaceConfig Required - provides cloud, storage, scheduling configuration
MetadataServices Required - must exist in same namespace for metadata endpoints
TrafficInfra Required for external access - provides Envoy + xDS traffic routing
Pool Optional - for burst capacity via poolRef
GreptimeDBCluster Optional - for real-time query history

Labels Applied

app.kubernetes.io/name: {planner|queue|executor}
app.kubernetes.io/instance: {cr-name}
app.kubernetes.io/component: {component}
app.kubernetes.io/managed-by: e6-operator
e6data.io/workspace: {workspace}
e6data.io/cluster: {alias}
e6data.io/strategy: {blue|green}

7. Troubleshooting

7.1 Common Issues

Cluster Stuck in "Waiting" Phase

Symptoms:

$ kubectl get qs
NAME                PHASE     STRATEGY
analytics-cluster   Waiting

Cause: MetadataServices not ready in same namespace.

Fix:

# Check MetadataServices status
kubectl get mds -n workspace-analytics-prod

# Verify MetadataServices is Ready
kubectl get mds analytics-prod -o jsonpath='{.status.phase}'

Executors Not Scaling

Symptoms: Executor count doesn't change despite autoscaling enabled.

Checks:

# Verify autoscaling config
kubectl get qs analytics-cluster -o jsonpath='{.spec.executor.autoscaling}'

# Check operator logs for scaling decisions
kubectl logs -n e6-operator-system -l app=e6-operator | grep -i scaling

# Check scaling history
kubectl get qs analytics-cluster -o jsonpath='{.status.scalingHistory}'

Pool Executors Not Starting

Symptoms: poolExecutorReplicas: 0 despite pool reference.

Checks:

# Verify pool exists and is active
kubectl get pool burst-pool -n e6-pools

# Check pool allocations
kubectl get pool burst-pool -o jsonpath='{.status.allocations}'

# Verify QueryService is in pool's allowed list
kubectl get pool burst-pool -o jsonpath='{.spec.allowedQueryServices}'

Auto-Resume Not Working

Symptoms: Cluster doesn't wake up on incoming queries.

Checks:

# Verify auto-resume is enabled
kubectl get qs analytics-cluster -o jsonpath='{.spec.autoResume.enabled}'

# Check Envoy proxy is running (handles traffic routing)
kubectl get pods -l e6data.io/component=envoy

# Check xDS control plane logs
kubectl logs -l app=xds-control-plane

7.2 Useful Commands

# Get QueryService status
kubectl get qs analytics-cluster -o yaml

# Watch all components
kubectl get pods -l app.kubernetes.io/instance=analytics-cluster -w

# Check component replicas
kubectl get deploy -l e6data.io/cluster=analytics

# View scaling history
kubectl get qs analytics-cluster -o jsonpath='{.status.scalingHistory[*]}' | jq

# Manual scale (for testing)
kubectl patch qs analytics-cluster --type=merge -p '{"spec":{"executor":{"replicas":8}}}'

# Trigger manual suspend (via annotation)
kubectl annotate qs analytics-cluster e6data.io/suspend=true

# Trigger manual resume
kubectl annotate qs analytics-cluster e6data.io/suspend-

# Check blue-green deployment status
kubectl get deploy -l e6data.io/strategy=blue
kubectl get deploy -l e6data.io/strategy=green

8. Autoscaling API

The operator exposes an HTTP API for autoscaling at:

http://e6-operator.{namespace}.svc.cluster.local:8082/endpoints/v1/cluster/{namespace}/{name}/autoscale

Request:

POST /endpoints/v1/cluster/workspace-analytics-prod/analytics-cluster/autoscale
{
  "desired_executors": 10
}

Response:

{
  "status": "scaling",
  "current_executors": 4,
  "target_executors": 10,
  "operation_id": "scale-xyz123"
}

The API automatically splits scaling between regular and pool executors based on minExecutors configuration.