QueryService¶
API Version: e6data.io/v1alpha1 Kind: QueryService Short Names: qs, cluster
1. Purpose¶
QueryService deploys and manages the query execution components of an e6data analytics cluster. This includes:
- Planner: Query parsing, optimization, and execution planning
- Queue: Query queuing, scheduling, and cluster coordination
- Executor: Distributed query execution workers
- Transpiler (optional): SQL dialect translation sidecar
- Gateway (optional): External API gateway for programmatic access
Note: Traffic routing is now handled by TrafficInfra which deploys Envoy + xDS for dynamic load balancing. HAProxy has been removed from QueryService.
Create a QueryService after NamespaceConfig and MetadataServices are running. Each QueryService represents a logical analytics cluster that users connect to for running queries.
Note: Infrastructure settings (cloud, tolerations, node selectors, image pull secrets) are now managed by NamespaceConfig. QueryService inherits these settings automatically.
2. High-level Behavior¶
When you create a QueryService CR, the operator:
- Inherits infrastructure settings from NamespaceConfig in the same namespace
- Discovers MetadataServices in the same namespace for metadata endpoints
- Deploys all components with blue-green deployment strategy
- Creates Services for internal and external access
- Configures autoscaling if enabled (executor scaling)
- Integrates with Pool for burst capacity (if
poolRefspecified) - Manages auto-suspension/resume if configured
Prerequisites¶
- NamespaceConfig must exist in the same namespace (provides cloud, storage backend, scheduling config)
- MetadataServices must exist in the same namespace (provides storage/schema endpoints)
Child Resources Created¶
| Resource Type | Name Pattern | Purpose |
|---|---|---|
| Deployment | {name}-planner-{blue\|green} | Query planner pods |
| Deployment | {name}-queue-{blue\|green} | Queue/coordinator pods |
| Deployment | {name}-executor-{blue\|green} | Query executor pods |
| Deployment | {name}-executor-pool-{blue\|green} | Pool executor pods (if poolRef) |
| Deployment | {name}-gateway | Gateway pods (if enabled) |
| ConfigMap | {name}-{component}-config-{blue\|green} | Component configurations |
| Service | {name}-planner-{blue\|green} | Headless planner services (for xDS discovery) |
| Service | {name}-queue-{blue\|green} | Queue services |
| Service | {name}-gateway | Gateway service (if enabled) |
Note: Planner services are now headless (ClusterIP: None) with a GRPC port (9001) for xDS endpoint discovery by TrafficInfra.
External Dependencies¶
- MetadataServices: Must exist in same namespace (provides storage/schema endpoints)
- TrafficInfra (required for external access): Provides Envoy + xDS for traffic routing
- Pool (optional): For burst capacity beyond regular nodes
- Object Storage: Access to data lake for query execution
3. Spec Reference¶
3.1 Top-level Fields¶
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
alias | string | Yes | - | Cluster alias (human-readable identifier) |
workspace | string | Yes | - | Workspace name (must match MetadataServices) |
clusterUUID | string | No | CR name | Unique cluster identifier |
baseDomain | string | No | - | Base domain for endpoints |
planner | PlannerSpec | Yes | - | Planner configuration |
queue | QueueSpec | Yes | - | Queue configuration |
executor | ExecutorSpec | Yes | - | Executor configuration |
transpiler | TranspilerSpec | No | Disabled | Transpiler sidecar |
gateway | GatewaySpec | No | Disabled | Gateway deployment |
autoSuspension | AutoSuspensionSpec | No | Disabled | Auto-suspend configuration |
autoResume | AutoResumeSpec | No | Disabled | Auto-resume configuration |
queryHistory | QueryHistorySpec | No | Disabled | Query event logging |
podAnnotations | map[string]string | No | {} | Annotations for all pods |
Inherited from NamespaceConfig¶
The following fields are inherited from NamespaceConfig and no longer specified in QueryService:
| Field | Description |
|---|---|
cloud | Cloud provider (AWS, GCP, AZURE) |
storageBackend | Object storage path (s3a://, gs://, abfs://) |
s3Endpoint | Custom S3 endpoint for S3-compatible storage |
imageRepository | Container registry path |
imagePullSecrets | Secrets for private registries |
tolerations | Pod tolerations for scheduling |
nodeSelector | Node labels for pod placement |
affinity | Advanced scheduling rules |
karpenterNodePool | Karpenter NodePool name |
serviceAccount | ServiceAccount for pods (via serviceAccounts.data) |
defaultPoolRef | Default pool for burst capacity |
3.2 Planner (PlannerSpec)¶
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
replicas | int32 | No | 1 | Number of planner pods |
image | ImageSpec | Yes | - | Planner image configuration |
resources | ResourceSpec | Yes | - | CPU/Memory limits |
ports.web | int32 | No | 10001 | Web/query port |
ports.grpc | int32 | No | 9001 | GRPC port (for Envoy xDS routing) |
ports.metrics | int32 | No | 8081 | Metrics port |
queryTimeoutSeconds | int64 | No | 900 | Query timeout (15 min default) |
jwksEnabled | bool | No | false | Enable JWKS authentication |
configVariables | map[string]string | No | {} | config.properties entries (see ConfigVariables Reference) |
environmentVariables | map[string]string | No | {} | Container env vars (see EnvironmentVariables Reference) |
extraConfig | map[string]string | No | {} | Additional config properties |
3.3 Queue (QueueSpec)¶
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
image | ImageSpec | Yes | - | Queue image configuration |
resources | ResourceSpec | Yes | - | CPU/Memory limits |
ports.cache | int32 | No | 9007 | Cache port |
ports.clusterManager | int32 | No | 9004 | Cluster manager port |
ports.thrift | int32 | No | 9003 | Thrift RPC port |
ports.web | int32 | No | 10003 | HTTP API port |
ports.metrics | int32 | No | 8081 | Metrics port |
configVariables | map[string]string | No | {} | config.properties entries (see ConfigVariables Reference) |
environmentVariables | map[string]string | No | {} | Container env vars (see EnvironmentVariables Reference) |
3.4 Executor (ExecutorSpec)¶
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
replicas | int32 | No | 1 | Number of executor pods |
image | ExecutorImageSpec | Yes | - | Executor image (tag required) |
resources | ResourceSpec | Yes | - | CPU/Memory limits |
ports.thrift | int32 | No | 9002 | Thrift port |
ports.web | int32 | No | 10002 | Health check port |
ports.metrics | int32 | No | 8081 | Metrics port |
health | HealthConfig | No | See defaults | Health probe config |
configVariables | map[string]string | No | {} | config.properties entries (see ConfigVariables Reference) |
environmentVariables | map[string]string | No | {} | Container env vars (see EnvironmentVariables Reference) |
affinity | Affinity | No | - | Pod affinity rules |
autoscaling | AutoscalingSpec | No | Disabled | Executor autoscaling |
poolRef | PoolReference | No | - | Pool for burst capacity |
3.5 Transpiler (TranspilerSpec)¶
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
enabled | bool | No | false | Enable transpiler sidecar |
image | ImageSpec | No | - | Transpiler image |
resources | ResourceSpec | No | - | CPU/Memory limits |
ports.http | int32 | No | 8100 | HTTP port |
environmentVariables | map[string]string | No | {} | Container env vars |
Note: Transpiler runs as a sidecar in the planner pod, not as a separate deployment.
3.6 Gateway (GatewaySpec)¶
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
enabled | bool | No | false | Enable gateway deployment |
replicas | int32 | No | 1 | Number of gateway pods |
image | ImageSpec | No | - | Gateway image |
resources | ResourceSpec | No | - | CPU/Memory limits |
ports.http | int32 | No | 8083 | HTTP port |
ports.metrics | int32 | No | 8081 | Metrics port |
configVariables | map[string]string | No | {} | config.properties entries (see ConfigVariables Reference) |
environmentVariables | map[string]string | No | {} | Container env vars (see EnvironmentVariables Reference) |
service.type | string | No | NodePort | Service type |
service.port | int32 | No | 8083 | Service port |
Note: Gateway is a standalone deployment (no blue-green). ENGINE_HOST/ENGINE_PORT are auto-configured.
3.7 AutoSuspension¶
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
enabled | bool | No | false | Enable auto-suspension |
maxIdleDurationMinutes | int32 | Conditional | - | Idle timeout before suspend |
3.8 AutoResume¶
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
enabled | bool | No | false | Enable auto-resume |
When enabled, the cluster automatically resumes on incoming requests via the operator API.
3.9 QueryHistory¶
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
enabled | bool | No | true | Enable query history collection |
image | QueryHistoryImageSpec | No | timberio/vector:0.41.0-alpine | Vector sidecar image |
s3Prefix | string | No | query-history | S3 prefix for logs |
s3Partition | string | No | hour | Partition by hour or day |
httpPort | int32 | No | 2020 | Vector HTTP source port |
timeoutSeconds | int32 | No | 5 | Publish timeout |
messageQueueCapacity | int32 | No | 10000 | Internal buffer size |
numPublishingThreads | int32 | No | 10 | Concurrent publishers |
maxPayloadBatchSize | int32 | No | 100 | Events per batch |
batchTimeoutSecs | int32 | No | 30 | S3 batch timeout |
batchMaxBytes | int64 | No | 10485760 | S3 batch size limit |
compression | string | No | gzip | Output compression |
resources | ResourceSpec | No | - | Vector sidecar resources |
greptimeRef | QueryHistoryGreptimeRef | No | - | GreptimeDB for real-time queries |
3.10 Autoscaling¶
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
enabled | bool | No | false | Enable autoscaling |
minExecutors | int32 | No | Current replicas | Minimum executors |
maxExecutors | int32 | Conditional | - | Maximum executors |
clusterManagementBaseURL | string | No | Auto-generated | Operator API URL |
windowBased.enabled | bool | No | false | Enable window-based scaling |
windowBased.slidingWindowDuration | int32 | Conditional | - | Window duration (seconds) |
3.11 PoolReference¶
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
name | string | Yes | - | Pool name |
namespace | string | No | Same namespace | Pool namespace |
3.12 ConfigVariables Reference¶
ConfigVariables are written to a config.properties file mounted in the container. These are application-level settings that configure the e6data services.
Auto-Populated Variables (Do Not Override)¶
The operator automatically sets these values. Specifying them in configVariables will be ignored:
| Variable | Auto-Value | Description |
|---|---|---|
CLOUD | Auto-detected | Cloud provider (AWS/GCP/AZURE) |
ALIAS | spec.alias | Cluster alias |
WORKSPACE | spec.workspace | Workspace name |
CLUSTER_UUID | spec.clusterUUID | Cluster identifier |
Common Planner ConfigVariables¶
| Variable | Type | Default | Description |
|---|---|---|---|
ENABLE_COST_BASED_OPTIMIZATION | bool | true | Enable cost-based query optimization |
MAX_SPLIT_SIZE_BYTES | int | 134217728 | Maximum split size (128MB) |
ENABLE_DYNAMIC_FILTERING | bool | true | Enable runtime filtering |
ENABLE_QUERY_CACHING | bool | false | Cache query results |
MAX_CACHED_QUERY_AGE_SECONDS | int | 3600 | Query cache TTL |
Common Queue ConfigVariables¶
| Variable | Type | Default | Description |
|---|---|---|---|
MAX_CONCURRENT_QUERIES | int | 100 | Max concurrent queries |
QUERY_QUEUE_SIZE | int | 1000 | Query queue buffer size |
ENABLE_QUERY_PRIORITIES | bool | true | Enable query priority levels |
DEFAULT_QUERY_PRIORITY | int | 5 | Default priority (1-10) |
Common Executor ConfigVariables¶
| Variable | Type | Default | Description |
|---|---|---|---|
SPILL_ENABLED | bool | true | Enable spilling to disk |
SPILL_PATH | string | /nvme/spill | Spill directory path |
MAX_MEMORY_PER_QUERY_BYTES | int | Auto | Memory limit per query |
ENABLE_BLOOM_FILTERS | bool | true | Use bloom filters |
Common Gateway ConfigVariables¶
| Variable | Type | Default | Description |
|---|---|---|---|
ENGINE_HOST | string | Auto | Envoy proxy host (auto-configured) |
ENGINE_PORT | int | Auto | Envoy proxy port (auto-configured) |
API_VERSION | string | v1 | API version prefix |
3.13 EnvironmentVariables Reference¶
EnvironmentVariables are set as container environment variables. These control runtime behavior.
Auto-Populated Variables (Do Not Override)¶
The operator automatically sets these values from pod metadata:
| Variable | Auto-Value | Description |
|---|---|---|
IS_KUBE | "true" | Indicates Kubernetes environment |
POD_NAME | Pod metadata | Current pod name |
POD_IP | Pod status | Current pod IP |
NAMESPACE | Pod metadata | Current namespace |
JAVA_TOOL_OPTIONS | Auto-calculated | JVM options (80% of memory) |
Common EnvironmentVariables¶
| Variable | Type | Default | Description |
|---|---|---|---|
E6_LOGGING_LEVEL | string | E6_INFO | Log level: E6_DEBUG, E6_INFO, E6_WARN, E6_ERROR |
LOG_FORMAT | string | json | Log format: json, text |
TZ | string | UTC | Timezone |
ENABLE_JMX | bool | true | Enable JMX metrics |
JMX_PORT | int | 9010 | JMX port |
Example: Custom Configuration¶
planner:
configVariables:
ENABLE_COST_BASED_OPTIMIZATION: "true"
MAX_SPLIT_SIZE_BYTES: "268435456" # 256MB
environmentVariables:
E6_LOGGING_LEVEL: "E6_DEBUG"
TZ: "America/New_York"
queue:
configVariables:
MAX_CONCURRENT_QUERIES: "50"
ENABLE_QUERY_PRIORITIES: "true"
environmentVariables:
E6_LOGGING_LEVEL: "E6_INFO"
executor:
configVariables:
SPILL_ENABLED: "true"
ENABLE_BLOOM_FILTERS: "true"
environmentVariables:
E6_LOGGING_LEVEL: "E6_INFO"
4. Example Manifests¶
4.1 Minimal Example¶
apiVersion: e6data.io/v1alpha1
kind: QueryService
metadata:
name: analytics-cluster
namespace: workspace-analytics-prod
spec:
alias: analytics
workspace: analytics-prod
planner:
image:
tag: "3.0.217"
resources:
memory: "4Gi"
cpu: "2"
queue:
image:
tag: "3.0.217"
resources:
memory: "4Gi"
cpu: "2"
executor:
replicas: 2
image:
tag: "3.0.217"
resources:
memory: "30Gi"
cpu: "15"
Note: For external access, create a TrafficInfra CR that references the planner services.
4.2 Production Example (Full Configuration)¶
apiVersion: e6data.io/v1alpha1
kind: QueryService
metadata:
name: analytics-cluster
namespace: workspace-analytics-prod
labels:
e6data.io/workspace: analytics-prod
e6data.io/environment: production
e6data.io/pool: burst-pool # For pool selector matching
spec:
alias: analytics
workspace: analytics-prod
clusterUUID: analytics-cluster-prod-001
baseDomain: e6data.example.com
# Planner - Query Optimization
planner:
replicas: 2
image:
tag: "3.0.217"
resources:
memory: "8Gi"
cpu: "4"
queryTimeoutSeconds: 1800 # 30 minutes
jwksEnabled: true
configVariables:
ENABLE_COST_BASED_OPTIMIZATION: "true"
# Queue - Query Scheduling
queue:
replicas: 2
image:
tag: "3.0.217"
resources:
memory: "8Gi"
cpu: "4"
configVariables:
MAX_CONCURRENT_QUERIES: "50"
# Executor - Query Workers
executor:
replicas: 4
image:
tag: "3.0.217"
resources:
memory: "60Gi"
cpu: "30"
health:
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 30
# Autoscaling configuration
autoscaling:
enabled: true
minExecutors: 2
maxExecutors: 20
windowBased:
enabled: true
slidingWindowDuration: 300 # 5 minutes
# Pool for burst capacity
poolRef:
name: burst-pool
namespace: e6-pools
# Transpiler - SQL Dialect Translation (optional)
transpiler:
enabled: true
image:
tag: "1.0.0"
resources:
memory: "1Gi"
cpu: "500m"
# Gateway - External API (optional)
gateway:
enabled: true
replicas: 2
image:
tag: "1.0.0"
resources:
memory: "1Gi"
cpu: "500m"
service:
type: LoadBalancer
port: 8083
# Auto-suspension for cost savings
autoSuspension:
enabled: true
maxIdleDurationMinutes: 30
autoResume:
enabled: true
# Query history logging
queryHistory:
enabled: true
s3Prefix: "query-history/analytics-cluster"
s3Partition: hour
compression: gzip
greptimeRef:
name: greptime-cluster
database: public
table: query_history
# Scheduling
tolerations:
- key: "e6data-workspace-name"
operator: "Equal"
value: "analytics-prod"
effect: "NoSchedule"
karpenterNodePool: query-services
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8081"
4.3 Development/Testing Example¶
apiVersion: e6data.io/v1alpha1
kind: QueryService
metadata:
name: dev-cluster
namespace: workspace-dev
spec:
alias: dev
workspace: dev
planner:
replicas: 1
image:
tag: "3.0.217"
resources:
memory: "2Gi"
cpu: "1"
queryTimeoutSeconds: 300 # 5 minutes for dev
queue:
replicas: 1
image:
tag: "3.0.217"
resources:
memory: "2Gi"
cpu: "1"
executor:
replicas: 1
image:
tag: "3.0.217"
resources:
memory: "8Gi"
cpu: "4"
# Auto-suspend after 10 minutes idle
autoSuspension:
enabled: true
maxIdleDurationMinutes: 10
autoResume:
enabled: true
5. Status & Lifecycle¶
5.1 Status Fields¶
| Field | Type | Description |
|---|---|---|
phase | string | Current lifecycle phase |
message | string | Human-readable status message |
ready | bool | true when all components ready |
activeStrategy | string | Current active deployment (blue or green) |
activeReleaseVersion | string | Currently running version |
pendingStrategy | string | Deployment being prepared |
deploymentPhase | string | Blue-green phase: Stable, Deploying, Switching, Draining, Cleanup |
plannerDeployment | DeploymentStatus | Planner status |
queueDeployment | DeploymentStatus | Queue status |
executorDeployment | DeploymentStatus | Executor status |
poolExecutorDeployment | DeploymentStatus | Pool executor status |
transpilerDeployment | DeploymentStatus | Transpiler status |
gatewayDeployment | DeploymentStatus | Gateway status |
plannerService | string | Planner service endpoint |
queueService | string | Queue service endpoint |
gatewayService | string | Gateway service endpoint |
poolName | string | Attached pool name |
poolNamespace | string | Attached pool namespace |
regularExecutorReplicas | int32 | Executors on regular nodes |
poolExecutorReplicas | int32 | Executors on pool nodes |
scalingHistory | []ScalingEvent | Last 20 scaling operations |
suspensionHistory | []SuspensionEvent | Last 20 suspend/resume events |
releaseHistory | []QueryReleaseRecord | Last 10 releases |
5.2 Phase Values¶
| Phase | Description |
|---|---|
Waiting | Waiting for dependencies (MetadataServices) |
Pending | Resource created, waiting to reconcile |
Creating | Initial deployment in progress |
Deploying | Blue-green deployment in progress |
Ready | All components healthy |
Running | Alias for Ready |
Updating | Configuration update in progress |
Failed | Deployment failed |
Degraded | Partially healthy |
5.3 Scaling Events¶
The operator tracks scaling operations:
scalingHistory:
- timestamp: "2024-01-15T10:30:00Z"
component: executor
oldReplicas: 4
newReplicas: 8
trigger: autoscaling-api
strategy: blue
5.4 Suspension Events¶
suspensionHistory:
- timestamp: "2024-01-15T22:00:00Z"
action: suspend
trigger: auto-suspension-api
strategy: blue
componentsSuspended: [executor, planner, queue]
preSuspensionReplicas:
executorReplicas: 4
plannerReplicas: 2
queueReplicas: 1
6. Related Resources¶
Dependencies¶
| CRD | Relationship |
|---|---|
| NamespaceConfig | Required - provides cloud, storage, scheduling configuration |
| MetadataServices | Required - must exist in same namespace for metadata endpoints |
| TrafficInfra | Required for external access - provides Envoy + xDS traffic routing |
| Pool | Optional - for burst capacity via poolRef |
| GreptimeDBCluster | Optional - for real-time query history |
Labels Applied¶
app.kubernetes.io/name: {planner|queue|executor}
app.kubernetes.io/instance: {cr-name}
app.kubernetes.io/component: {component}
app.kubernetes.io/managed-by: e6-operator
e6data.io/workspace: {workspace}
e6data.io/cluster: {alias}
e6data.io/strategy: {blue|green}
7. Troubleshooting¶
7.1 Common Issues¶
Cluster Stuck in "Waiting" Phase¶
Symptoms:
Cause: MetadataServices not ready in same namespace.
Fix:
# Check MetadataServices status
kubectl get mds -n workspace-analytics-prod
# Verify MetadataServices is Ready
kubectl get mds analytics-prod -o jsonpath='{.status.phase}'
Executors Not Scaling¶
Symptoms: Executor count doesn't change despite autoscaling enabled.
Checks:
# Verify autoscaling config
kubectl get qs analytics-cluster -o jsonpath='{.spec.executor.autoscaling}'
# Check operator logs for scaling decisions
kubectl logs -n e6-operator-system -l app=e6-operator | grep -i scaling
# Check scaling history
kubectl get qs analytics-cluster -o jsonpath='{.status.scalingHistory}'
Pool Executors Not Starting¶
Symptoms: poolExecutorReplicas: 0 despite pool reference.
Checks:
# Verify pool exists and is active
kubectl get pool burst-pool -n e6-pools
# Check pool allocations
kubectl get pool burst-pool -o jsonpath='{.status.allocations}'
# Verify QueryService is in pool's allowed list
kubectl get pool burst-pool -o jsonpath='{.spec.allowedQueryServices}'
Auto-Resume Not Working¶
Symptoms: Cluster doesn't wake up on incoming queries.
Checks:
# Verify auto-resume is enabled
kubectl get qs analytics-cluster -o jsonpath='{.spec.autoResume.enabled}'
# Check Envoy proxy is running (handles traffic routing)
kubectl get pods -l e6data.io/component=envoy
# Check xDS control plane logs
kubectl logs -l app=xds-control-plane
7.2 Useful Commands¶
# Get QueryService status
kubectl get qs analytics-cluster -o yaml
# Watch all components
kubectl get pods -l app.kubernetes.io/instance=analytics-cluster -w
# Check component replicas
kubectl get deploy -l e6data.io/cluster=analytics
# View scaling history
kubectl get qs analytics-cluster -o jsonpath='{.status.scalingHistory[*]}' | jq
# Manual scale (for testing)
kubectl patch qs analytics-cluster --type=merge -p '{"spec":{"executor":{"replicas":8}}}'
# Trigger manual suspend (via annotation)
kubectl annotate qs analytics-cluster e6data.io/suspend=true
# Trigger manual resume
kubectl annotate qs analytics-cluster e6data.io/suspend-
# Check blue-green deployment status
kubectl get deploy -l e6data.io/strategy=blue
kubectl get deploy -l e6data.io/strategy=green
8. Autoscaling API¶
The operator exposes an HTTP API for autoscaling at:
http://e6-operator.{namespace}.svc.cluster.local:8082/endpoints/v1/cluster/{namespace}/{name}/autoscale
Request:
POST /endpoints/v1/cluster/workspace-analytics-prod/analytics-cluster/autoscale
{
"desired_executors": 10
}
Response:
{
"status": "scaling",
"current_executors": 4,
"target_executors": 10,
"operation_id": "scale-xyz123"
}
The API automatically splits scaling between regular and pool executors based on minExecutors configuration.