TrafficInfra¶

API Version: e6data.io/v1alpha2 Kind: TrafficInfra Short Names: ti

1. Purpose¶

TrafficInfra manages the traffic infrastructure for routing gRPC queries to e6data query services. This includes:

• xDS Control Plane: Dynamic service discovery using Kubernetes Endpoints
• Envoy Proxy: High-performance gRPC load balancing with weighted routing

TrafficInfra replaces the HAProxy component that was previously part of QueryService. It provides: - Dynamic endpoint discovery (no static configuration) - Weighted blue-green traffic routing via xDS admin API - Horizontal autoscaling for Envoy proxies - In-flight query affinity via plannerip header routing

Note: For authentication, use the separate AuthGateway CRD which manages Pomerium for identity-aware access control.

Note: Infrastructure settings (tolerations, nodeSelector, affinity, imagePullSecrets) are inherited from NamespaceConfig in the same namespace.

2. High-level Behavior¶

When you create a TrafficInfra CR, the operator:

1. Inherits infrastructure settings from NamespaceConfig (tolerations, node selectors, affinity)
2. Deploys xDS Control Plane which watches Kubernetes Endpoints for planner services
3. Deploys Envoy Proxy configured to receive dynamic configuration from xDS
4. Creates LoadBalancer Service for external gRPC access (or ClusterIP if using AuthGateway)

Prerequisites¶

• NamespaceConfig must exist in the same namespace
• QueryService planner services must exist (e.g., {cluster}-planner-blue, {cluster}-planner-green)

Architecture¶

Client -> Envoy Proxy -> Planner (blue/green)
              ^
              |
        xDS Control Plane (watches planner services)

The xDS control plane: 1. Watches Kubernetes Endpoints for configured planner services 2. Builds Envoy cluster configurations with endpoint addresses 3. Applies traffic weights for blue-green routing 4. Pushes configuration updates to Envoy via ADS (Aggregated Discovery Service)

Child Resources Created¶

3. Spec Reference¶

3.1 Top-level Fields¶

Note: For authentication, use the separate AuthGateway CRD.

3.2 XDS (XDSSpec)¶

Configuration for the xDS control plane.

Discovery Services¶

The discovery.services field specifies which Kubernetes Services to watch for endpoints. In most cases, you don't need to configure this - the operator automatically registers planner services with xDS when QueryService creates them.

Automatic Registration (Recommended): When you create a QueryService, the operator: 1. Creates headless planner services ({name}-planner-blue, {name}-planner-green) 2. Automatically calls the xDS /services API to register them for endpoint discovery 3. Traffic weights are managed automatically during blue-green deployments

Manual Configuration (Optional): If you need to manually specify services:

discovery:
  services:
    - freshworks-planner-blue   # Blue strategy planner
    - freshworks-planner-green  # Green strategy planner

The xDS control plane will: 1. Watch these services using the Kubernetes Endpoints API 2. Build weighted clusters for blue and green 3. Apply traffic weights when routing to planners

3.3 Envoy (EnvoySpec)¶

Configuration for the Envoy proxy.

3.4 TrafficDefaults¶

Initial traffic weights for blue-green routing.

4. Status Reference¶

5. Example CR¶

Minimal Example (POC)¶

apiVersion: e6data.io/v1alpha2
kind: TrafficInfra
metadata:
  name: traffic
  namespace: workspace-freshworks-prod
spec:
  xds:
    replicas: 2
    # discovery.services not needed - operator auto-registers planner services
  envoy:
    replicas: 2
    service:
      type: LoadBalancer

Note: discovery.services and trafficDefaults are optional. The operator automatically registers planner services with xDS when QueryService is created, and manages traffic weights during blue-green deployments.

Production Example (with AuthGateway)¶

For production with authentication, use TrafficInfra with ClusterIP and route through AuthGateway:

apiVersion: e6data.io/v1alpha2
kind: TrafficInfra
metadata:
  name: traffic
  namespace: workspace-prod
spec:
  xds:
    replicas: 3
    image:
      tag: "1.0.14"
    resources:
      cpu: "500m"
      memory: "512Mi"
    discovery:
      services:
        - myapp-planner-blue
        - myapp-planner-green
    pollInterval: 3

  envoy:
    replicas: 3
    maxReplicas: 20
    resources:
      cpu: "1000m"
      memory: "1Gi"
    hpa:
      enabled: true
      targetCPUUtilization: 60
      targetMemoryUtilization: 70
    service:
      type: ClusterIP  # Internal only, accessed via AuthGateway
      annotations: {}

  trafficDefaults:
    blueWeight: 100
    greenWeight: 0
---
# AuthGateway routes external traffic through Pomerium for authentication
apiVersion: e6data.io/v1alpha1
kind: AuthGateway
metadata:
  name: auth
  namespace: workspace-prod
spec:
  domain: grpc.mycompany.com
  authentication:
    enabled: true
    idp:
      provider: google
      credentialsSecretRef:
        name: pomerium-idp-credentials
    policy:
      allowedDomains:
        - "mycompany.com"
  tls:
    certManager:
      enabled: true
      issuerRef:
        name: letsencrypt-prod
  services:
    - name: query
      subdomain: query
      backend:
        serviceName: traffic-envoy  # Points to TrafficInfra's Envoy
        servicePort: 8080
      timeout: "300s"

6. Traffic Management¶

Automatic Traffic Weight Management¶

The operator automatically manages traffic weights during QueryService blue-green deployments:

1. When QueryService enters the "Switching" phase, the operator calls the xDS admin API
2. Traffic weights are set to route 100% traffic to the new active strategy
3. No manual intervention required for standard deployments

Manual Traffic Weight Control (Optional)¶

For canary deployments or manual control, traffic weights can be updated via the xDS admin API:

# Get current weights
curl http://xds-control-plane:18080/traffic

# Set 50/50 split (canary)
curl -X POST "http://xds-control-plane:18080/traffic?blue=50&green=50"

# Switch to green
curl -X POST "http://xds-control-plane:18080/traffic?blue=0&green=100"

# Rollback to blue
curl -X POST "http://xds-control-plane:18080/traffic?blue=100&green=0"

Dynamic Service Registration API¶

The xDS control plane supports dynamic service registration:

# List registered services
curl http://xds-control-plane:18080/services

# Add a service manually
curl -X PUT "http://xds-control-plane:18080/services?service=myapp-planner-blue"

# Remove a service
curl -X DELETE "http://xds-control-plane:18080/services?service=myapp-planner-blue"

Blue-Green Deployment Workflow¶

Automatic (Default): 1. Update QueryService spec (triggers new deployment to inactive strategy) 2. Operator waits for new deployment to be ready 3. Operator automatically switches traffic via xDS 4. Old deployment remains as standby

Manual Canary: 1. Deploy new version to inactive strategy (update QueryService) 2. Gradually shift traffic: 90/10 -> 70/30 -> 50/50 -> 30/70 -> 0/100 3. Monitor metrics at each step 4. If issues occur, rollback by shifting traffic back 5. Once verified, the old strategy becomes standby for next deployment

7. Relationship with QueryService¶

TrafficInfra replaces HAProxy that was previously part of QueryService:

QueryService Changes¶

• Planner services are now headless for xDS endpoint discovery
• GRPC port (9001) added to planner services
• HAProxy deployment removed (cleanup on reconcile)

8. Troubleshooting¶

Common Issues¶

Debugging Commands¶

# Check xDS control plane status
kubectl logs -l app=xds-control-plane -n <namespace>

# Check Envoy config
kubectl exec -it <envoy-pod> -n <namespace> -- curl localhost:9901/config_dump

# Check discovered endpoints
curl http://<xds-service>:18080/endpoints

# Check current traffic weights
curl http://<xds-service>:18080/traffic

9. Singleton Constraint¶

Only one TrafficInfra is allowed per namespace. The webhook validates this constraint on create.

If you need multiple traffic configurations, use separate namespaces for each QueryService.