Kubernetes

Deploy self-hosted Tracecat to Kubernetes with the OCI Helm chart: install Tracecat services and Temporal on EKS, GKE, or any conformant cluster.

The Tracecat Helm chart is distributed as a private OCI artifact hosted in AWS ECR. Please contact customer success for access for evaluations or internal enterprise use.

Prerequisites

• Kubernetes 1.27+
• Helm 3.12+ (with OCI support)
• kubectl configured for your target cluster
• Access to the Tracecat OCI Helm chart (see above)
• External PostgreSQL instance (e.g., Amazon RDS, Cloud SQL)
• External Redis instance (e.g., Amazon ElastiCache, Memorystore)
• S3-compatible object storage (e.g., Amazon S3, MinIO)
• openssl (for generating secrets)

Core secrets

Tracecat requires four cryptographic secrets.

Secrets management

The Helm chart supports three strategies for providing secrets to Tracecat pods.

External Secrets Operator (recommended)

The chart creates ExternalSecret resources that sync from AWS Secrets Manager into Kubernetes secrets via a ClusterSecretStore. You need External Secrets Operator installed and a ClusterSecretStore configured with IRSA.

externalSecrets:
  enabled: true
  clusterSecretStoreRef: "my-cluster-secret-store"

  coreSecrets:
    enabled: true
    secretArn: "arn:aws:secretsmanager:us-west-2:123456789012:secret:tracecat/core"

  postgres:
    enabled: true
    secretArn: "arn:aws:secretsmanager:us-west-2:123456789012:secret:tracecat/postgres"

  redis:
    enabled: true
    secretArn: "arn:aws:secretsmanager:us-west-2:123456789012:secret:tracecat/redis"

Expected secret formats in AWS Secrets Manager:

Secret	Format
Core	JSON: { "dbEncryptionKey": "...", "serviceKey": "...", "signingSecret": "...", "userAuthSecret": "..." }
PostgreSQL	JSON: { "username": "...", "password": "..." }
Redis	Raw URL string, e.g. rediss://:password@host:6379
Temporal	Raw API key string

The chart refreshes secrets every 1m by default. Override with externalSecrets.refreshInterval.

Existing Kubernetes secret

Reference a secret you created with kubectl or your GitOps pipeline.

kubectl create secret generic tracecat-secrets \
  --namespace tracecat \
  --from-literal=dbEncryptionKey="$(openssl rand 32 | base64 | tr -d '\n' | tr '+/' '-_')" \
  --from-literal=serviceKey="$(openssl rand -hex 32)" \
  --from-literal=signingSecret="$(openssl rand -hex 32)" \
  --from-literal=userAuthSecret="$(openssl rand -hex 32)"

Then set secrets.existingSecret in your values:

secrets:
  existingSecret: "tracecat-secrets"

You also need separate secrets for PostgreSQL and Redis, referenced by externalPostgres.auth.existingSecret and externalRedis.auth.existingSecret.

Chart-managed secret templates

Only use this approach if secret values are encrypted in your repository and injected at deploy time through a pipeline like ArgoCD with sealed secrets or SOPS. Storing plaintext secrets in version control is not safe.

secrets:
  create:
    tracecat:
      enabled: true
      dbEncryptionKey: "${DB_ENCRYPTION_KEY}"
      serviceKey: "${SERVICE_KEY}"
      signingSecret: "${SIGNING_SECRET}"
      userAuthSecret: "${USER_AUTH_SECRET}"

    postgres:
      enabled: true
      username: "${DB_USERNAME}"
      password: "${DB_PASSWORD}"

    redis:
      enabled: true
      url: "${REDIS_URL}"

The chart renders these as Helm pre-install hooks so they exist before the migrations job runs.

External services

The Helm chart does not bundle databases. You must provide connection details for PostgreSQL, Redis, S3, and Temporal.

PostgreSQL

externalPostgres:
  host: "your-rds-instance.us-west-2.rds.amazonaws.com"
  port: 5432
  database: tracecat
  sslMode: "require"
  auth:
    existingSecret: "tracecat-postgres-credentials"  # keys: username, password
  tls:
    verifyCA: true
    caCert: |
      -----BEGIN CERTIFICATE-----
      ...RDS CA bundle...
      -----END CERTIFICATE-----

For Amazon RDS, download the global CA bundle from the AWS trust store.

Redis

externalRedis:
  auth:
    existingSecret: "tracecat-redis-credentials"  # key: url

The url key should be a full Redis connection string, e.g. rediss://:password@host:6379. Use rediss:// (double s) for TLS.

S3

externalS3:
  endpoint: ""        # leave empty for AWS S3
  region: "us-west-2"
  auth:
    existingSecret: "tracecat-s3-credentials"  # keys: accessKeyId, secretAccessKey

When using IRSA for S3 access, omit auth.existingSecret and annotate the service account instead (see Service accounts).

Temporal

Self-hosted (bundled)

The chart includes a Temporal subchart. Point it at your external PostgreSQL instance for persistence.

temporal:
  enabled: true
  server:
    config:
      persistence:
        datastores:
          default:
            sql:
              connectAddr: "your-rds-host:5432"
              user: "temporal"
              existingSecret: "tracecat-postgres-credentials"
          visibility:
            sql:
              connectAddr: "your-rds-host:5432"
              user: "temporal"
              existingSecret: "tracecat-postgres-credentials"

The Temporal schema setup job runs automatically and creates the temporal and temporal_visibility databases if they do not exist.

Temporal Cloud

Disable the bundled subchart and point to your Temporal Cloud namespace.

temporal:
  enabled: false

externalTemporal:
  enabled: true
  clusterUrl: "your-namespace.tmprl.cloud:7233"
  clusterNamespace: "your-namespace"
  auth:
    existingSecret: "tracecat-temporal-credentials"  # key: apiKey

Service accounts

The chart creates a shared tracecat-app service account for most workloads (API, worker, migrations). Optionally create dedicated service accounts for executor, agent-executor, and litellm.

IRSA (EKS)

Annotate service accounts with an IAM role ARN for AWS API access (S3, Secrets Manager, Bedrock).

serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/tracecat-app"

executor:
  serviceAccount:
    create: true
    name: "tracecat-executor"
    annotations:
      eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/tracecat-executor"

Dedicated executor service accounts let you scope S3 and secret permissions separately from the main application role.

Cross-account role assumption is supported. Set tracecat.aws.assumeRoleAccountId and tracecat.aws.assumeRolePrincipalArn to assume roles in other AWS accounts from the executor.

Networking

The chart supports Kubernetes Ingress and Istio VirtualService for external traffic routing.

Ingress

The chart creates a single Ingress resource with path-based routing. Set ingress.split: true to generate separate Ingress resources per service (useful when the API requires different annotations than the UI, such as longer timeouts).

ingress:
  enabled: true
  className: "alb"  # or nginx, traefik, etc.
  host: "tracecat.example.com"
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
  tls:
    - hosts:
        - tracecat.example.com
      secretName: tracecat-tls

Default path routing:

Path	Service	Port
/api	api	8000
/mcp	mcp	8099
/	ui	3000

MCP routes (/mcp, /.well-known/oauth-*, /authorize, /token, /register, /consent, /auth/callback) are included automatically when mcp.enabled: true.

Istio VirtualService

Disable ingress.enabled when using VirtualServices to avoid conflicting resources.

ingress:
  enabled: false

virtualService:
  enabled: true
  tracecat:
    enabled: true
    configs:
      - name: tracecat
        hosts:
          - "tracecat.example.com"
        gateways:
          - "istio-system/my-gateway"

Optional webhook and Temporal Web UI VirtualServices are available via virtualService.webhooks and virtualService.temporal.

Installation

1. Add the OCI registry

   helm registry login <ecr-registry-url>
   

   Use the credentials provided by the Tracecat team.

2. Create your values file

   Combine secrets, external services, networking, and URL configuration from the sections above into a single values.yaml.

   secrets:
     existingSecret: "tracecat-secrets"
   
   urls:
     publicApp: "https://tracecat.example.com"
     publicApi: "https://tracecat.example.com/api"
   
   tracecat:
     auth:
       types: "oidc"
       superadminEmail: "[e-mail]"
   
   ingress:
     enabled: true
     className: "alb"
     host: "tracecat.example.com"
   
   externalPostgres:
     host: "your-db-host"
     auth:
       existingSecret: "tracecat-postgres-credentials"
   
   externalRedis:
     auth:
       existingSecret: "tracecat-redis-credentials"
   
   externalS3:
     region: "us-west-2"
   
3. Install the chart

   helm install tracecat oci://<ecr-registry-url>/tracecat \
     --version <chart-version> \
     --namespace tracecat --create-namespace \
     -f values.yaml \
     --wait --timeout 10m
   

   Replace <chart-version> with the version provided by the Tracecat team (e.g. 0.4.5). The --wait flag blocks until all pods are ready. Initial provisioning takes a few minutes depending on Temporal schema setup.

Access Tracecat

Once deployed, access your instance at:

• UI: https://<your-domain>
• API docs: https://<your-domain>/api/docs
• MCP: https://<your-domain>/mcp

Upgrade the Helm chart

helm upgrade tracecat oci://<ecr-registry-url>/tracecat \
  --version <chart-version> \
  --namespace tracecat \
  -f values.yaml \
  --wait --timeout 10m

--version selects the chart version. If you set image tags in values.yaml, a chart upgrade keeps using those tags. Otherwise, it uses the new chart’s appVersion and may also update Tracecat.

Do not change or lose your dbEncryptionKey, serviceKey, signingSecret, or userAuthSecret values between upgrades. Losing these secrets makes encrypted credentials unrecoverable and invalidates existing webhook URLs.

Update Tracecat

The chart version and Tracecat version are independent. You can use the chart’s appVersion or set image tags explicitly.

Use the chart appVersion

Each chart version includes an appVersion. When image tags are unset, the chart uses appVersion for both the backend and UI images.

helm show chart oci://<ecr-registry-url>/tracecat \
  --version <chart-version>

Choose a chart with the required appVersion, then run the chart upgrade command above. This updates the chart and Tracecat together.

Set image tags

To update Tracecat without changing the chart version, keep --version unchanged and set both image tags in values.yaml:

image:
  tag: "<app-version>"

uiImage:
  tag: "<app-version>"

image.tag sets the backend image and the default migrations image. uiImage.tag sets the UI image. Both override the chart’s appVersion; keep them on the same compatible release.

Run the chart upgrade command after updating values.yaml. If you set migrations.image.tag separately, it continues to override image.tag for migrations.

Security

Execution sandboxing

The chart enables nsjail by default (tracecat.sandbox.disableNsjail: false). nsjail isolates user-defined Python scripts and custom actions inside the executor.

tracecat:
  sandbox:
    disableNsjail: false  # default

nsjail requires privileged pods with SYS_ADMIN capability and an Unconfined seccomp profile. The chart sets these automatically on executor and agent-executor containers.

See Security for backend choices (ephemeral, direct) and isolation tradeoffs.

Authentication

Set tracecat.auth.types to oidc or saml for production deployments. See OIDC and SAML for configuration details.

Autoscaling

The chart uses KEDA with a Temporal queue-based scaler to auto-scale worker, executor, and agent-executor deployments.

Prerequisites:

• keda.enabled: true (installs the KEDA subchart)
• metricsserver.enabled: true (or an existing metrics-server in the cluster)

keda:
  enabled: true

metricsserver:
  enabled: true

worker:
  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 8
    targetQueueSize: 5
    cooldownPeriod: 120

executor:
  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 8

When autoscaling is enabled, the static replicas value is ignored. KEDA polls the Temporal task queue and scales based on queue depth.

Minimum resources

These are the default resource requests and limits from the chart.

Component	CPU request	Memory request	CPU limit	Memory limit	Default replicas
UI	500m	512Mi	500m	1024Mi	1
API	2000m	4096Mi	2000m	4096Mi	2
Worker	2000m	2048Mi	2000m	2048Mi	4
Executor	4000m	8192Mi	4000m	8192Mi	4
Agent executor	2000m	4096Mi	4000m	16384Mi	2
Agent worker	2000m	2048Mi	2000m	2048Mi	2
LiteLLM	4000m	8192Mi	4000m	8192Mi	2
MCP	1000m	1024Mi	1000m	1024Mi	2

The agent executor uses burstable limits (requests 2000m/4096Mi, limits 4000m/16384Mi) to handle variable LLM response sizes. All other services use guaranteed QoS where requests equal limits.

Observability

The chart bundles optional subcharts for Prometheus, Grafana, and Grafana Alloy (k8s-monitoring).

tracecat:
  temporal:
    metrics:
      enabled: true

monitoring:
  enabled: true

prometheus:
  enabled: true

grafana:
  enabled: true
  adminPassword: "change-me"

For Grafana Cloud or other external providers, enable only tracecat.temporal.metrics.enabled and configure your own scraping. See values-eks-grafana-cloud.yaml in the chart examples directory.

For production observability with custom dashboards, OpenTelemetry export, and alerting, contact the Tracecat team for enterprise support.

FAQ

Can executors assume cross-account AWS roles?

Yes. Set tracecat.aws.assumeRoleAccountId and tracecat.aws.assumeRolePrincipalArn in your values. The executor uses STS AssumeRole to access resources in other AWS accounts. Ensure the target account’s trust policy allows the executor’s IRSA role.

I'm seeing 'No hosts available' errors from Temporal

Undersized Temporal clusters cause this error. Ensure the Temporal server pods have at least 4 CPU cores and 8 GB of memory.

If using self-hosted Temporal with an external PostgreSQL, verify the database can handle the query throughput. Managed services like Amazon Aurora Serverless are recommended for production workloads.

Does the chart support autoscaling?

Yes. The chart includes KEDA ScaledObject resources for worker, executor, and agent-executor. Set keda.enabled: true and <component>.autoscaling.enabled: true. Scaling is driven by Temporal task queue depth, not CPU or memory utilization.

Can I use non-AWS infrastructure?

Yes. The chart requires PostgreSQL, Redis, and S3-compatible storage but does not require AWS. Use any provider for these services. For secrets management without AWS Secrets Manager, use the existing Kubernetes secret or chart-managed secret template strategies instead of External Secrets Operator.

How do I pin workloads to a specific CPU architecture?

Set scheduling.architecture to arm64 or amd64. The chart adds a kubernetes.io/arch node selector to all pods. Set it to an empty string to disable architecture pinning.