Enabling Observability on KubeArchive

KubeArchive is instrumented to emit observability data: metrics, traces and logs. This can be enabled and configured with the following environment variables present on all of the KubeArchive components:

  • KUBEARCHIVE_OTEL_MODE: one of "enabled", "delegated" or "disabled". Defaults to an "disabled". It controls when observability data is sent to OTEL_EXPORTER_OTLP_ENDPOINT:

    • "disabled": do not send traces or metrics.

    • "enabled": always send traces and metrics. Useful for observability in development.

    • "delegated": always send metrics but send traces only when the incoming request sent its traces (see W3 Trace Context sampled flag). Useful for observability in production.

  • OTEL_EXPORTER_OTLP_ENDPOINT: an OTLP compatible endpoint where traces are sent. By default it is set to an empty string.

  • KUBEARCHIVE_OTLP_SEND_LOGS: if set to "true" logs are sent to the OTEL_EXPOTER_OTLP_ENDPOINT and not printed to stdout. Defaults to "false", so logs are printed on stdout by default. When enabled, a single log line is printed to stdout informing that logs will be redirected.

To change these environment variables values, edit or patch the kubearchive-api-server, kubearchive-sink and kubearchive-operator Deployments on the kubearchive namespace:

  1. Patch for the kubearchive-api-server Deployment:

    ---
apiVersion: v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: kubearchive-api-server
          env:
            - name: KUBEARCHIVE_OTEL_MODE
              value: "delegated"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://otel-collector.observability.svc.cluster.local:4318"

  2. Patch for the kubearchive-sink Deployment:

    ---
apiVersion: v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: kubearchive-sink
          env:
            - name: KUBEARCHIVE_OTEL_MODE
              value: "delegated"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://otel-collector.observability.svc.cluster.local:4318"

  3. Patch for the kubearchive-operator Deployment:

    ---
apiVersion: v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: manager
          env:
            - name: KUBEARCHIVE_OTEL_MODE
              value: "delegated"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://traces-collector.observability.svc.cluster.local:4318"

  4. Apply the patches:

    kubectl patch -n kubearchive deployment <deployment-name> --patch-file path/to/patch.yaml