Observability on KubeArchive

KubeArchive emits observability data using OpenTelemetry: metrics, traces and logs. Environment variables on KubeArchive’s components control the enablement and configuration of features related with observability.

Custom Metrics

KubeArchive exposes the following custom metrics:

kubearchive.cloudevents.total

Labels: result, event_type resource_type

Tracks the total number of Cloud Events (resource updates) received aggregated by resource_type, event_type and result.

  • result: one of insert, update, none, error, no_match or no_conf.

    • error: there was an error during the processing of the resource update. This may indicate problems with the database, deleting resources, Cloud Event corruption, etc. Check the logs to find the root cause.

    • no_conf: the resource update received is not configured on any KubeArchiveConfig. This may mean there is something sending Cloud Events to KubeArchive but it should not.

    • no_match: the resource update received does not match the conditions for processing. This may indicate that KubeArchiveConfig rules should be refined.

    • insert: the resource was inserted into the database.

    • update: a resource with the same metadata.uid exists in the database and it was updated.

    • none: a resource with the same metadata.uid exists in the database and its metadata.managedFields.time was newer, so there was no update. This may indicate a problem with the system that sends resource updates.

  • resource_type: a combination of the resource apiVersion and kind. For example apps/v1/Deployment, v1/Pods or tekton.dev/v1/PipelineRun.

  • event_type: one of dev.knative.apiserver.resource.new, dev.knative.apiserver.resource.update or dev.knative.apiserver.resource.delete.

Configuring Observability

The following environment variables present on all of the KubeArchive components and control the observability behavior:

  • KUBEARCHIVE_OTEL_MODE: one of "enabled", "delegated" or "disabled". Defaults to "disabled". It controls when observability data is sent to OTEL_EXPORTER_OTLP_ENDPOINT:

    • "disabled": do not send traces or metrics.

    • "enabled": always send traces and metrics. Useful for observability in development.

    • "delegated": always send metrics but send traces only when the incoming request sent its traces (see W3 Trace Context sampled flag). Useful for observability in production.

  • OTEL_EXPORTER_OTLP_ENDPOINT: an OTLP compatible endpoint where traces are sent. By default it is set to an empty string.

  • KUBEARCHIVE_OTLP_SEND_LOGS: if set to "true" logs are sent to the OTEL_EXPOTER_OTLP_ENDPOINT and not printed to stdout. Defaults to "false", so logs are printed on stdout by default. When enabled, a single log line is printed to stdout informing that logs will be redirected.

To change these environment variables values, edit or patch KubeArchive’s components: deployments/kubearchive-api-server, deployments/kubearchive-sink and deployments/kubearchive-operator.