Metrics

On this page

This document is for an unreleased version of Crossplane.

This document applies to the Crossplane master branch and not to the latest release v2.1.

Crossplane produces Prometheus style metrics for effective monitoring and alerting in your environment. These metrics are essential for helping to identify and resolve potential issues. This page offers explanations of all these metrics gathered from Crossplane. Understanding these metrics helps you maintain the health and performance of your resources. Please note that this document focuses on Crossplane specific metrics and doesn’t cover standard Go metrics.

To enable the export of metrics it’s necessary to configure the --set metrics.enabled=true option in the helm chart.

1metrics:
2  enabled: true

These Prometheus annotations expose the metrics:

1prometheus.io/path: /metrics
2prometheus.io/port: "8080"
3prometheus.io/scrape: "true"

Crossplane core metrics

The Crossplane pod emits these metrics.

Metric Name	Description
`function_run_function_request_total`	Total number of RunFunctionRequests sent
`function_run_function_response_total`	Total number of RunFunctionResponses received
`function_run_function_seconds`	Histogram of RunFunctionResponse latency (seconds)
`function_run_function_response_cache_hits_total`	Total number of RunFunctionResponse cache hits
`function_run_function_response_cache_misses_total`	Total number of RunFunctionResponse cache misses
`function_run_function_response_cache_errors_total`	Total number of RunFunctionResponse cache errors
`function_run_function_response_cache_writes_total`	Total number of RunFunctionResponse cache writes
`function_run_function_response_cache_deletes_total`	Total number of RunFunctionResponse cache deletes
`function_run_function_response_cache_bytes_written_total`	Total number of RunFunctionResponse bytes written to cache
`function_run_function_response_cache_bytes_deleted_total`	Total number of RunFunctionResponse bytes deleted from cache
`function_run_function_response_cache_read_seconds`	Histogram of cache read latency (seconds)
`function_run_function_response_cache_write_seconds`	Histogram of cache write latency (seconds)
`circuit_breaker_opens_total`	Number of times the XR circuit breaker transitioned from closed to open
`circuit_breaker_closes_total`	Number of times the XR circuit breaker transitioned from open to closed
`circuit_breaker_events_total`	Number of XR watch events handled by the circuit breaker, labeled by outcome
`engine_controllers_started_total`	Total number of controllers started
`engine_controllers_stopped_total`	Total number of controllers stopped
`engine_watches_started_total`	Total number of watches started
`engine_watches_stopped_total`	Total number of watches stopped

Provider metrics

Crossplane providers emit these metrics. All providers built with crossplane-runtime emit the crossplane_managed_resource_* metrics.

Providers expose metrics on the metrics port (default 8080). To scrape these metrics, configure a PodMonitor or add Prometheus annotations to the provider’s DeploymentRuntimeConfig.

Metric Name	Description
`crossplane_managed_resource_exists`	The number of managed resources that exist
`crossplane_managed_resource_ready`	The number of managed resources in `Ready=True` state
`crossplane_managed_resource_synced`	The number of managed resources in `Synced=True` state
`crossplane_managed_resource_deletion_seconds`	The time it took to delete a managed resource
`crossplane_managed_resource_first_time_to_readiness_seconds`	The time it took for a managed resource to become ready first time after creation
`crossplane_managed_resource_first_time_to_reconcile_seconds`	The time it took to detect a managed resource by the controller
`crossplane_managed_resource_drift_seconds`	Time elapsed after the last successful reconcile when detecting an out-of-sync resource

Upjet provider metrics

These metrics are only emitted by Upjet-based providers (such as provider-upjet-aws, provider-upjet-azure, provider-upjet-gcp).

Metric Name	Description
`upjet_resource_ext_api_duration`	Measures in seconds how long it takes a Cloud SDK call to complete
`upjet_resource_external_api_calls_total`	The number of external API calls to cloud providers, with labels describing the endpoints and resources
`upjet_resource_reconcile_delay_seconds`	Measures in seconds how long the reconciles for a resource delay from the configured poll periods
`upjet_resource_ttr`	Measures in seconds the time-to-readiness (TTR) for managed resources
`upjet_resource_cli_duration`	Measures in seconds how long it takes a Terraform CLI invocation to complete
`upjet_resource_active_cli_invocations`	The number of active (running) Terraform CLI invocations
`upjet_resource_running_processes`	The number of running Terraform CLI and Terraform provider processes

Controller-runtime and Kubernetes client metrics

These metrics come from the controller-runtime framework and Kubernetes client libraries. Both Crossplane and providers emit these metrics.

Metric Name	Description
`certwatcher_read_certificate_errors_total`	Total number of certificate read errors
`certwatcher_read_certificate_total`	Total number of certificate reads
`controller_runtime_active_workers`	Number of workers (threads processing jobs from the work queue) per controller
`controller_runtime_max_concurrent_reconciles`	Maximum number of concurrent reconciles per controller
`controller_runtime_reconcile_errors_total`	Total number of reconciliation errors per controller. Sharp or continuous rising of this metric indicates a problem.
`controller_runtime_reconcile_time_seconds`	Histogram of time per reconciliation per controller
`controller_runtime_reconcile_total`	Total number of reconciliations per controller
`controller_runtime_webhook_latency_seconds`	Histogram of the latency of processing admission requests
`controller_runtime_webhook_requests_in_flight`	Current number of admission requests served
`controller_runtime_webhook_requests_total`	Total number of admission requests by HTTP status code
`rest_client_requests_total`	Number of HTTP requests, partitioned by status code, method, and host
`workqueue_adds_total`	Total number of adds handled by `workqueue`
`workqueue_depth`	Current depth of `workqueue`
`workqueue_longest_running_processor_seconds`	How long the longest running processor for `workqueue` has been running
`workqueue_queue_duration_seconds`	Histogram of time an item stays in `workqueue` before processing starts
`workqueue_retries_total`	Total number of retries handled by `workqueue`
`workqueue_unfinished_work_seconds`	Seconds of work in progress not yet observed by `work_duration`. Large values suggest stuck threads.
`workqueue_work_duration_seconds`	Histogram of time to process an item from `workqueue` (from start to completion)