System settings¶

Several global settings are available to be configured in Operations Center:

Network settings¶

Configuration	Description	Value(s)	Default
`address`	Address of Operations Center which is used by managed servers to connect	https://address	same as `rest_server_address`
`rest_server_address`	Address/port over which the REST API will be served	address:port	`*:7443`

Security settings¶

Configuration	Description	Value(s)
`trusted_tls_client_cert_fingerprints`	List of SHA256 certificate fingerprints belonging to trusted TLS clients	list of strings
`trusted_tls_client_certificates`	List of X509 PEM encoded certificates belonging to trusted TLS clients	list of strings
`oidc`	OIDC configuration
`openfga`	OpenFGA configuration
`acme`	ACME certificate renewal configuration

The SHA256 fingerprint of every certificate in trusted_tls_client_certificates is derived when the configuration is loaded and added to the trusted fingerprints in memory, therefore both settings grant the respective client access to Operations Center. The derived fingerprints are never written to the configuration, so removing a certificate is all it takes to revoke the access it grants.

In contrast to bare fingerprints, the certificates are additionally passed on to the servers and clusters deployed by Operations Center, so whoever has access to Operations Center also has access to what Operations Center deploys. See application configuration for how to opt out of this for a specific cluster.

A certificate is passed on at the time a system is deployed. Removing it from trusted_tls_client_certificates therefore only revokes the access to Operations Center itself, the already deployed systems keep trusting the certificate until it is removed from their trust store as well.

OIDC¶

Configuration	Description	Value(s)
`oidc.issuer`	OIDC issuer	string
`oidc.client_id`	OIDC client ID used for communication with OIDC issuer	string
`oidc.scope`	Scopes to be requested	string
`oidc.audience`	Audience the OIDC tokens should be verified against	string
`oidc.claim`	Claim which should be used to identify the user or subject	string

OpenFGA¶

Configuration	Description	Value(s)
`api_token`	API token used for communication with the OpenFGA system	string
`api_url`	URL of the OpenFGA API	string
`store_id`	ID of the OpenFGA store	string

ACME¶

Certificate renewal will be re-attempted every 24 hours, The certificate will be replaced after 80% of its validity period has passed.

Configuration	Description	Value(s)	Default
`agree_tos`	Agree to ACME terms of service.	true/false	false
`ca_url`	URL to the directory resource of the ACME service.	string	`https://acme-v02.api.letsencrypt.org/directory`
`challenge`	ACME challenge type to use.	HTTP-01 or DNS-01	`HTTP-01`
`domain`	Domain for which the certificate is issued.	string
`email`	Email address used for the account registration.	string
`http_challenge_address`	Address and interface for HTTP server (used by HTTP-01).	string	`:80`
`provider`	Backend provider for the challenge (used by DNS-01).	string
`provider_environment`	Environment variables to set during the challenge (used by DNS-01).	list of strings
`provider_resolvers`	List of DNS resolvers (used by DNS-01).	list of strings

Note

Renewal of ACME certificates after a change of the configuration is happening asynchronously in the background. It may take some time until the new certificates are available.

System settings¶

Configuration	Description	Value(s)	Default
`log_level`	Default log level for Operations Center logs	string	`WARN`
`log_levels`	Log levels per component, overriding `log_level`, see Per component log levels below for details	map of string:string
`server_registration_scriptlet`	Scriptlet which is executed during server registration, see Server registration scriptlet below for details	string
`pprof_enabled`	Enable the pprof debug endpoints, see pprof debug endpoints below for details	true/false	false
`metrics_enabled`	Enable the metrics endpoint, see Metrics components below for details	true/false	true
`metrics_components`	Components that expose metrics, see Metrics components below for details	list of strings	`runtime`

The supported log levels are TRACE, DEBUG, INFO, WARN and ERROR.

Per component log levels¶

log_level applies to the whole daemon. log_levels raises or lowers the verbosity for individual components, so a component under investigation can be logged verbosely while the rest of the daemon stays quiet.

Component names are hierarchical, the levels are separated by dots. A level configured for a component applies to all of its children as well and the most specific configuration wins. Components without a matching entry use log_level.

settings:
  log_level: WARN
  log_levels:
    provisioning: DEBUG
    provisioning.server_repo: TRACE

With this configuration everything below provisioning is logged at DEBUG, provisioning.server_repo is logged at TRACE and the rest of the daemon is logged at WARN.

Every log record emitted by a component carries the component name in the component log attribute.

A level configured for a component which is not known is rejected.

The top level component names are:

Component	Covers
`access_log`	The access log of the REST API
`api`	Serving the REST API requests
`config`	Applying a configuration change
`daemon`	Startup and shutdown of the daemon, including one component per background task, e.g. `daemon.task.refresh_inventory`
`image`	The Incus image service
`inventory`	The inventory of the cluster resources
`provisioning`	Servers, clusters, tokens, updates, channels and cluster templates
`security`	Authentication (`security.authn`) and authorization (`security.authz`)
`system`	The system settings
`warning`	The warnings

Below the top level, the services, repositories and adapters of a component each have a name of their own, e.g. provisioning.server_service, provisioning.server_repo or inventory.image_server_client.

Complete list of log components

This is the complete list of the log components the daemon registers, grouped by their top level. A level configured for a component applies to all of its children as well, so a top level name is enough to cover everything below it.

access_log:

access_log

api:

api

config:

config

daemon:

daemon
daemon.task.certificates_validity_check
daemon.task.cluster_update
daemon.task.poll_evacuating_servers
daemon.task.poll_pending_servers
daemon.task.poll_ready_servers
daemon.task.poll_rebooting_servers
daemon.task.poll_restoring_servers
daemon.task.poll_unresponsive_servers
daemon.task.poll_updating_application_servers
daemon.task.poll_updating_servers
daemon.task.prune_seed_image_cache
daemon.task.refresh_bmc_data
daemon.task.refresh_image_sources
daemon.task.refresh_inventory
daemon.task.refresh_updates
daemon.task.renew_acme_certificate
daemon.task.server_deployment

image:

image.image_incus_file_repo
image.image_incus_repo
image.image_incus_service
image.incus_image_source_port
image.incus_image_source_repo
image.incus_image_source_service
image.simplestreams_port

inventory:

inventory.image_repo
inventory.image_server_client
inventory.image_service
inventory.instance_repo
inventory.instance_server_client
inventory.instance_service
inventory.inventory_aggregate_repo
inventory.inventory_aggregate_service
inventory.network_acl_repo
inventory.network_acl_server_client
inventory.network_acl_service
inventory.network_address_set_repo
inventory.network_address_set_server_client
inventory.network_address_set_service
inventory.network_forward_repo
inventory.network_forward_server_client
inventory.network_forward_service
inventory.network_integration_repo
inventory.network_integration_server_client
inventory.network_integration_service
inventory.network_load_balancer_repo
inventory.network_load_balancer_server_client
inventory.network_load_balancer_service
inventory.network_peer_repo
inventory.network_peer_server_client
inventory.network_peer_service
inventory.network_repo
inventory.network_server_client
inventory.network_service
inventory.network_zone_repo
inventory.network_zone_server_client
inventory.network_zone_service
inventory.profile_repo
inventory.profile_server_client
inventory.profile_service
inventory.project_repo
inventory.project_server_client
inventory.project_service
inventory.provisioning_cluster_service
inventory.provisioning_server_service
inventory.server_client
inventory.storage_bucket_repo
inventory.storage_bucket_server_client
inventory.storage_bucket_service
inventory.storage_pool_repo
inventory.storage_pool_server_client
inventory.storage_pool_service
inventory.storage_volume_repo
inventory.storage_volume_server_client
inventory.storage_volume_service

provisioning:

provisioning.adapter.scriptlet.scriptlet_client_port
provisioning.bios_profile_port
provisioning.bmc_server_client_port
provisioning.channel_repo
provisioning.channel_service
provisioning.cluster_artifact_repo
provisioning.cluster_client_port
provisioning.cluster_provisioning_port
provisioning.cluster_repo
provisioning.cluster_service
provisioning.cluster_template_repo
provisioning.cluster_template_service
provisioning.flasher_port
provisioning.secure_boot_certificate_catalog_port
provisioning.secure_boot_certificate_source_port
provisioning.secure_boot_media_port
provisioning.seed_image_progress_port
provisioning.server_client_port
provisioning.server_repo
provisioning.server_scriptlet_port
provisioning.server_service
provisioning.token_client_port
provisioning.token_repo
provisioning.token_service
provisioning.update_files_repo
provisioning.update_repo
provisioning.update_service
provisioning.update_source_port
provisioning.warning_service_port

security:

security.authn
security.authn.oidc
security.authn.tls
security.authn.unixsocket
security.authz
security.authz.chain
security.authz.oidc
security.authz.openfga
security.authz.tls
security.authz.unixsocket

system:

system.backup_repo
system.cache_repo
system.database_repo
system.provisioning_cluster_service
system.provisioning_server_service
system.system_service

warning:

warning.warning_repo
warning.warning_service

Changes to log_levels take effect immediately, a restart is not required. The --verbose and --debug command line flags of operations-centerd only govern the default log level, the per component log levels always apply.

Metrics components¶

Operations Center serves metrics at /1.0/metrics. See Metrics. metrics_enabled enables or disables the endpoint. If it is false, the endpoint returns an error and no metrics are collected.

metrics_components lists the components that expose metrics. A component that is not in the list exposes no metrics.

Component names are hierarchical. Dots separate the levels. Enabling a component also enables all of its children.

settings:
  metrics_enabled: true
  metrics_components:
    - runtime

Operations Center rejects an unknown component.

Changes to metrics_enabled and metrics_components take effect immediately. A restart is not required.

pprof debug endpoints¶

Operations Center exposes the Go runtime profiling data in the format of pprof on the REST API below /1.0/debug/pprof/. This allows to collect CPU and memory profiles, goroutine dumps and execution traces, e.g. when investigating an issue together with the development team.

The endpoints are disabled by default and need to be enabled with the pprof_enabled setting. They are served on the regular REST API listener and require authentication. With OpenFGA, the can_debug entitlement on the server is required, which is granted to the admin relation.

A profile is downloaded with the command line client:

operations-center admin debug pprof heap
operations-center admin debug pprof profile --seconds 30
operations-center admin debug pprof goroutine --debug 2

For profile and trace, --seconds sets the duration of the capture, which is limited to 300 seconds. Only one capture runs at a time. The downloaded profile can be analyzed with go tool pprof <file> respectively go tool trace <file>.

Server registration scriptlet¶

The server registration scriptlet is a Starlark language (which is a subset of Python) scriptlet which is executed during server registration. It can be used to set properties of the registered server and perform additional actions against the server.

The entry point for the server registration scriptlet is the server_registration function which takes a single argument candidate which represents the server being registered.

Example:

def server_registration(candidate):
  server.set_description("some description")

The functions available in the scriptlet are provided through different namespaces as follows:

Server namespace¶

Functions in the server namespace can be used to set properties of the server being registered:

Function	Description
`server.set_connection_url(connection_url)`	Set the connection URL of the server being registered.
`server.set_description(description)`	Set the description of the server being registered.
`server.set_name(name)`	Set the name of the server being registered.
`server.set_properties(properties)`	Set the properties of the server being registered. Properties is a dictionary of string key-value pairs of type string.
`server.set_update_channel(update_channel)`	Set the update channel of the server being registered. Update channel is a string which should match the name of an existing update channel.

IncusOS namespace¶

Functions in the incusos namespace can be used to get and set information (config, state) from the actual server being registered:

Function	Description
`incusos.add_application(name)`	Add a non-primary application to the server. See the IncusOS documentation for the supported non-primary applications.
`incusos.get_service(service)`	Get the configuration of a service on the server being registered. Service is a string which identifies the service to get the configuration of, e.g. `lvm`. See the IncusOS API documentation for the complete list of services.
`incusos.get_system(resource)`	Get the state and configuration of a system resource from the server being registered. Resource is a string which identifies the resource to get, e.g. `kernel` or `logging`. See the IncusOS API documentation for the complete list of resources.
`incusos.set_service(service, config)`	Set the configuration of a service on the server being registered. Service is a string which identifies the service to set the configuration of, e.g. `lvm`. Configuration is the configuration dictionary expected by the respective service. See the IncusOS API documentation for the complete list of services and their expected configuration.
`incusos.set_system(resource, config)`	Set the configuration of a system resource on the server being registered. Resource is a string which identifies the resource to set, e.g. `kernel` or `logging`. Configuration is the configuration dictionary expected by the respective resource. See the IncusOS API documentation for the complete list of resources and their expected configuration.
`incusos.trigger_action(resource, action, body)`	Execute a command on the server being registered. Resource is a string which identifies the resource to execute the command on (e.g. `storag`”). Use empty string (`""`) for root level commands like `reboot`. Action is a string which identifies the action to be executed (e.g. `reboot` or `scrub-pool`) The colon (`:`) prefix present in the REST API is optional and added automatically if missing. Body is a dictionary containing the parameters for the action. See the IncusOS API documentation for the complete list of resources, their actions and the expected parameters.

Log namespace¶

The log namespace provides functions to log messages during server registration.

Function	Description
`log.error(*messages)`	Add a log entry to operations-center’s log at error level. `messages` is one or more message arguments.
`log.info(*messages)`	Add a log entry to operations-center’s log at info level. `messages` is one or more message arguments.
`log.warn(*messages)`	Add a log entry to operations-center’s log at warning level. `messages` is one or more message arguments.

Update settings¶

Configuration	Description	Value(s)	Default
`source`	Source is the URL of the origin, the updates should be fetched from	string	`https://images.linuxcontainers.org/os/`
`signature_verification_root_ca`	Certificate used to verify the signature of updates provided by `source`	string	root certificate used to sign updates from default `source`
`filter_expression`	Filter expression to filter updates, see [update] for details	string	`"stable" in upstream_channels`
`file_filter_expression`	Filter expression to filter update files, see [update] for details	string	`applies_to_architecture(architecture, "x86_64")`
`updates_default_channel`	Default channel for updates, see [channel] for details	string	`stable`
`server_default_channel`	Default channel for servers/clusters, see [channel] for details	string	`stable`