Cluster¶

Clusters represent a group of Incus servers running on top of IncusOS, that allow to spread workloads across multiple servers.

Operations Center allows to provision clusters from registered servers.

Provisioning of a cluster can be done through two slightly different approaches:

One off clustering
Template based clustering

In both cases, the administrator needs to provide the service configuration and the application configuration.

Once one or many servers are clustered, Operations Center will automatically keep track of their inventory.

Service Configuration¶

IncusOS system services are optional system-wide features, typically used to integrate with an external system like storage or networking. The complete list of services can be found in the IncusOS services documentation.

During clustering, service configuration is applied on each server. The clustering process accepts a single configuration file (YAML or JSON) containing the configuration for all services, where each service name is a top-level key with the respective configuration underneath it.

Example with LVM and nvme service:

---
lvm:
  enabled: true
  # System ID is automatically determined by Operations Center during clustering.
  # system_id: 0
nvme:
  enabled: true
  targets:
    - transport: tcp
      address: 192.168.1.100
      port: 8009

Member Dependent Addresses¶

The tunnel_address of the ovn service and the listen_address of the linstor service are different for each server.

If such an address is set to a concrete IP address, this address needs to be assigned to one of the servers of the cluster. For the other servers, Operations Center takes the address from the network interface with the same role and of the same IP family. The port of the listen_address is kept as it is.

Empty and wildcard addresses (e.g. [::]:3366) are applied as they are on each server. If the tunnel_address is empty, IncusOS uses the address of the network interface with the cluster role on each server. This is the recommended setting.

Copying the Service Configuration when Adding Servers¶

When servers are added to an existing cluster, the service configuration of the added servers is required to be consistent with the one of the existing cluster members.

The storage pools of all servers are required to have the same names and types. The devices of a storage pool can differ between servers.

Instead of configuring the services on the new servers manually, the service configuration can be copied from an existing cluster member by adding the --copy-services-config flag:

operations-center cluster add-servers my-cluster --server-names new-server --copy-services-config

The configuration of the lvm, iscsi, multipath, nvme, ceph, linstor and ovn services is copied. The LVM system_id is never copied. If the LVM service needs to be enabled on an added server, the value is determined by Operations Center for each added server individually, otherwise the system_id of the added server is kept as is.

The listen_address of the linstor service and the tunnel_address of the ovn service are member dependent, if they are set to a concrete IP address. For those, the address of the added server is used, which is taken from the network interface with the same role and of the same IP family as the address of the cluster member the configuration is copied from. Empty and wildcard addresses (e.g. [::]:3366) are copied as they are. If the tunnel_address is empty, IncusOS uses the address of the network interface with the cluster role of the added server.

The configuration is copied before the servers join the cluster. If one of the steps up to the join fails, Operations Center tries to restore the previous service configuration of the added servers. Failures during the restore are logged.

Application Configuration¶

The application configuration provided during clustering follows the same format as the preseed configuration used by Incus for non-interactive configuration (see InitLocalPreseed struct definition for full details).

Example:

---
config:
  user.ui.title: "My wonderful cluster"
certificates:
  - type: client
    name: my-client-cert
    description: "Client certificate for accessing the cluster"
    certificate: |
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----

If certificates is empty, Operations Center adds the certificates from the trusted_tls_client_certificates security setting to the cluster, so whoever has access to Operations Center also has access to the cluster. Providing at least one certificate disables this.

Certificates, which the cluster does already trust, e.g. because they have been applied with the seed config of the servers, are not added a second time. They are imported into the state of the generated Terraform configuration instead, so they are managed the same way as the ones added by it.

One Off Clustering¶

One off clustering takes a service configuration file, an application configuration file and the list of to be clustered servers as arguments.

Template Based Clustering¶

Template based clustering uses a cluster-template, a file containing key-value pairs for the defined variables in the cluster-template and the list of to be clustered servers as arguments.

The file containing the variables has the following format (YAML):

---
SOME_VARIABLE: "the value"
A_BOOLEAN_VARIABLE: true
A_NUMERIC_VARIABLE: 42

Removing Servers from a Cluster¶

Servers are removed from a cluster with:

operations-center provisioning cluster remove-servers <cluster> --server-names <server>

A regular removal expects the server to be evacuated and reachable. Operations Center removes the storage volumes it has created on the server, performs a factory reset on it and then removes it from the cluster. The server record is removed from Operations Center as part of the factory reset.

Removing a Lost Cluster Member¶

A server, which has been lost and is not expected to come back online, can not be evacuated and can not be factory reset, so the regular removal does not apply to it. Such a server is removed with the --force flag:

operations-center provisioning cluster remove-servers <cluster> --server-names <server> --force

With --force, Operations Center:

does not require the server to be evacuated,
reports, but does not refuse, the local instances, the custom storage volumes and the images, which are only present on the server and are therefore lost,
attempts the clean up of the storage volumes it has created on the server, but continues, if the server does not respond,
does not perform a factory reset on the server,
removes the server from the cluster, the equivalent of incus cluster remove --force. A server, which has already been removed from the Incus cluster, is accepted as well.

The server record is kept in Operations Center, no longer being part of the cluster. This preserves the data, which has been provided for the server, such as the BMC configuration, the description and the properties, so it remains available for the server, which replaces the lost one. Once the data is no longer needed, the record is removed with:

operations-center provisioning server remove <server>

At least one reachable member has to remain in the cluster to perform the removal, so the last member of a cluster can not be removed this way. Use operations-center provisioning cluster remove <cluster> --force to remove a cluster, which is gone entirely.

Cluster Bulk Operations¶

Operations Center allows to perform bulk operations on clusters, which are then applied to all members of the cluster. Operations Center supports the following bulk operations:

Adding or removing a vlan tags from network interfaces
Adding or removing a storage target for iSCSI/NVME/multipath services
Deploying of secondary application
Updating of system settings:
- Kernel
- Logging

In order to execute a bulk operation, the action and its arguments need to be provided.

Cluster Bulk Operations Payload Reference¶

add_network_interface_vlan_tags:

{
  "interface_name": "eth0",
  "vlan_tags": [100, 200]
}

remove_network_interface_vlan_tags:

{
  "interface_name": "eth0",
  "vlan_tags": [100, 200]
}

update_system_logging:

{
  "config": {
    "syslog": {
      "address": "127.0.0.1",
      "log_format": "",
      "protocol": "tcp"
    }
  }
}

see Update System Logging for the full list of accepted parameters.

update_system_kernel:

{
  "config": {
    "blacklist_modules": [
      "bad-module"
    ],
    "network": {
      "buffer_size": 33554432,
      "queuing_discipline": "fq",
      "tcp_congestion_algorithm": "bbr"
    },
    "pci": {
      "passthrough": [
        {
          "pci_address": "0000:04:00.0",
          "product_id": "1050",
          "vendor_id": "1af4"
        }
      ]
    }
  }
}

see Update System Kernel for the full list of accepted parameters.

add_application:

{
  "name": "debug"
}

see Non-primary applications for the list of supported applications.

add_iscsi_storage_target:

{
  "target": "",
  "address": "",
  "port": 1234
}

remove_iscsi_storage_target:

{
  "target": "",
  "address": "",
  "port": 1234
}

add_multipath_storage_target:

{
  "wwn": ""
}

remove_multipath_storage_target:

{
  "wwn": ""
}

add_nvme_storage_target:

{
  "transport": "tcp",
  "address": "",
  "port": 1234
}

remove_nvme_storage_target:

{
  "transport": "tcp",
  "address": "",
  "port": 1234
}