Cluster¶

Clusters represent a group of Incus servers running on top of IncusOS, that allow to spread workloads across multiple servers.

Operations Center allows to provision clusters from registered servers.

Provisioning of a cluster can be done through two slightly different approaches:

One off clustering
Template based clustering

In both cases, the administrator needs to provide the service configuration and the application configuration.

Once one or many servers are clustered, Operations Center will automatically keep track of their inventory.

Service Configuration¶

IncusOS system services are optional system-wide features, typically used to integrate with an external system like storage or networking. The complete list of services can be found in the IncusOS services documentation.

During clustering, service configuration is applied on each server. The clustering process accepts a single configuration file (YAML or JSON) containing the configuration for all services, where each service name is a top-level key with the respective configuration underneath it.

Example with LVM and nvme service:

---
lvm:
  enabled: true
  # System ID is automatically determined by Operations Center during clustering.
  # system_id: 0
nvme:
  enabled: true
  targets:
    - transport: tcp
      address: 192.168.1.100
      port: 8009

Copying the Service Configuration when Adding Servers¶

When servers are added to an existing cluster, the service configuration of the added servers is required to be consistent with the one of the existing cluster members.

Instead of configuring the services on the new servers manually, the service configuration can be copied from an existing cluster member by adding the --copy-services-config flag:

operations-center cluster add-servers my-cluster --server-names new-server --copy-services-config

The configuration of the lvm, iscsi, multipath, nvme, ceph, linstor and ovn services is copied. The LVM system_id is never copied. If the LVM service needs to be enabled on an added server, the value is determined by Operations Center for each added server individually, otherwise the system_id of the added server is kept as is.

The listen_address of the linstor service and the tunnel_address of the ovn service are member dependent, if they are set to a concrete IP address. For those, the address of the added server is used, which is taken from the network interface with the same role and of the same IP family as the address of the cluster member the configuration is copied from. Empty and wildcard addresses (e.g. [::]:3366) are copied as they are.

The configuration is copied before the servers join the cluster. If one of the steps up to the join fails, Operations Center tries to restore the previous service configuration of the added servers. Failures during the restore are logged.

Application Configuration¶

The application configuration provided during clustering follows the same format as the preseed configuration used by Incus for non-interactive configuration (see InitLocalPreseed struct definition for full details).

Example:

---
config:
  user.ui.title: "My wonderful cluster"
certificates:
  - type: client
    name: my-client-cert
    description: "Client certificate for accessing the cluster"
    certificate: |
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----

One Off Clustering¶

One off clustering takes a service configuration file, an application configuration file and the list of to be clustered servers as arguments.

Template Based Clustering¶

Template based clustering uses a cluster-template, a file containing key-value pairs for the defined variables in the cluster-template and the list of to be clustered servers as arguments.

The file containing the variables has the following format (YAML):

---
SOME_VARIABLE: "the value"
A_BOOLEAN_VARIABLE: true
A_NUMERIC_VARIABLE: 42

Cluster Bulk Operations¶

Operations Center allows to perform bulk operations on clusters, which are then applied to all members of the cluster. Operations Center supports the following bulk operations:

Adding or removing a vlan tags from network interfaces
Adding or removing a storage target for iSCSI/NVME/multipath services
Deploying of secondary application
Updating of system settings:
- Kernel
- Logging

In order to execute a bulk operation, the action and its arguments need to be provided.

Cluster Bulk Operations Payload Reference¶

add_network_interface_vlan_tags:

{
  "interface_name": "eth0",
  "vlan_tags": [100, 200]
}

remove_network_interface_vlan_tags:

{
  "interface_name": "eth0",
  "vlan_tags": [100, 200]
}

update_system_logging:

{
  "config": {
    "syslog": {
      "address": "127.0.0.1",
      "log_format": "",
      "protocol": "tcp"
    }
  }
}

see Update System Logging for the full list of accepted parameters.

update_system_kernel:

{
  "config": {
    "blacklist_modules": [
      "bad-module"
    ],
    "network": {
      "buffer_size": 33554432,
      "queuing_discipline": "fq",
      "tcp_congestion_algorithm": "bbr"
    },
    "pci": {
      "passthrough": [
        {
          "pci_address": "0000:04:00.0",
          "product_id": "1050",
          "vendor_id": "1af4"
        }
      ]
    }
  }
}

see Update System Kernel for the full list of accepted parameters.

add_application:

{
  "name": "debug"
}

see Non-primary applications for the list of supported applications.

add_iscsi_storage_target:

{
  "target": "",
  "address": "",
  "port": 1234
}

remove_iscsi_storage_target:

{
  "target": "",
  "address": "",
  "port": 1234
}

add_multipath_storage_target:

{
  "wwn": ""
}

remove_multipath_storage_target:

{
  "wwn": ""
}

add_nvme_storage_target:

{
  "transport": "tcp",
  "address": "",
  "port": 1234
}

remove_nvme_storage_target:

{
  "transport": "tcp",
  "address": "",
  "port": 1234
}