Cluster¶

Clusters represent a group of Incus servers running on top of IncusOS, that allow to spread workloads across multiple servers.

Operations Center allows to provision clusters from registered servers.

Provisioning of a cluster can be done through two slightly different approaches:

One off clustering
Template based clustering

In both cases, the administrator needs to provide the service configuration and the application configuration.

Once one or many servers are clustered, Operations Center will automatically keep track of their inventory.

Service Configuration¶

IncusOS system services are optional system-wide features, typically used to integrate with an external system like storage or networking. The complete list of services can be found in the IncusOS services documentation.

During clustering, service configuration is applied on each server. The clustering process accepts a single configuration file (YAML or JSON) containing the configuration for all services, where each service name is a top-level key with the respective configuration underneath it.

Example with LVM and nvme service:

---
lvm:
  enabled: true
  # System ID is automatically determined by Operations Center during clustering.
  # system_id: 0
nvme:
  enabled: true
  targets:
    - transport: tcp
      address: 192.168.1.100
      port: 8009

Application Configuration¶

The application configuration provided during clustering follows the same format as the preseed configuration used by Incus for non-interactive configuration (see InitLocalPreseed struct definition for full details).

Example:

---
config:
  user.ui.title: "My wonderful cluster"
certificates:
  - type: client
    name: my-client-cert
    description: "Client certificate for accessing the cluster"
    certificate: |
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----

One Off Clustering¶

One off clustering takes a service configuration file, an application configuration file and the list of to be clustered servers as arguments.

Template Based Clustering¶

Template based clustering uses a cluster-template, a file containing key-value pairs for the defined variables in the cluster-template and the list of to be clustered servers as arguments.

The file containing the variables has the following format (YAML):

---
SOME_VARIABLE: "the value"
A_BOOLEAN_VARIABLE: true
A_NUMERIC_VARIABLE: 42

Cluster Bulk Operations¶

Operations Center allows to perform bulk operations on clusters, which are then applied to all members of the cluster. Operations Center supports the following bulk operations:

Adding or removing a vlan tags from network interfaces
Adding or removing a storage target for iSCSI/NVME/multipath services
Deploying of secondary application
Updating of system settings:
- Kernel
- Logging

In order to execute a bulk operation, the action and its arguments need to be provided.

Cluster Bulk Operations Payload Reference¶

add_network_interface_vlan_tags:

{
  "interface_name": "eth0",
  "vlan_tags": [100, 200]
}

remove_network_interface_vlan_tags:

{
  "interface_name": "eth0",
  "vlan_tags": [100, 200]
}

update_system_logging:

{
  "config": {
    "syslog": {
      "address": "127.0.0.1",
      "log_format": "",
      "protocol": "tcp"
    }
  }
}

see Update System Logging for the full list of accepted parameters.

update_system_kernel:

{
  "config": {
    "blacklist_modules": [
      "bad-module"
    ],
    "network": {
      "buffer_size": 33554432,
      "queuing_discipline": "fq",
      "tcp_congestion_algorithm": "bbr"
    },
    "pci": {
      "passthrough": [
        {
          "pci_address": "0000:04:00.0",
          "product_id": "1050",
          "vendor_id": "1af4"
        }
      ]
    }
  }
}

see Update System Kernel for the full list of accepted parameters.

add_application:

{
  "name": "debug"
}

see Non-primary applications for the list of supported applications.

add_iscsi_storage_target:

{
  "target": "",
  "address": "",
  "port": 1234
}

remove_iscsi_storage_target:

{
  "target": "",
  "address": "",
  "port": 1234
}

add_multipath_storage_target:

{
  "wwn": ""
}

remove_multipath_storage_target:

{
  "wwn": ""
}

add_nvme_storage_target:

{
  "transport": "tcp",
  "address": "",
  "port": 1234
}

remove_nvme_storage_target:

{
  "transport": "tcp",
  "address": "",
  "port": 1234
}