Access and manage embedded clusters (Beta)

This topic describes managing nodes in clusters created with Replicated Embedded Cluster.

Access the cluster

You can use the CLI to access the cluster. This is useful for development or troubleshooting.

To access the cluster and use other included binaries:

1. SSH into a controller node.

   note

   You cannot run the shell command on worker nodes.

2. Use the Embedded Cluster shell command to start a shell with access to the cluster:

   sudo ./APP_SLUG shell

   Where APP_SLUG is the unique slug for the application.

   The output looks similar to the following:

      __4___
   _  \ \ \ \   Welcome to APP_SLUG debug shell.
   <'\ /_/_/_/   This terminal is now configured to access your cluster.
   ((____!___/) Type 'exit' (or CTRL+d) to exit.
    \0\0\0\0\/  Happy hacking.
   ~~~~~~~~~~~
   root@alex-ec-1:/home/alex# export KUBECONFIG="/var/lib/embedded-cluster/k0s/pki/admin.conf"
   root@alex-ec-1:/home/alex# export PATH="$PATH:/var/lib/embedded-cluster/bin"
   root@alex-ec-1:/home/alex# source <(k0s completion bash)
   root@alex-ec-1:/home/alex# source <(cat /var/lib/embedded-cluster/bin/kubectl_completion_bash.sh)
   root@alex-ec-1:/home/alex# source /etc/bash_completion

   The appropriate kubeconfig is exported, and the location of useful binaries like kubectl and Replicated’s preflight and support-bundle plugins is added to PATH.

3. Use the available binaries as needed.

   Example:

   kubectl version

   Client Version: v1.29.1
   Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
   Server Version: v1.29.1+k0s

4. Type exit or Ctrl + D to exit the shell.

Configure multi-node clusters

This section describes how to join nodes to a cluster with Embedded Cluster.

Limitations

Multi-node clusters with Embedded Cluster have the following limitations:

• All nodes joined to the cluster use the same Embedded Cluster data directory as the installation node. You cannot choose a different data directory for Embedded Cluster when joining nodes.

• You should not join more than one controller node at the same time. When joining a controller node, Embedded Cluster prints a warning explaining that you should not attempt to join another node until the controller node joins successfully.

• Use one primary host (controller node) or at least three. Installation and upgrades are blocked while a cluster has exactly two primary hosts. After adding or removing primary hosts, finish with one or with three or more. When adding primary hosts, join the third after the second finishes joining.

• You cannot change a node's role assignment after you join the node. To change a node's roles, reset the node and add it again with the new role selection. For more information about customizing node roles, see roles in Embedded Cluster Config.

Requirement

To deploy multi-node clusters with Embedded Cluster, you must enable the Multi-node Cluster (Embedded Cluster only) license field for the customer. For more information about managing customer licenses, see Create and Manage Customers.

You can enable this field after installation without reinstalling the instance. Refresh the license before adding nodes. For more information about applying entitlement changes, see Refresh the license in this document.

Join nodes

To join a node:

1. SSH into a controller node.

2. Run the following command to generate the .tar.gz bundle for joining a node:

   sudo ./APP_SLUG create-join-bundle --roles ROLE[,ROLE...]

   Where:

   • APP_SLUG is the unique slug for the application.

   • --roles is one or more role names to assign the node, comma-separated. Valid role names are the default controller role and any custom roles declared in the Embedded Cluster Config under roles.

     note

     You cannot change role assignments after a node is joined. To change them, reset the node and add it again with the new role selection.

3. Use scp to copy the .tar.gz bundle to the node that you want to join.

4. Extract the .tar.gz.

5. Run the join command to add the node to the cluster:

   sudo ./APP_SLUG node join

6. Repeat these steps for each node you want to add.

Headless joins

For automation and orchestrated deployments, you can join nodes without going through the web UI by calling the Embedded Cluster external API from any host in the cluster network. The flow is:

1. Fetch a JWT token from a controller using the installer password:

   TOKEN=$(curl -X POST https://<controller-ip>:30081/v1/auth/token \
     -H 'Content-Type: application/json' \
     -d '{"password":"<installer-password>"}' | jq -r .token)

2. Poll GET /v1/nodes until at least one controller reports ready: true. This is the signal that the cluster is ready to issue join commands:

   until curl -s https://<controller-ip>:30081/v1/nodes \
     -H "Authorization: Bearer $TOKEN" \
     | jq -e '.nodes | any(.ready)' >/dev/null; do
     sleep 5
   done

3. Request the join commands. Pass the desired role names from the roles key in the Embedded Cluster Config:

   RESPONSE=$(curl -X POST https://<controller-ip>:30081/v1/create-join-command \
     -H "Authorization: Bearer $TOKEN" \
     -H 'Content-Type: application/json' \
     -d '{"roles":["controller"]}')

4. Execute each command in the returned joinSteps array on the joining node:

   echo "$RESPONSE" | jq -r '.joinSteps[].command' | while IFS= read -r cmd; do
     eval "$cmd"
   done

For the full set of endpoints and response schemas, see External API.

Refresh the license

License refresh through the persistent console and selection of the newest license during upgrade --license require Embedded Cluster 3.15.0-beta.1 or later. Earlier versions require an installation or upgrade using a release bundle and the updated license.

Use an updated license for the same license ID, application, and customer. License refresh cannot change that identity. The console rejects a license older than the installed license.

Refresh from the persistent console

1. Open License in the persistent console.
2. Apply the license using the option for your environment:
   • For online installations, the page checks for license updates automatically. If an update is available, click Update license and deploy. Otherwise, click Check for license updates to check again. Checking for updates alone does not apply the license.
   • For air gap installations or online installations using offline updates, select the updated license file, then click Upload license and deploy. No release bundle is required for this console operation.

Both options deploy the current application version with the saved configuration. If the license requires additional configuration, the console saves the license pending deployment. Select Review required configuration and deploy, complete the required fields, and deploy.

Refresh from the command line

Without the persistent console, obtain the updated license file and run the following command on the primary controller (the first controller installed). Use the installer and assets for the installed application version:

sudo ./APP_SLUG upgrade --license /path/to/updated-license.yaml

Replace APP_SLUG with the application slug. For air gap installations, keep the matching air gap bundles alongside the installer. Add --headless to run without the browser.

This redeploys the installed version and preserves saved configuration when --config-values is omitted. If supplied, --config-values replaces the configuration; omitted overrides are discarded.

Online upgrades check for a newer license. If that check fails, the command warns and uses the newest verified local license. Offline upgrades use the newer of the supplied and installed licenses. An older bundle cannot revert a newer installed license.

Upgrades, including license refresh through upgrade --license, are blocked while the cluster has exactly two primary hosts. For more information about host counts, see Limitations in this document.

High availability for multi-node clusters

The Kubernetes control plane becomes highly available (HA) with at least three controller nodes. Existing air gap installations require an additional step to complete HA setup. For more information about completing HA setup, see Enable HA for an existing cluster in this document.

Application workloads and Helm extensions must also be configured for high availability.

Best practices for high availability

Consider the following best practices and recommendations for HA clusters:

• HA requires at least three controller nodes that run the Kubernetes control plane. This is because clusters use a quorum system, in which more than half the nodes must be up and reachable. In clusters with three controller nodes, the Kubernetes control plane can continue to operate if one node fails because the remaining two nodes can still form a quorum.

• Always use an odd number of controller nodes in HA clusters. Using an odd number of controller nodes ensures that the cluster can make decisions efficiently with quorum calculations. Clusters with an odd number of controller nodes also avoid split-brain scenarios where the cluster runs as two, independent groups of nodes, resulting in inconsistencies and conflicts.

• You can have any number of worker nodes in HA clusters. Worker nodes do not run the Kubernetes control plane, but can run workloads such as application workloads.

Create a multi-node cluster with HA

To create a multi-node cluster with HA:

• During installation with Embedded Cluster, follow the steps in the Embedded Cluster UI to join a total of three controller nodes to the cluster. For more information about joining nodes, see Join nodes on this page.

  Embedded Cluster automatically converts the installation to HA when three or more controller nodes are present.

Enable HA for an existing cluster

For online installations, the Kubernetes control plane becomes highly available once three controller nodes are present. For existing air gap installations, also run enable-ha to complete HA setup. Schedule a maintenance window because the internal registry is unavailable while its data is migrated. Air gap upgrades are blocked until this HA setup is complete.

To enable HA for an existing air gap installation with three or more controller nodes:

• On the controller named in the join output, which holds the registry data, run this command:

  sudo ./APP_SLUG enable-ha

  Where APP_SLUG is the unique slug for the application.

Reboot nodes

Embedded Cluster runs Kubernetes (k0s) as a systemd service, so a node automatically rejoins the cluster after a reboot. No special preparation is required — you do not need to cordon or drain the node before rebooting.

Single-node clusters: The application is unavailable while the node is rebooting. Schedule reboots during a maintenance window, and expect a short outage (typically a few minutes) while the node restarts and pods are rescheduled.

Multi-node clusters: Reboot one node at a time. Wait for the rebooted node to fully rejoin the cluster and report Ready before rebooting the next node. Do not reboot two or more controller nodes at the same time — losing multiple controllers simultaneously can break etcd quorum and take the entire cluster offline.

To check whether a node has rejoined after a reboot, run the following command from a controller node:

sudo ./APP_SLUG shell -c "kubectl get nodes"

Where APP_SLUG is the unique slug for the application. All nodes should show Ready status before proceeding with the next reboot.

Remove nodes and reset machines

This section describes how to remove individual nodes from a cluster and how to delete an entire multi-node cluster using the Embedded Cluster remove-node and reset commands.

About the remove-node and reset commands

To remove a node from a multi-node cluster, run the remove-node command from a controller node. Then, if you want to reuse the machine that you removed, run the reset command on it to remove Embedded Cluster from that machine.

Run the remove-node command from a controller node other than the node that you want to remove. The command performs the following steps:

1. Drain workloads from the node if it is healthy so that they reschedule onto other nodes in the cluster
2. For controller nodes, remove the node from the control plane
3. Remove the node from the cluster

The reset command removes Embedded Cluster and your application from the machine where it is run. This is useful for iteration, development, and when you make mistakes because you can reuse the machine instead of having to procure a new one.

The reset command performs the following steps:

1. Run safety checks. For example, reset does not proceed on a controller node that is still a member of a multi-node cluster's etcd
2. For worker nodes that are still in the cluster, drain the node and evict all the Pods gracefully
3. Stop and reset k0s
4. Remove all Embedded Cluster files

The reset command does not remove the node from the cluster. To reset a controller node in a multi-node cluster, first run remove-node from another controller node. If you run reset on a worker node without first running remove-node from a controller node, the node's workloads are drained, but the node remains in the cluster in a NotReady state until you remove it with remove-node.

For more information about these commands, see remove-node and reset.

Remove a node from a multi-node cluster

To remove a node from a multi-node cluster:

1. SSH onto a controller node other than the node that you want to remove.

2. Remove the target node from the cluster:

   sudo ./APP_SLUG remove-node NODE_NAME

   Where:

   • APP_SLUG is the unique slug for the application.
   • NODE_NAME is the Kubernetes node name of the node to remove. To list the node names in the cluster, run sudo ./APP_SLUG shell -c "kubectl get nodes".
   note

   Nodes that are NotReady skip draining automatically. Use --skip-drain to explicitly skip draining. If the node is already missing from the cluster, the CLI can clean up remaining records. It requires --force to delete stuck Pods or orphaned PVCs; deleting orphaned PVCs discards local data.

3. (Optional) Reset the machine that you removed so that you can reuse it. See Reset a machine in this document.

Reset a machine

For single-node installations, or for the last remaining controller node when deleting a multi-node cluster, run the reset command directly. For all other nodes in a multi-node cluster, first remove the node from the cluster by running remove-node from a controller node. See Remove a node from a multi-node cluster in this document.

To reset a machine:

1. SSH onto the node. Ensure that the Embedded Cluster binary is still available on the machine.

2. Run the following command to reset the machine:

   sudo ./APP_SLUG reset

   Where APP_SLUG is the unique slug for the application.

3. Reboot the machine.

Remove a multi-node cluster

To remove an entire multi-node cluster, remove each node with remove-node, starting with the worker nodes and ending with the controller nodes. On the last remaining controller node, run reset directly, because there is no other controller left to run remove-node from. Optionally, reset each machine that you remove so that you can reuse it. See Reset a machine in this document.

For more information, see Remove a node from a multi-node cluster in this document.

Limitations and best practices

Before you remove a node or delete a cluster, consider the following limitations and best practices:

• When you reset a node, Embedded Cluster deletes OpenEBS PVCs on that node. Kubernetes automatically recreates only PVCs created as part of a StatefulSet on another node in the cluster. To recreate other PVCs, redeploy the application in the cluster.

• If you need to remove one controller node in a three-node cluster, first join a fourth controller node to the cluster before removing the target node. This ensures that you maintain a minimum of three nodes for the Kubernetes control plane. You can add and remove worker nodes as needed because they do not have any control plane components.

• When resetting a single node or deleting a test environment, you can include the --force flag with the reset command to skip the confirmation prompt and continue past any errors.

• When removing a multi-node cluster, remove the worker nodes before the controller nodes. For more information, see Remove a multi-node cluster in this document.

• In air gap installations with high availability, SeaweedFS stores object storage data across nodes in the cluster. After removing a node, verify that all pods in the seaweedfs namespace are in a Running state before removing additional nodes.