Using Embedded Cluster (Beta)

This topic describes how to use the Replicated embedded cluster to configure, install, and manage your application in an embedded Kubernetes cluster.

note

Embedded cluster is in beta. If you are instead looking for information about creating Kubernetes Installers with Replicated kURL, see the Replicated kURL section.

Overview

Replicated embedded cluster allows you to distribute a Kubernetes cluster and your application together as a single appliance, making it easy for enterprise users to install, update, and manage the application and the cluster in tandem. Embedded cluster is based on the open source Kubernetes distribution k0s. For more information, see the k0s documentation.

For software vendors, embedded cluster provides a simplified config for defining characteristics of the cluster that will be created in the customer environment. Additionally, each version of embedded cluster includes a specific version of Replicated KOTS, ensuring compatibility between KOTS and the cluster. For enterprise users, cluster updates are done automatically at the same time as application updates, allowing users to more easily keep the cluster up-to-date without needing to use kubectl.

Embedded cluster is a successor to Replicated kURL. Compared to kURL, embedded cluster offers several improvements such as:

• Significantly faster installation, updates, and node joins
• A redesigned admin console UI for managing the cluster
• Improved support for multi-node clusters
• One-click updates of both the application and the cluster at the same time

Requirements

Embedded cluster has the following requirements:

• Linux operating system
• x86-64 architecture
• Embedded cluster is based on k0s, so all k0s system requirements apply. See System requirements in the k0s documentation.

Limitations and Known Issues

This section lists the limitations and known issues for embedded cluster.

Limitations

Embedded cluster has the following limitations:

• Multi-node support is in alpha: You can create multi-node embedded clusters, but this feature is in alpha. Only single-node embedded clusters are in beta.

• No migration from kURL: There is not yet a way to migrate a kURL instance to embedded cluster. We intend to build a migration or upgrade for kURL instances. There are many ways that this could be built, each with various tradeoffs.

• Cannot change spec.api and spec.storage after installation: The spec.api and spec.storage keys in the k0s config cannot be changed after installation. Whereas most keys in the k0s config apply to the whole cluster, these two keys are set for each node. Embedded cluster cannot update these keys on each individual node during updates, so they cannot be changed after installation.

• GitOps workflow not supported: Embedded cluster does not support using the GitOps workflow. We do not intend to support the GitOps workflow for embedded cluster in the future. If an end-user is interested in GitOps, consider the Helm install method instead.

Known Issue

If you use Helm extensions and a Helm chart fails to install, you can make changes to your embedded cluster config, promote a new release, and update to that version from the admin console to upgrade the chart. However, there will be a several minute delay before the Helm chart updates.

For more information about using Helm extensions, see Helm Extensions in Embedded Cluster Config.

kURL Feature Parity

The following features that are available in kURL are not yet available in embedded cluster. Unless otherwise noted, some of these features might not be available at GA:

• Air gap: Air gap support is not yet implemented for embedded cluster. We intend to implement support for air gap installations in the future.

• Backup and restore with Velero: Velero is not yet installed as part of embedded cluster, so backup and restore is unavailable.

• Distributed storage with Rook: OpenEBS is the only storage provider available today. We plan to introduce Rook as an alternative for multi-node clusters after GA.

• Monitoring with Prometheus: The Prometheus stack is not deployed as part of embedded cluster. No default graphs show on the admin console dashboard, and you cannot create custom graphs.

• Access the admin console with an identity provider: Identity provider is a beta feature that is exclusive to kURL. It is not available for embedded cluster installations.

note

If you require any of these features to use embedded cluster, reach out to Alex Parker at [email protected].

Quick Start

You can use the following steps to get started quickly with embedded cluster. More detailed documentation is available below.

1. Create a new customer or edit an existing customer and select the Embedded Cluster Enabled (Beta) license option. Save the customer.

2. Create a new release that includes your application. In that release, create an embedded cluster config that includes, at minimum, the embedded cluster version you want to use.

   Example embedded cluster config:

   apiVersion: embeddedcluster.replicated.com/v1beta1
   kind: Config
   spec:
     version: 1.29.2+ec.4

3. Save the release and promote it to the channel your customer is assigned to.

4. Return to the customer page where you enabled embedded cluster. At the top right, click Install instructions and choose Embedded cluster. A dialog appears with instructions on how to download the embedded cluster binary and install your application.

   

   View a larger version of this image

5. On your VM, run the commands in the Embedded Cluster install instructions dialog.

   

   View a larger version of this image

6. Enter an admin console password when prompted.

   The admin console URL is printed when the installation finishes. Access the admin console to begin installing your application. You’ll have the opportunity to add nodes if you want a multi-node cluster. Then you can provide application config, run preflights, and deploy your application.

Configure Embedded Cluster

To install your application with embedded cluster, an embedded cluster config must be created in a release. The embedded cluster config lets you define several characteristics about the cluster that will be created.

For more information, see Embedded Cluster Config.

Download the Embedded Cluster Binary

To install your application with embedded cluster, you need to download an embedded cluster binary and a license. The binary is specific to your application release: it installs that particular version of your application using the embedded cluster config defined in that release.

Your customers can download the binary using the Replicated app service, or you can host/serve the binary yourself using the Replicated Vendor API.

Using the Replicated App Service

For online installations, your customers can run a command to download the embedded cluster binary and their license directly to the machine where they will install your application. The command to download the binary and license is easily available on the customer page in the vendor portal. You can copy this and share it with your customers.

To download the binary using the Replicated app service:

1. Create a new customer or edit an existing customer and select the Embedded Cluster Enabled (Beta) license option. Save the customer.

2. At the top right of the customer page, click Install instructions and choose Embedded cluster. An Embedded cluster installation instructions dialog appears with instructions on how to download the embedded cluster binary and install your application.

   

   View a larger version of this image

3. On a VM, run the first two commands displayed on the Embedded cluster installation instructions dialog to download the binary and license and extract them.

   The following shows an example of the Embedded cluster installation instructions dialog:

   

   View a larger version of this image

   note

   You can configure a custom domain for the replicated.app endpoint so that your customers can download the binary from your own domain instead of from replicated.app. For more information, see About Custom Domains.

Using the Vendor API

Some vendors already have a portal that their customers log in to to access documentation or download artifacts. In cases like this, you can serve the embedded cluster binary yourself using the Replicated Vendor API.

To serve the binary with the Vendor API:

1. If you have not done so already, create an API token for the Vendor API. See Using the Vendor API v3.

2. Call the Get an embedded cluster release endpoint to download the artifacts needed to install your application in an embedded cluster. Your customers must take this binary and their license and copy them to the machine where they will install your application.

   Note the following:

   • (Recommended) Provide the customerId query parameter so that the customer’s license is included in the downloaded tarball. This mirrors what is returned when a customer downloads the binary directly using the Replicated app service and is the most useful option. Excluding the customerId is useful if you plan to distribute the license separately.

   • If you do not provide any query parameters, this endpoint downloads the embedded cluster binary for the latest release on the specified channel. You can provide the channelSequence query parameter to download the binary for a particular release.

Install

Your customers can install your application once they have obtained the embedded cluster binary and their license and copied them to the machine where they will install your application. For more information about how to get the binary and license, see Download the Embedded Cluster Binary above.

To install your application in an embedded cluster:

1. SSH onto the machine where you will install your application.

2. Ensure that the embedded cluster binary and the license are available in the working directory.

3. Install the application:

   sudo ./APP_SLUG install --license LICENSE_FILE

   Where:

   • APP_SLUG is the unique slug for the application.
   • LICENSE_FILE is the customer's license.

4. Enter an admin console password when prompted.

   The admin console URL is printed when the installation finishes. Access the admin console to begin installing your application. You’ll have the opportunity to add nodes if you want a multi-node cluster. Then you can provide application config, run preflights, and deploy your application.

Update

When you update an application installed in an embedded cluster, you update both the application and the cluster together. There is no need or mechanism to update the embedded cluster on its own.

You can check for available application updates in the admin console, or use automatic updates, to retrieve newly available versions. When you deploy a new version, any changes to your application are deployed first. The admin console waits until your application is ready, as indicated by status informers, before making any changes to the cluster.

Any changes made to the embedded cluster config, including changes to the embedded cluster version, Helm extensions, and unsupported overrides, trigger a cluster update.

During cluster updates, the admin console can experience downtime and become unavailable if the Kubernetes version is updated. If you’re already viewing the admin console when the downtime occurs, you will see a message informing you that a cluster update is in progress and that the connection will resume once the admin console is available again:

View a larger version of this image

important

Do not downgrade the embedded cluster version. This is not supported but is not prohibited, and it can lead to unexpected behavior.

Access the Cluster

With embedded cluster, end-users are rarely supposed to need to use the CLI. Typical workflows, like updating the application and the cluster, are driven through the admin console.

Nonetheless, there are times when vendors or their customers need to use the CLI for development or troubleshooting.

To access the cluster and use other included binaries:

1. SSH onto a controller node.

2. Use the embedded cluster shell command to start a shell with access to the cluster:

   sudo ./APP_SLUG shell

   The output looks similar to the following:

      __4___
   _  \ \ \ \   Welcome to APP_SLUG debug shell.
   <'\ /_/_/_/   This terminal is now configured to access your cluster.
   ((____!___/) Type 'exit' (or CTRL+d) to exit.
   \0\0\0\0\/  Happy hacking.
   ~~~~~~~~~~~
   root@alex-ec-2:/home/alex# export KUBECONFIG="/var/lib/k0s/pki/admin.conf"
   root@alex-ec-2:/home/alex# export PATH="$PATH:/var/lib/embedded-cluster/bin"
   root@alex-ec-2:/home/alex# source <(kubectl completion bash)
   root@alex-ec-2:/home/alex# source /etc/bash_completion

   The appropriate kubeconfig is exported, and the location of useful binaries like kubectl and Replicated’s preflight and support-bundle plugins is added to PATH.

   note

   The shell command cannot be run on non-controller nodes.

3. Use the available binaries as needed.

   Example:

   kubectl version

   Client Version: v1.29.1
   Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
   Server Version: v1.29.1+k0s

4. Type exit or use Ctrl+D to exit the shell.

   note

   If you encounter a typical workflow where your customers have to use the embedded cluster shell, reach out to Alex Parker at [email protected]. These workflows might be candidates for additional admin console functionality.

Manage Nodes

The section describes managing nodes in clusters created with embedded cluster, including how to add or reset nodes.

Add Nodes (Alpha)

You can add nodes and create a multi-node cluster. When adding nodes, you select one or more roles for that node, depending on which roles are defined in the embedded cluster config. The admin console provides the join command you use to join nodes to the cluster.

For more information about defining node roles, see Roles in Embedded Cluster Config.

To add nodes to a cluster:

1. In the admin console, click Cluster Management at the top.

   When initially installing the application, you are brought to this page automatically after logging into the admin console.

2. Click Add node.

3. In the Add a Node dialog, select one or more roles for this node. If no custom roles are defined, the role selection will not appear and only the join command will show.

   

   View a larger version of this image

4. Copy the provided join command.

   Example:

   sudo ./APP_SLUG join 10.128.0.43:30000 bM8DO3MNvkouz9TFK3TcFanI

5. SSH onto the machine you want to join to the cluster. Ensure that the embedded cluster binary is available on that node. For more information on downloading the embedded cluster binary, see Installing.

   important

   You must join nodes with the same installer that you used for the first node. If you use a different installer from a different release of your application, the cluster will not be stable.

6. Run the copied join command using the embedded cluster binary.

7. Check the Cluster Management page in the admin console to see the node appear and wait for the status to change to Ready.

8. Repeat the process for as many nodes as you would like to add.

Reset a Node

Resetting a node removes the cluster and your application from that node. This is useful for iteration, development, and when mistakes are made, so you can reset a machine and reuse it instead of having to procure another machine.

If you want to completely remove a cluster, you need to reset each node individually.

To reset the node of a cluster:

1. SSH onto the machine. Ensure that the embedded cluster binary is still available on that machine. For more information on downloading the embedded cluster binary, see Download the Embedded Cluster Binary above.

2. Run the reset command to reset the node. The --reboot flag automatically reboots the machine to ensure that transient configuration is also reset.

   sudo ./APP_SLUG reset --reboot
   note

   Pass the --no-prompt flag to disable interactive prompts. Pass the --force flag to ignore any errors encountered during the reset.

Additional Use Cases

This section outlines some additional use cases for embedded cluster. These are not officially supported features from Replicated, but are ways of using embedded cluster that we or our customers have experimented with that might be useful to you.

NVIDIA GPU Operator

The NVIDIA GPU Operator uses the operator framework within Kubernetes to automate the management of all NVIDIA software components needed to provision GPUs. For more information about this operator, see the NVIDIA GPU Operator documentation. You can include the operator in your release as an additional Helm chart, or using the embedded cluster Helm extensions. For information about Helm extensions, see Helm extensions in Embedded Cluster Config.

Using this operator with embedded cluster requires configuring the containerd options in the operator as follows:

toolkit:
   env:
   - name: CONTAINERD_CONFIG
     value: /etc/k0s/containerd.d/nvidia.toml
   - name: CONTAINERD_SOCKET
     value: /run/k0s/containerd.sock