Embedded Cluster installation requirements (Beta)

This topic lists the installation requirements for Replicated Embedded Cluster. Ensure that the installation environment meets these requirements before you attempt to install.

System requirements

• Linux operating system

• cgroups v2 (required for Kubernetes versions 1.35 and later)

• x86-64 architecture

• systemd

• At least 2GB of memory and 2 CPU cores

• The disk on the host must have a maximum P99 write latency of 10 ms. This supports etcd performance and stability. For more information about the disk write latency requirements for etcd, see Disks in Hardware recommendations and What does the etcd warning “failed to send out heartbeat on time” mean? in the etcd documentation.

• The user performing the installation must have root access to the machine, such as with sudo.

• The data directory used by Embedded Cluster must have 40Gi or more of total space and be less than 80% full. By default, the data directory is /var/lib/APP_SLUG, where APP_SLUG is the unique slug of the application. The directory can be changed by passing the --data-dir flag with the Embedded Cluster install command. For more information, see install.

  Note that in addition to the primary data directory, Embedded Cluster creates directories and files in the following locations:

  • /etc/cni
  • /etc/k0s
  • /opt/cni
  • /opt/containerd
  • /run/calico
  • /run/containerd
  • /run/k0s
  • /sys/fs/cgroup/kubepods
  • /sys/fs/cgroup/system.slice/containerd.service
  • /sys/fs/cgroup/system.slice/k0scontroller.service
  • /usr/libexec/k0s
  • /var/lib/calico
  • /var/lib/cni
  • /var/lib/containers
  • /var/lib/kubelet
  • /var/log/calico
  • /var/log/containers
  • /var/log/APP_SLUG, where APP_SLUG is the unique slug for the application
  • /var/log/pods
  • /usr/local/bin/k0s

• (Online installations only) Access to replicated.app and proxy.replicated.com or your custom domain for each

• Embedded Cluster is based on k0s, so all k0s system requirements and external runtime dependencies apply. See System requirements and External runtime dependencies in the k0s documentation.

Host configuration

Embedded Cluster configures most of the required kernel settings itself at the start of installation, before the host preflight checks run. The preflight checks then verify the result and report anything that could not be applied.

This means you do not need to set these values in advance. This section describes what Embedded Cluster changes on the host, which is often needed for security review, and what the host must provide for those changes to succeed.

Kernel settings that Embedded Cluster configures

Every change below is best effort. If one cannot be applied, installation continues and the host preflight checks report the problem.

Kernel modules. Loaded immediately, and written to /etc/modules-load.d/99-embedded-cluster.conf so that they persist across reboots:

overlay, ip_tables, nf_tables, nft_compat, br_netfilter, and nf_conntrack

Network parameters. Written to /etc/sysctl.d/99-embedded-cluster.conf and applied:

Parameter	Value
net.ipv4.ip_forward	1
net.ipv4.conf.all.forwarding	1
net.ipv4.conf.default.forwarding	1
net.ipv6.conf.all.forwarding	1
net.ipv6.conf.default.forwarding	1
net.bridge.bridge-nf-call-iptables	1
net.bridge.bridge-nf-call-ip6tables	1
net.ipv4.conf.all.arp_filter	0
net.ipv4.conf.default.arp_filter	0
net.ipv4.conf.all.arp_ignore	0
net.ipv4.conf.default.arp_ignore	0
net.ipv4.conf.all.rp_filter	2
net.ipv4.conf.default.rp_filter	2

Reverse path filtering is set to loose mode. Strict mode, which is the value 1, drops pod traffic.

Resource limits. Written to /etc/sysctl.d/99-dynamic-embedded-cluster.conf, and only for values that are currently lower on the host:

Parameter	Minimum value
fs.inotify.max_user_instances	1024
fs.inotify.max_user_watches	65536

What the host must provide

Embedded Cluster cannot supply the following, so the host must have them before you install:

• The modprobe command. If it is missing, Embedded Cluster skips kernel module setup entirely.
• The sysctl command. If it is missing, Embedded Cluster writes the network parameter file but cannot apply it until the host reboots, and does not set the inotify limits at all.
• The mount and umount commands.
• A kernel that provides the modules listed above. Embedded Cluster can load them, but cannot supply them on a kernel that was built without them.
• The following cgroup controllers: cpu, cpuacct, cpuset, memory, devices, freezer, and pids.
• A hostname of 54 characters or fewer. Internal component labels are derived from the node name, and a longer hostname exceeds the Kubernetes 63-character label limit, which prevents the cluster from starting.

Hardened hosts

On hosts that enforce their own kernel parameter policy, through configuration management or a file in /etc/sysctl.d that loads after the Embedded Cluster file, that policy can override these values after Embedded Cluster applies them. The host preflight checks report the resulting mismatch. Allow the values above in the host's policy rather than setting them by hand.

Port requirements

This section lists the ports used by Embedded Cluster. These ports must be open and available for both single- and multi-node installations.

Ports used by local processes

The following ports must be open and available for use by local processes running on the same node. It is not necessary to create firewall openings for these ports.

• 2379/TCP
• 7443/TCP
• 9099/TCP
• 10248/TCP
• 10257/TCP
• 10259/TCP

Ports required for bidirectional communication between nodes

The following ports are used for bidirectional communication between nodes.

For multi-node installations, create firewall openings between nodes for these ports.

For single-node installations, ensure that there are no other processes using these ports. Although there is no communication between nodes in single-node installations, these ports are still required.

• 2380/TCP
• 4789/UDP
• 6443/TCP
• 9091/TCP
• 9443/TCP
• 10249/TCP
• 10250/TCP
• 10256/TCP

Embedded Cluster UI port

Create a firewall opening for port 30080/TCP. The Embedded Cluster UI requires that port 30080/TCP is open and available.

Additionally, port 30080 must be accessible by nodes joining the cluster.

If port 30080 is occupied, you can select a different port for the Admin Console during installation. For more information, see install.

Embedded Cluster API port

The Embedded Cluster daemon serves an HTTP API on port 30081/TCP on every controller node. Nodes use it to join the cluster and to receive streamed upgrades from the primary controller, and external automation uses it for headless installs and programmatic node joins.

For multi-node installations, create firewall openings between nodes for port 30081/TCP. Single-node installations do not need a firewall opening for this port unless you drive the API from outside the host.

If port 30081 is occupied, you can select a different port for the API during installation. For more information, see install and External API.

LAM port

The Local Artifact Mirror (LAM) requires that port 50000/TCP is open and available.

If port 50000 is occupied, you can select a different port for the LAM during installation. For more information, see install.

Unix accounts for Kubernetes components

During installation, Embedded Cluster automatically creates the following Unix accounts, which internal Kubernetes components require:

• etcd: Used by the Kubernetes etcd database, which stores cluster state.
• konnectivity-server: Used by the Konnectivity service, which facilitates secure communication between internal components.
• kube-apiserver: Used by the Kubernetes API server.
• kube-scheduler: Used by the Kubernetes scheduler to schedule workloads such as pods.

You do not need to take any action to create these roles. Removing them will make the cluster non-functional.

For more information about the internal Kubernetes components, see Kubernetes Components in the Kubernetes documentation.

For more information about the Konnectivity service, see Set up Konnectivity service in the Kubernetes documentation.

Firewall openings for online installations with Embedded Cluster

The domains for the services listed below need to be accessible from servers performing online installations. No outbound internet access is required for air gap installations.

For services hosted at domains owned by Replicated, the table includes a link to the list of IP addresses for the domain at replicatedhq/ips in GitHub. These are the addresses that Replicated services listen on, so they are the addresses that an online installation connects to.

That repository covers every Replicated service, so it lists more domains than any one installation needs. Use the table below to determine which of its entries apply to you.

note

Replicated does not publish the source IP addresses that its services use for outbound connections, such as when the Replicated proxy registry pulls an image from a vendor's registry, and does not commit to keeping those addresses stable. Do not use the addresses in replicatedhq/ips to build an allowlist for traffic originating from Replicated. Where access to a registry must be restricted, use registry credentials rather than a source IP allowlist.

Some of the domains listed below belong to public services that Replicated does not own, such as Docker Hub, GitHub and Amazon S3. Their IP addresses are not in replicatedhq/ips, so consult the provider's documentation for their ranges.

Domain	Description
proxy.replicated.com	Private Docker images are proxied through proxy.replicated.com. This domain is owned by Replicated, Inc., which is headquartered in Los Angeles, CA. For the range of IP addresses for proxy.replicated.com, see replicatedhq/ips in GitHub.
replicated.app	Upstream application YAML and metadata is pulled from replicated.app. The current running version of the application (if any), as well as a license ID and application ID to authenticate, are all sent to replicated.app. This domain is owned by Replicated, Inc., which is headquartered in Los Angeles, CA. For the range of IP addresses for replicated.app, see replicatedhq/ips in GitHub.
registry.replicated.com *	Some applications host private images in the Replicated registry at this domain. The on-prem docker client uses a license ID to authenticate to registry.replicated.com. This domain is owned by Replicated, Inc which is headquartered in Los Angeles, CA. For the range of IP addresses for registry.replicated.com, see replicatedhq/ips in GitHub.

* Required only if the application uses the Replicated private registry.

note

If you monitor the outbound traffic attempts made by Embedded Cluster, you might see an attempted call to updates.k0sproject.io approximately every 30 minutes. A feature of the upstream k0s project called update-prober makes these calls to check for k0s updates. Embedded Cluster does not use the update-prober feature and blocking it in your firewall rules will not affect Replicated product functionality.

About firewalld configuration

When Firewalld runs in the installation environment, Embedded Cluster modifies the Firewalld config to allow traffic over the pod and service networks. It also opens the required ports on the host. No additional configuration is necessary.

Embedded Cluster adds the following rule to Firewalld:

<?xml version="1.0" encoding="utf-8"?>
<zone target="ACCEPT">
  <interface name="cali+"/>
  <interface name="tunl+"/>
  <interface name="vxlan-v6.calico"/>
  <interface name="vxlan.calico"/>
  <interface name="wg-v6.cali"/>
  <interface name="wireguard.cali"/>
  <source address="[pod-network-cidr]"/>
  <source address="[service-network-cidr]"/>
</zone>

Embedded Cluster opens the following ports in the default zone:

Port	Protocol
6443	TCP
10250	TCP
9443	TCP
2380	TCP
4789	UDP

Security tools on the host

Endpoint security, firewall management, and application control tools, such as Illumio VEN or CrowdStrike Falcon, can interfere with Embedded Cluster. Replicated recommends disabling these tools on Embedded Cluster hosts. If disabling a tool is not an option in the installation environment, configure the tool to meet the requirements in the following table. Each requirement includes the reason that Embedded Cluster needs it. Configuration steps vary by product and version, so work with the tool's vendor to apply them.

Requirement	Why
Do not block, flush, replace, or reorder the KUBE-* and cali-* firewall rules, whether the host uses iptables or nftables. The usual approach is to exclude cluster nodes from the tool's host firewall management, or to use the tool's Kubernetes mode if it has one.	The cluster's networking components, kube-proxy and Calico, route all pod and service traffic through these rules. If a tool blocks the rules during installation, kube-proxy restarts repeatedly and the installation does not complete. If a tool removes the rules later, such as after the tool restarts, receives a policy update, or the host reboots, pods can no longer reach the Kubernetes API and the application stops working.
Allow traffic to and from the pod and service networks, and on the cali* and vxlan.calico network interfaces. For more information about the interfaces and networks, see About firewalld configuration in this document.	All traffic between pods, services, and the host uses these networks and interfaces. A default-deny policy drops this traffic, with the same result as removing the firewall rules.
Allow binaries to run from the Embedded Cluster executable paths. For more information about the paths, see Executable paths in this document.	Embedded Cluster runs Kubernetes and its networking components from these paths. If a tool blocks execution, installation or upgrade fails, or the node cannot start the cluster.
Do not quarantine or remove files in the Embedded Cluster executable paths or data directory. For more information about the executable paths, see Executable paths in this document. For more information about the data directory, see Data directory contents in this document. If the tool scans files, also exclude DATA_DIR/k0s/containerd from scanning, where DATA_DIR is the Embedded Cluster data directory.	Removing a cluster binary, a systemd unit, or a networking binary stops the node. Scanners that hold files open in container filesystems can also prevent containers from restarting.
Do not change the kernel modules and parameters that Embedded Cluster configures. For more information about the kernel modules and parameters, see Host configuration in this document.	Hardening tools that reset values such as net.ipv4.ip_forward or net.ipv4.conf.all.rp_filter drop pod traffic, even when no firewall rule changed.

Executable paths

Security tools that control which binaries can run must allow execution in the following paths, where DATA_DIR is the Embedded Cluster data directory:

• /usr/local/bin: The Embedded Cluster CLI and its services
• DATA_DIR/bin: Kubernetes and the tools that Embedded Cluster uses to install and support the cluster
• DATA_DIR/k0s: Kubernetes components and the container runtime
• DATA_DIR/tmp: Scripts that Embedded Cluster runs during installation and upgrade
• /opt/cni: Calico networking binaries

Before it installs or upgrades, Embedded Cluster checks that the first four paths allow execution. If a path does not, Embedded Cluster stops with an error that names the blocked path.

Data directory contents

Security tools often flag unfamiliar binaries and certificates in /var/lib. Knowing what each directory holds helps security teams write narrow exceptions instead of broad ones, which are easier to approve. The data directory contains the following:

• bin: Kubernetes and the tools that Embedded Cluster uses to install and support the cluster
• k0s: Kubernetes components and their data, including the container runtime and every container image and container filesystem on the node. This includes the application images. The pki subdirectory holds certificates that cluster components use only to communicate with each other.
• openebs-local: Persistent storage for the application
• tmp: Temporary files that Embedded Cluster uses during installation and upgrade