Embedded Cluster installation requirements (Beta)

This topic lists the installation requirements for Replicated Embedded Cluster. Ensure that the installation environment meets these requirements before you attempt to install.

System requirements

• Linux operating system

• cgroups v2 (required for Kubernetes versions 1.35 and later)

• x86-64 architecture

• systemd

• At least 2GB of memory and 2 CPU cores

• The disk on the host must have a maximum P99 write latency of 10 ms. This supports etcd performance and stability. For more information about the disk write latency requirements for etcd, see Disks in Hardware recommendations and What does the etcd warning “failed to send out heartbeat on time” mean? in the etcd documentation.

• The user performing the installation must have root access to the machine, such as with sudo.

• The data directory used by Embedded Cluster must have 40Gi or more of total space and be less than 80% full. By default, the data directory is /var/lib/APP_SLUG, where APP_SLUG is the unique slug of the application. The directory can be changed by passing the --data-dir flag with the Embedded Cluster install command. For more information, see install.

  Note that in addition to the primary data directory, Embedded Cluster creates directories and files in the following locations:

  • /etc/cni
  • /etc/k0s
  • /opt/cni
  • /opt/containerd
  • /run/calico
  • /run/containerd
  • /run/k0s
  • /sys/fs/cgroup/kubepods
  • /sys/fs/cgroup/system.slice/containerd.service
  • /sys/fs/cgroup/system.slice/k0scontroller.service
  • /usr/libexec/k0s
  • /var/lib/calico
  • /var/lib/cni
  • /var/lib/containers
  • /var/lib/kubelet
  • /var/log/calico
  • /var/log/containers
  • /var/log/APP_SLUG, where APP_SLUG is the unique slug for the application
  • /var/log/pods
  • /usr/local/bin/k0s

• (Online installations only) Access to replicated.app and proxy.replicated.com or your custom domain for each

• Embedded Cluster is based on k0s, so all k0s system requirements and external runtime dependencies apply. See System requirements and External runtime dependencies in the k0s documentation.

Host configuration

Embedded Cluster configures most of the required kernel settings itself at the start of installation, before the host preflight checks run. The preflight checks then verify the result and report anything that could not be applied.

This means you do not need to set these values in advance. This section describes what Embedded Cluster changes on the host, which is often needed for security review, and what the host must provide for those changes to succeed.

Kernel settings that Embedded Cluster configures

Every change below is best effort. If one cannot be applied, installation continues and the host preflight checks report the problem.

Kernel modules. Loaded immediately, and written to /etc/modules-load.d/99-embedded-cluster.conf so that they persist across reboots:

overlay, ip_tables, nf_tables, nft_compat, br_netfilter, and nf_conntrack

Network parameters. Written to /etc/sysctl.d/99-embedded-cluster.conf and applied:

Parameter	Value
net.ipv4.ip_forward	1
net.ipv4.conf.all.forwarding	1
net.ipv4.conf.default.forwarding	1
net.ipv6.conf.all.forwarding	1
net.ipv6.conf.default.forwarding	1
net.bridge.bridge-nf-call-iptables	1
net.bridge.bridge-nf-call-ip6tables	1
net.ipv4.conf.all.arp_filter	0
net.ipv4.conf.default.arp_filter	0
net.ipv4.conf.all.arp_ignore	0
net.ipv4.conf.default.arp_ignore	0
net.ipv4.conf.all.rp_filter	2
net.ipv4.conf.default.rp_filter	2

Reverse path filtering is set to loose mode. Strict mode, which is the value 1, drops pod traffic.

Resource limits. Written to /etc/sysctl.d/99-dynamic-embedded-cluster.conf, and only for values that are currently lower on the host:

Parameter	Minimum value
fs.inotify.max_user_instances	1024
fs.inotify.max_user_watches	65536

What the host must provide

Embedded Cluster cannot supply the following, so the host must have them before you install:

• The modprobe command. If it is missing, Embedded Cluster skips kernel module setup entirely.
• The sysctl command. If it is missing, Embedded Cluster writes the network parameter file but cannot apply it until the host reboots, and does not set the inotify limits at all.
• The mount and umount commands.
• A kernel that provides the modules listed above. Embedded Cluster can load them, but cannot supply them on a kernel that was built without them.
• The following cgroup controllers: cpu, cpuacct, cpuset, memory, devices, freezer, and pids.
• A hostname of 54 characters or fewer. Internal component labels are derived from the node name, and a longer hostname exceeds the Kubernetes 63-character label limit, which prevents the cluster from starting.

Hardened hosts

On hosts that enforce their own kernel parameter policy, through configuration management or a file in /etc/sysctl.d that loads after the Embedded Cluster file, that policy can override these values after Embedded Cluster applies them. The host preflight checks report the resulting mismatch. Allow the values above in the host's policy rather than setting them by hand.

Port requirements

This section lists the ports used by Embedded Cluster. These ports must be open and available for both single- and multi-node installations.

Ports used by local processes

The following ports must be open and available for use by local processes running on the same node. It is not necessary to create firewall openings for these ports.

• 2379/TCP
• 7443/TCP
• 9099/TCP
• 10248/TCP
• 10257/TCP
• 10259/TCP

Ports required for bidirectional communication between nodes

The following ports are used for bidirectional communication between nodes.

For multi-node installations, create firewall openings between nodes for these ports.

For single-node installations, ensure that there are no other processes using these ports. Although there is no communication between nodes in single-node installations, these ports are still required.

• 2380/TCP
• 4789/UDP
• 6443/TCP
• 9091/TCP
• 9443/TCP
• 10249/TCP
• 10250/TCP
• 10256/TCP

Embedded Cluster UI port

Create a firewall opening for port 30080/TCP. The Embedded Cluster UI requires that port 30080/TCP is open and available.

Additionally, port 30080 must be accessible by nodes joining the cluster.

If port 30080 is occupied, you can select a different port for the Admin Console during installation. For more information, see install.

LAM port

The Local Artifact Mirror (LAM) requires that port 50000/TCP is open and available.

If port 50000 is occupied, you can select a different port for the LAM during installation. For more information, see install.

Unix accounts for Kubernetes components

During installation, Embedded Cluster automatically creates the following Unix accounts, which internal Kubernetes components require:

• etcd: Used by the Kubernetes etcd database, which stores cluster state.
• konnectivity-server: Used by the Konnectivity service, which facilitates secure communication between internal components.
• kube-apiserver: Used by the Kubernetes API server.
• kube-scheduler: Used by the Kubernetes scheduler to schedule workloads such as pods.

You do not need to take any action to create these roles. Removing them will make the cluster non-functional.

For more information about the internal Kubernetes components, see Kubernetes Components in the Kubernetes documentation.

For more information about the Konnectivity service, see Set up Konnectivity service in the Kubernetes documentation.

Firewall openings for online installations with Embedded Cluster

The domains for the services listed below need to be accessible from servers performing online installations. No outbound internet access is required for air gap installations.

For services hosted at domains owned by Replicated, the table includes a link to the list of IP addresses for the domain at replicatedhq/ips in GitHub. These are the addresses that Replicated services listen on, so they are the addresses that an online installation connects to.

That repository covers every Replicated service, so it lists more domains than any one installation needs. Use the table below to determine which of its entries apply to you.

note

Replicated does not publish the source IP addresses that its services use for outbound connections, such as when the Replicated proxy registry pulls an image from a vendor's registry, and does not commit to keeping those addresses stable. Do not use the addresses in replicatedhq/ips to build an allowlist for traffic originating from Replicated. Where access to a registry must be restricted, use registry credentials rather than a source IP allowlist.

Some of the domains listed below belong to public services that Replicated does not own, such as Docker Hub, GitHub and Amazon S3. Their IP addresses are not in replicatedhq/ips, so consult the provider's documentation for their ranges.

Domain	Description
proxy.replicated.com	Private Docker images are proxied through proxy.replicated.com. This domain is owned by Replicated, Inc., which is headquartered in Los Angeles, CA. For the range of IP addresses for proxy.replicated.com, see replicatedhq/ips in GitHub.
replicated.app	Upstream application YAML and metadata is pulled from replicated.app. The current running version of the application (if any), as well as a license ID and application ID to authenticate, are all sent to replicated.app. This domain is owned by Replicated, Inc., which is headquartered in Los Angeles, CA. For the range of IP addresses for replicated.app, see replicatedhq/ips in GitHub.
registry.replicated.com *	Some applications host private images in the Replicated registry at this domain. The on-prem docker client uses a license ID to authenticate to registry.replicated.com. This domain is owned by Replicated, Inc which is headquartered in Los Angeles, CA. For the range of IP addresses for registry.replicated.com, see replicatedhq/ips in GitHub.

* Required only if the application uses the Replicated private registry.

note

If you monitor the outbound traffic attempts made by Embedded Cluster, you might see an attempted call to updates.k0sproject.io approximately every 30 minutes. A feature of the upstream k0s project called update-prober makes these calls to check for k0s updates. Embedded Cluster does not use the update-prober feature and blocking it in your firewall rules will not affect Replicated product functionality.

About firewalld configuration

When Firewalld runs in the installation environment, Embedded Cluster modifies the Firewalld config to allow traffic over the pod and service networks. It also opens the required ports on the host. No additional configuration is necessary.

Embedded Cluster adds the following rule to Firewalld:

<?xml version="1.0" encoding="utf-8"?>
<zone target="ACCEPT">
  <interface name="cali+"/>
  <interface name="tunl+"/>
  <interface name="vxlan-v6.calico"/>
  <interface name="vxlan.calico"/>
  <interface name="wg-v6.cali"/>
  <interface name="wireguard.cali"/>
  <source address="[pod-network-cidr]"/>
  <source address="[service-network-cidr]"/>
</zone>

Embedded Cluster opens the following ports in the default zone:

Port	Protocol
6443	TCP
10250	TCP
9443	TCP
2380	TCP
4789	UDP