Skip to main content

Embedded Cluster Config (Beta)

This topic is a reference for the Replicated Embedded Cluster Config custom resource. For more information about Embedded Cluster, see Use Embedded Cluster.

Limitations​

  • Most of the fields in the Config spec are plain YAML and do not support Go template functions, including Replicated template functions. The values field for each entry in extensions.helmCharts does support the use of template functions.

  • There are additional, property-specific limitations. For more information, see the following sections.

Example​

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
metadata:
  name: my-embedded-cluster-config
spec:
  version: "3.0.0-alpha-31+k8s-1.34"
  domains:
    proxyRegistryDomain: proxy.example.com
    replicatedAppDomain: updates.example.com
  unsupportedOverrides:
    k0s: |
      config:
        metadata:
          name: foo
        spec:
          telemetry:
            enabled: false
          workerProfiles:
            - name: ip-forward
              values:
                allowedUnsafeSysctls:
                  - net.ipv4.ip_forward
  extensions:
    helmCharts:
      - chart:
          name: ingress-nginx
          chartVersion: "4.11.3"
        releaseName: ingress-nginx
        namespace: ingress-nginx
        values: |
          controller:
            service:
              type: NodePort
              nodePorts:
                http: "80"
                https: "443"
            image:
              repository: 'repl{{ ReplicatedImageName "registry.k8s.io/ingress-nginx/controller" }}'
              tag: 'repl{{ HelmValue ".Values.controller.image.tag" }}'
              digest: ""
              digestChroot: ""
            admissionWebhooks:
              patch:
                image:
                  repository: 'repl{{ ReplicatedImageName "registry.k8s.io/ingress-nginx/kube-webhook-certgen" }}'
                  tag: 'repl{{ HelmValue ".Values.controller.admissionWebhooks.patch.image.tag" }}'
                  digest: ""
      - chart:
          name: goldpinger
          chartVersion: "6.1.2"
        releaseName: goldpinger
        namespace: goldpinger
        weight: 11
        values: |
          image:
            repository: 'repl{{ ReplicatedImageName "docker.io/bloomberg/goldpinger" }}'

version (Required)​

Specify the versions of Embedded Cluster and Kubernetes to install.

Embedded Cluster appends the version of Kubernetes to the version label in the format +k8s-1.34. For example, Embedded Cluster 3.0.0-alpha-26+k8s-1.34 uses Embedded Cluster 3.0.0-alpha-26 and Kubernetes 1.34.

Each version of Embedded Cluster includes specific versions of components like OpenEBS. For more information, see the Embedded Cluster Release Notes.

Replicated recommends that you update the version frequently to ensure that you are using the latest version of Embedded Cluster.

important

Don't downgrade the Embedded Cluster version or the Kubernetes version. Also, only increase the Kubernetes version by one minor version at a time. For more information, see Limitations in Embedded Cluster Overview.

domains​

Configure the domains key so that Embedded Cluster uses your custom domains for the Replicated proxy registry and Replicated app service.

Embedded Cluster uses the custom domains specified in domains.proxyRegistryDomain and domains.replicatedAppDomain when making requests to the given service.

You must add the custom domains that you specify in the domains.proxyRegistryDomain and domains.replicatedAppDomain fields to the Vendor Portal before Embedded Cluster can use them. For more information, see Add a Custom Domain in the Vendor Portal in Using Custom Domains.

If domains.proxyRegistryDomain and domains.replicatedAppDomain are not set, Embedded Cluster uses the default Replicated domains. For more information about aliasing Replicated endpoints with custom domains, see About Custom Domains.

Example​

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  domains:
    proxyRegistryDomain: proxy.mycompany.com
    replicatedAppDomain: updates.mycompany.com

extensions​

Embedded Cluster installs Helm charts before your application as part of Embedded Cluster itself. Specify these charts in spec.extensions.helmCharts. Each item describes one chart release.

helmCharts​

Each object in helmCharts can include the following fields:

  • chart: (Required) Identifies the chart.
    • name: Chart name.
    • chartVersion: Chart version string to install.
  • releaseName: (Required) Helm release name.
  • namespace: (Required) Namespace for the release.
  • weight: Controls installation order relative to other Helm chart extensions. Lower weights install before higher weights.
  • values: Multi-line string of Helm values. This field supports Replicated template functions. Embedded Cluster updates Helm extensions when users deploy new versions of your application. For example, you can change values from one release to another, and those changes apply when the user deploys the new version.

Requirements​

  • You must provide chart.chartVersion. Omitting it can cause upgrade issues.

  • For air gap bundles, images used by Helm extensions must not refer to a multi-architecture image by digest. Air gap bundles include only x64 images, and the digest for the x64 image will be different from the digest for the multi-architecture image, preventing Kubernetes from locating the image in the bundle. For an example of how to set digests to empty strings and pull by tag only, see the ingress-nginx Example.

Example​

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  extensions:
    helmCharts:
      - chart:
          name: ingress-nginx
          chartVersion: "4.11.3"
        releaseName: ingress-nginx
        namespace: ingress-nginx
        values: |
          controller:
            service:
              type: NodePort
              nodePorts:
                http: "80"
                https: "443"
            image:
              digest: ""
              digestChroot: ""

unsupportedOverrides​

important

Advanced users who understand the risks and ramifications of changing the default configuration should use this feature with caution.

Use unsupported overrides to change Embedded Cluster's default configuration, including the k0s config and Helm values for built-in extensions. Use overrides carefully. Replicated has not tested these combinations and they can disrupt clusters. Replicated does not support issues caused by unsupported overrides.

While you should use them with caution, unsupported overrides are useful if you need to make changes that Embedded Cluster does not otherwise expose.

Override the k0s Config​

By default, Embedded Cluster uses a k0s config that Replicated has tested and confirmed works for Embedded Clusters. In some circumstances, you might want to change the k0s config.

For more information on the k0s config, see Configuration options in the k0s documentation.

The value of unsupportedOverrides.k0s is a string containing YAML that becomes (or patches) the k0s configuration Embedded Cluster uses. How deeply Embedded Cluster merges that YAML with its defaults depends on the field. Treat overrides as high risk and validate in a non-production cluster.

For example, you can enable WireGuard-based encryption. You might need to add other configuration. See spec.network.calico in the k0s documentation:

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  unsupportedOverrides:
    k0s: |
      config:
        spec:
          network:
            calico:
              wireguard: true

Limitations​

  • You cannot change the spec.api and spec.storage keys in the k0s config after installation. Whereas most keys in the k0s config apply to the whole cluster, k0s sets these two keys for each node individually. Embedded Cluster cannot update these keys on each individual node during updates, so you cannot change them after installation.

  • Overrides can replace whole sections of the k0s configuration rather than deep-merge with every default. When overriding a list (for example, workerProfiles), you may need to include entries that Embedded Cluster relies on by default, or behavior can change in ways that are hard to predict. Confirm the effective k0s config for your release if you rely on defaults plus overrides.

Override the Helm values for built-in extensions​

Embedded Cluster deploys built-in extensions with Helm. You can patch their values with unsupportedOverrides.builtInExtensions. Each item sets name (the chart to patch) and values (a multi-line YAML string). Embedded Cluster merges your values into the defaults it uses.

Configure the Kubelet​

You can configure the Kubelet to customize worker nodes with Embedded Cluster. One common use case is raising the per-node pod limit (maxPods). Another is tuning parallel image pulls (maxParallelImagePulls).

Add a worker profile in unsupportedOverrides.k0s at config.spec.workerProfiles[]. When you define a worker profile, Embedded Cluster uses it for every node during initial installation and when joining nodes.

Each profile requires:

  • name: Profile name. Do not use default because k0s reserves that name.
  • values: Kubelet settings. See KubeletConfiguration in the Kubernetes documentation.

For more information, see spec.workerProfiles in the k0s documentation.

You can combine worker profiles with other k0s settings in the same k0s string (for example, metadata, telemetry, or network).

Limitations​

  • Embedded Cluster sets the worker profile during initial installation. You cannot change or add worker profiles on upgrade.
  • Embedded Cluster supports only one worker profile and applies it to all nodes. If you add more than one profile in workerProfiles[], Embedded Cluster applies only the first profile.

Example​

apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  unsupportedOverrides:
    k0s: |
      config:
        spec:
          workerProfiles:
          - name: custom-maxpods
            values:
              maxPods: 150
On this page