Deprecation / Removal

• [Cilium] Delete the probe option of cilium_kube_proxy_replacement (#9929, @XiuguangHuang)
• [Cilium] Remove use_localhost_as_kubeapi_loadbalancer and detect wether we can use localhost apiserver loadbalancer if cilium/calico replace kube-proxy (#9718, @MrFreezeex)
• Drop crun_bin_dir unused variable, now using only bin_dir var (#9845, @electrocucaracha)
• Drop the canal network_plugin support because the network_plugin is unmaintained. (#10100, @oomichi)
• Remove the support of Debian 9 (#10097, @yankay)
• Replaces storage.googleapis.com/kubernetes-release with dl.k8s.io (#10066, @KlwntSingh)

Feature / Major Changes

• Add Kubernetes 1.26.x (#9570, @mzaian ; #9732, @yankay; #9829, @mzaian; #9900, @mzaian)
• Make kubernetes v1.26.5 default (#9983, @mzaian)
• "native" snapshotter of nerdctl config is replaced by new var nerdctl_snapshotter with default "overlayfs" value (#9979, @dmitrytretyakov)
• Support multi-arch using the same image name (#9978, @ErikJiang)
• Add DNS configuration for cert-manager (using new variables cert_manager_dns_policy|config) (#9673, @ErikJiang)
• Add Retry for restart kube-controller-manager (#10013, @hangscer8)
• Add coredns_additional_configuration variable to define extra Coredns configurations (#10025, @navidnabavi)
• Add coredns_rewrite_block to perform internal message rewriting (#10045, @maxime1907)
• Add a new simple network_plugins custom_cni to install user provided manifests (#9819, @MrFreezeex)
• Add back openssh-client to docker image (#9835, @maxime1907)
• Add download retries option download_retries (#9911, @tu1h)
• Add support to install ContainerD on any Linux Distributions using new var allow_unsupported_distribution_setup (#9827, @XDRAGON2002)
• Add the kube-profile config to the kubeadm's kube-scheduler config. (#9993, @yankay)
• Add vim to kubespray docker image (#9805, @XDRAGON2002)
• Adds support for Kubelet-CSR-approver to auto-approve kubelet CSR when kubelet_rotate_server_certificates. (#9877, @j4m3s-s)
• Add dns_cpu_limit value to support large scaled coredns deployments (#10103, @mzaian)
• Add provider meta module_name in Equinix Metal TF configs (#10044, @vasubabu)
• Allow to configure image garbage collection (using kubelet_image_gc_high_threshold and kubelet_image_gc_low_threshold) (#9832, @zhan9san)
• Apply kubeadm patches during upgrade as recommended by k8s (#9781, @mvandergiesen)
• Cinder-csi: Allow VolumeSnapshotClass' deletionPolicy to be configurable (#9736, @huangkevin404)
• Containerd add containerd_use_config_path config field. (#9770, @lengrongfu)
• Enable control plane load balancing for kube-vip (#9785, @ErikJiang)
• Feat(contrib/terraform): support custom ssh port (#9836, @maxime1907)
• Fix kube-bench 1.2.20 to enhance security (Ensure that the --audit-log-maxbackup argument is set to 10) (#9939, @yankay)
• Fix kube-bench 1.1.19 to enhance security (Change Kubernetes Cert directory and file ownership is set to root:root) (#9937, @yankay)
• Fix kube-bench 4.1.1 to enhance security (Change kubelet systemd init file from 644 to 600) (#9934, @yankay)
• Fix kubernetes-app/argocd: download related things with the download role (#9786, @pli01)
• Kube.py now supports kubeconfig (#9982, @liupeng0518)
• MetricsServer: Add extras nodeselector, affinity, tolerations (using metrics_server_nodeselector, metrics_server_extra_affinity ,metrics_server_extra_tolerations) (#9972, @pli01)
• Refactor Hetzner terraform (fixing flatcar configs and remove deprecated provider) (#10002, @ThisIsQasim)
• Support for MetalLB v0.13.9 with CRD (#9120, @Jeroen0494)
• Throw an error when specifying unsupported os in Vagrant (#9965, @THUzxj)
• Update CoreDNS manifests (remove deprecated annotations) (#9977, @mzaian)
• Update dns-autoscaler configuration and remove deprecated annotations (#9996, @mzaian)
• Update metrics server to v0.6.3 (#10026, @mzaian)
• Upgrade argocd to v2.6.3 (#9848, @panguicai008)
• Upgrades the following Python libraries to their latest available releases (cryptography / jinja2 / jmespath / MarkupSafe/ netaddr / pbr / ruamel.yaml / ruamel.yaml.clib) (#9938, @luksi1)
• Add IPv6 listen directive to haproxy if enable_dual_stack_networks (#9674, @yankay)
• Add support for Ansible collections in Kubespray (⚠️ See notes !) (#9582, @luksi1)
• Support mTLS for Hubble and upgrade backend to v0.11.0 (#9959, @jeremythuon)
• Update nodelocaldns to 1.22.18 (#9800, @sathieu)
• Replace disable_swap variable with kubelet_fail_swap_on (#10036, @Manuelraa)
• Replace nodelocaldns label to k8s-app: node-local-dns (#9745, @stelucz)
• Upgrade rancher local-path-provisioner to v0.0.23 (#9855, @panguicai008)
• Use kube_apiserver_address variable for advertiseAddress (#9967, @liupeng0518)
• Use string for ipv6 forward conf value (#9992, @liupeng0518)
• Update pause image version to v3.9 (#10112, @mzaian)
• Upgrade cni version to v1.3.0 (#10058, @cyclinder)
• [argocd] update argocd to v2.6.7 (#9953, @mzaian)
• [helm] support to 3.11.1 (#9849, @mzaian)
• [helm] support to 3.11.3 (#10022, @mzaian)
• [helm] support to 3.11.2 (#9951, @mzaian)
• [helm] upgrade to 3.12.0 (#10085, @mzaian)
• [UpCloud] Add server group support for vms and target port for loadbalancers (#9831, @robinAwallace)
• [argocd] update argocd to v2.5.10 (#9753, @yanggangtony)
• [cert-manager] Upgrade to v1.11.1 (#9964, @rtsp)
• [flannel] update to v0.21.4 (#10027, @mzaian)
• [nerdctl] support version 1.3.1 (#10024, @mzaian)
• [nerdctl] update to version 1.4.0 (#10119, @mzaian)

Applications

• [kube-vip] Support to v0.5.8 (#9734, @hangscer8)
• [kube-vip] Support kube-vip to v0.5.11 (#9852, @panguicai008)
• [kube-vip] Update default kube-vip to v0.5.12 (#10005, @hangscer8)
• [vSphere-csi] Add resources section to all containers releated to Vsphere CSI driver (#9687, @JRaver)
• [argocd] update argocd to v2.7.2 (#10086, @mzaian)

Container-Managers

• [containerd] Add hashes for containerd version 1.6.19 (#9838, @mzaian)
• [containerd] Add hashes for containerd version 1.6.20 (#9954, @mzaian)
• [containerd] Add hashes for containerd version 1.7.0 (#9892, @mzaian)
• [containerd] Add hashes for containerd versions 1.7.1, 1.6.21 (#10061, @mzaian)
• [containerd] Support version 1.6.16 (#9727, @yanggangtony)
• [cri-o] Bump versions to 1.26.3, 1.25.3, 1.24.5 (#9999, @dkasanic)
• [cri-o] Fix install order -> first runc then crictl (#9780, @mvandergiesen)
• [cri-o] Fix missed double quotes in cri-o config (#10040, @turbosnail)
• [cri-o] Fix CRI-O amd64 v1.26.0 wrong archive checksum (#9872, @panguicai008)
• [cri-o] cri-o restart if config change (#10057, @MrFreezeex)
• [cri-o] Remove deprecated crio_pids_limit (default is now unlimited) (#10056, @j4m3s-s)
• [cri-o] Fix cri-o restart if config change (#10057, @MrFreezeex)
• [runc] Upgrade to v1.1.7 (#10039, @pomland-94)

Network

• [Calico] Add Retry and Ignore Error for Checking calico ready (#9883, @hangscer8)
• [Calico] Add option calico_kubeconfig_wait_timeout (#9994, @tu1h)
• [Calico] Improve version check command (#9861, @zhan9san)
• [Calico] Optimize the detection of calico existence (#9873, @hangscer8)
• [Calico] Support calico version v3.25.0 (#9860, @cyclinder)
• [Calico] upgrade default calico version to v3.25.1 (#9950, @mzaian)
• [Calico] Add missing ipamconfigs resource in RBAC (#9755, @chaunceyjiang)
• [Calico] Fix installation while applying CRD (#10068, @hangscer8)
• [Calico] Add calico version to v3.24.6 (#10113, @mzaian)
• [Cilium] Add and support v1.13.0 (#9879, @utam0k)
• [Cilium] Fix Hubble relay configuration (#9876, @prashantchitta)
• [Cilium] Fix the configuration of TLS for hubble (#9880, @utam0k)
• [Cilium] Remove duplicates in the configuration of tls for hubble (#9932, @CaMoPeZzz)
• [Cilium] Support version above 1.13.x (#9914, @wbh1)
• [Cilium] Updates hubble certgen arguments (wrong since v0.1.7) (#9856, @XDRAGON2002)
• [Cilium] IPAM uses "Cluster Scope" mode by default. Also add the parameters required for this mode (#9443, @dcwbq)
• [flannel] Update image repo from flannelcni to flannel (#10041, @ErikJiang)
• [multus] fix multus include error (#10105, @darkobas2)

API Change

• Openstack cloud controller manager bind address is now configurable using external_openstack_cloud_controller_bind_address (#9958, @dominykasn)

Documentation

• Add a mention for custom_cni in CNI list (#9878, @j4m3s-s)
• ArgoCD no longer uses the pod name as initial password (#9930, @peschmae)
• Drop remaining part for supporting ansible 2.9 and 2.10 (#9842, @oomichi)
• Fix sidebar documentation (#9988, @lijin-union)
• Fixup link in docs/calico.md (#9940, @kundan2707)
• Remove stale contents for cni documention (#9778, @tu1h)
• Reword confusing etcd download url comment when etcd_deployment=host (#9686, @tjanson)
• Suggest to run reset.yml playbook for first-time users (#9865, @kerryeon)
• Update docker tag to v2.21.0 in README.md (#9802, @Payback159)
• Update link for baremetel consideration (#9944, @kundan2707)
• Add port requirements documentation (#9969, @yankay)

Failing Test

• Update Terraform to 1.3.7 and Vagrant to 2.3.4 (#9699, @floryut)
• [CI] Migrate CI_BUILD_ID to CI_JOB_ID and CI_BUILD_REF to CI_COMMIT_SHA following gitlab upgrade (#10063, @floryut)

Bug or Regression

• Add PSS labels to metallb namespace (#9713, @manzsolutions-lpr)
• Add jmespath back to Dockerfile image (#9697, @floryut)
• Add missing krew_download_url to offline.yml (#9788, @jianse)
• Add proxy_env variable to apt_key cleanup task (#9766, @SamuelBECK1)
• Add rsync in Dockerfile (#9839, @zhan9san)
• Add ruamel.yaml back to Dockerfile image (#9707, @floryut)
• Cleanup MetalLB install following update (#10004, @eugene-marchanka)
• Copy contrib/ to Dockerfile (#9774, @oomichi)
• Downgrade the version of CoreDNS to 1.8.6 for compatibility with Kubernetes versions older than 1.25. (#9846, @JiffsMaverick)
• Explicitly disable rhsm repo when rhel_enable_repos is false (#9973, @tu1h)
• Fix cert_manager_trusted_internal_ca manifest failing when dns policy is set (#9922, @peschmae)
• Fix containerd_insecure_registries => move with_item to with_dict (#9729, @lengrongfu)
• Fix allow unsupported distribution (#9904, @ErikJiang)
• Fix cilium's hubble ui configuration (#9735, @j4m3s-s)
• Fix comma-separated-list splitting of kubelet_enforce_node_allocatable variable (#9694, @Tristan971)
• Fix confusing instance sizing (etcd, kube_master) in Vagrantfile (#9966, @THUzxj)
• Fix ingress url not found issue (#9789, @JaneLiuL)
• Fix playbook names to support import via galaxy (#10021, @dkasanic)
• Fix restart k8s components, checking yml files instead of manifest (#9962, @liupeng0518)
• Fix uniontech OS installation failure (#9862, @ErikJiang)
• Fixing default cgroups for kubelet and container_manager (#9834, @MrFreezeex)
• Localhost task (validate mirror) don't need to ask for become (#9669, @chok)
• Remove unneeded access_ip when not wanted in terraform scripts (#9869, @maxime1907)
• Replace semicolons by commas in networkmanager dns configuration options (#9840, @lystor)
• Retry other masters during upgrade and not only the first one (#9768, @maxime1907)
• Skip steps of ensuring NTP and tzdata packages in the CoreOS and Flatcar (#9742, @ErthoAers)
• Support extended settings for the Debian os family (#9943, @ErikJiang)
• Fix calico rbac issue (#9806, @JaneLiuL)
• Update nodes in etc hosts after cluster scale (#9837, @zhan9san)
• Update rhsm repo trigger if no subscriptions is found (#10001, @tu1h)
• Bootstrap ansible requirement in the facts playbook (#10069, @MrFreezeex)
• Clear http scheme on containerd insecure-registry tls config (#10084, @tu1h)
• Ignore errors in check mode performing "Disable swapOnZram for Fedora" (#10077, @gorozhin)
• [etcd] fix make-ssl-etcd.sh.j2; move pem files only if any new certs exist (#9974, @2k0ri)
• [vSphere-csi-driver] Fixes the run of the cluster.yml playbook when vsphere_csi_namespace is set to non-default (#9946, @eugene-marchanka)

Other (Cleanup or Flake)

• Add checksum verification for kubectl binary in dockerfile (#9963, @alekseyolg)
• Add generic pre-commit hook to the repository (#9750, @bbaassssiiee)
• Cleanup of external-openstack-cloud-config to be in the same order/values as the documentation and not clutter config when defaults are used. (#9899, @jadams)
• Cleanup v1.23.x references/conditions/hashes (#9698, @floryut)
• Dockerfile update ubuntu version to 22.04 which has newer system packages with fewer (#10033, @alekseyolg)
• Drop support for Kubernetes 1.23.x (move min version to 1.24.x) (#9691, @floryut)
• Fix(contrib/terraform): do not set ansible_ssh_port to 22 (#9828, @maxime1907)
• Move multus url to k8snetworkplumbingwg repository (#9850, @panguicai008)
• New automated method to collect binaries checksums (#9782, @electrocucaracha)
• Reducing the number of layers and commands for docker image (#9822, @alekseyolg)
• Remove deprecated udpIdleTimeout field in KubeProxyConfiguration (#9925, @HirazawaUi)
• Remove invalid character in crictl tasks file (#9970, @tu1h)
• Replace bash for loop when checking API server SANs (#9060, @rptaylor)
• Use var etcd_deployment_type instead of etcd_kubeadm_enabled (#9823, @liupeng0518)
• Reducing the number of layers, increasing readability, reducing the size of the image (#9821, @alekseyolg)
• Fix arithmetic outside of jinja (#10106, @MrFreezeex)
• Fix CI broken by flannel-cni-plugin docker hub rate limit (#10083, @yankay)
• [CI] Add CI for containerd insecure_registries (#9797, @yankay)
• [CI] Updated version of ara included in CI job logs collection from 1.5.7 to 1.6.1 (#9737, @dmsimard)
• [CI] Add checksum verification of kubectl binary in pipeline image (#9971, @alekseyolg)
• [CI] Fix CentOS Extras repo url for Oracle Linux 7 aarch64 (#9791, @bin456789)
• [CI] Use Docker buildkit + caching for builds to speed up the CI pipeline (#10008, @luksi1)
• [CI] Add six module into openstack-cleanup/requirements.txt (#10099, @oomichi)
• [CI] Fix tests for files lookup path for custom-cni (#10088, @j4m3s-s)

Supported Components

• Core
  • kubernetes v1.26.5
  • etcd v3.5.6
  • docker v20.10 (see note)
  • containerd v1.7.1
  • cri-o v1.24 (experimental: see CRI-O Note. Only on fedora, ubuntu and centos based OS)
• Network Plugin
  • cni-plugins v1.2.0
  • calico v3.25.1
  • cilium v1.13.0
  • flannel v0.21.4
  • kube-ovn v1.10.7
  • kube-router v1.5.1
  • multus v3.8
  • weave v2.8.1
  • kube-vip v0.5.12
• Application
  • cert-manager v1.11.1
  • coredns v1.9.3
  • ingress-nginx v1.7.1
  • krew v0.4.3
  • argocd v2.7.2
  • helm v3.12.0
  • metallb v0.13.9
  • registry v2.8.1
• Storage Plugin
  • cephfs-provisioner v2.1.0-k8s1.11
  • rbd-provisioner v2.1.1-k8s1.11
  • aws-ebs-csi-plugin v0.5.0
  • azure-csi-plugin v1.10.0
  • cinder-csi-plugin v1.22.0
  • gcp-pd-csi-plugin v1.4.0
  • local-path-provisioner v0.0.23
  • local-volume-provisioner v2.5.0

Known issues

N/A

Notes

• Support for MetalLB v0.13.9 with CRD (⚠️ This release includes user facing changes for which there is action required. The way the inventory is setup for MetalLB deployment has changed significantly. Most prominently, we have switched from underscores to a dictionary for defining resources. Please follow the documentation for restructuring your MetalLB inventory variables.
• Replace disable_swap variable with kubelet_fail_swap_on
• Fix playbook names to support import via galaxy (⚠️ ADD NOTE : recover-control-panel => recover_control_plane, remove-node => remove_node, upgrade-cluster => upgrade_cluster)
• [Cilium] IPAM uses "Cluster Scope" mode by default.
• Add support for Ansible collections in Kubespray (This would cause a change to the repository's structure, meaning downstream users would either need to change their code to point to the playbooks directory or use the ansible.builtin.import_playbook module)