Deprecation / Removal

• Ubuntu 16 and 18 are no longer tested (#10107, @MrFreezeex)
• Drop support for ansible-core 2.11 and update tests dependencies (#10034, @MrFreezeex)
• Drop Kubernetes 1.24 support (#10234, @MrFreezeex)

Feature / Major Changes

• Make kubernetes v1.27.5 default (#10392, @mzaian)
• Add kubernetes v1.27.4 (#10359, @mzaian)
• Add Kubernetes 1.27.2 (#9976, @mzaian)
• Add hashes for 1.27.3 1.26.6, 1.25.11 (#10220, @mzaian)
• Add hashes for 1.27.4 1.26.7, 1.25.12 (#10300, @mzaian)
• Add CPU Management Policies on the Node (#10309, @yankay)
• Add Debian 12(bookworm) support (#10221, @tu1h)
• Add download.timeout to update download timeout value (#10149, @yjqg6666)
• Add corresponding coredns versions to all the supported kubernetes releases. (#10233, @mzaian)
• Add growpart azure enabled (#10241, @pedro-peter)
• Add ingressClass resource for ingress_nginx by default (#10091, @peschmae)
• Add kubelet topology manager policy on the node (kubelet_topology_manager_scope and kubelet_topoloy_manager_policy) (#10370, @tu1h)
• Add labels to kube-vip static pods (#10139, @liupeng0518)
• Add node_taints to aws_inventory script (#10170, @mstoetzer)
• Add option to set SSL_CERT_FILE for offline installation using custom CA for https proxy (#10215, @HappyFX)
• Add terraform support for NIFCLOUD (#10227, @ystkfujii)
• Add the huawei cloud controller as external cloud controller (#10198, @dabeck)
• Show detected ansible version when it isn't compatible with kubespray (#10109, @jcpunk)
• Allow to override etcd listen-metrics-urls configuration (using etcd_listen_metrics_urls variable) (#10332, @forselli-stratio)
• Don't let find search filesystem mounts in docker build run step (#10131, @tomodachi)
• Permit custom names for API server lb/proxy containers (#10166, @jcpunk)
• Permit skipping helm update (#10169, @jcpunk)
• Split defaults main file into 2 files (checksums and version) (#10121, @electrocucaracha)
• System upgrade for Debian-family nodes is available with system_upgrade=true (#10184, @sathieu)
• Update download_hash.sh script (#10120, @electrocucaracha)
• Use a uniform way to get the local path of the binaries (#10211, @ErikJiang)
• Disable fapolicyd service (#10081, @epif4nio)
• Upgrade the load balancer ( nginx and haproxy ) image version to Nginx 1.25, Haproxy 2.8. (#10409, @yankay)
• [etcd] Default version to 3.5.7 for kubernetes 1.27 (#10410, @mzaian)

Applications

• [argocd] update argocd to v2.7.4 (#10226, @mzaian)
• [argocd] update argocd to v2.8.0 (#10364, @mzaian)
• [argocd] Add argocd_install_url option to allow changing argocd url (#10176, @liupeng0518)
• [helm] upgrade to 3.12.1 (#10225, @mzaian)
• [helm] upgrade to 3.12.3 (#10365, @mzaian)
• [helm] add python dependency check for helm-apps (#10192, @palmeXx)
• [krew] add krew_no_upgrade_check (#10175, @liupeng0518)
• [coredns] Bump coredns version to 1.10.1 (#10199, @eminaktas)
• [coredns] Bump nodelocaldns version to 1.22.20 (#10200, @eminaktas)
• [cert-manager] This introduces a new variable for the cert-manager implementation that will allow one to pass in extra arguments to the cert-manager controller.(#10049, @phunyguy)
• Update Helm (v3.12.2) / Skopeo (v1.13.0) and yq (v4.34.2) (#10295, @tu1h)
• Upgrade many tool versions (Helm, crun, kata, youki, gvisor, skopeo, Calico, Cilium etc...) (#9798, @electrocucaracha)
• [local_path_provisioner] Fix invalid podhelper yaml (#10237, @MrFreezeex)
• Update metrics server to v0.6.4 (#10400, @mzaian)

Container-Managers

• [containerd] Make containerd 1.7.5 default (#10397, @mzaian)
• [containerd] Support containerd v1.7.2 (#10219, @Dentrax)
• [containerd] Support containerd 1.7.3 (#10368, @mzaian)
• [containerd] containerd config_path enable mirrors config using new variable containerd_registries_mirrors (deprecate and remove containerd_insecure_registries for containrd and nerdctl_extra_flags and insecure_registry setting for nerdctl (#10196, @yckaolalala)
• [crio] Add crio_insecure_registries option for specifying insecure_registries of crio (#10142, @qlijin)
• [crio] runroot now needs to be setup in storage.conf instead of crio.conf (#10372, @floryut)
• [crio] Fix etcdctl copy operation (#10242, @ErikJiang)
• [Kata] Set/keep owner/group root/root when unarchiving kata-containers (#10338, @rybnico)
• [youki] Fix youki binary download url (not requiring 'v' in version) (#10337, @ErikJiang)

Network

• [calico] Use configmap to configure calico cni config (#10177, @cyclinder)
• [calico] Update calico v3.25.2 (#10414, @mzaian)
• [calico] Add calico version to v3.26.0 (#10224, @mzaian)
• [calico] Add calico version to v3.26.1 (#10235, @mzaian)
• [calico] Clean up calicoctl_alternate_download_url and calicoctl.mirrors (#10271, @yckaolalala)
• [cilium] Add custom rules to clusterrole for cilium operator (#10267, @jeremythuon)
• [cilium] Upgrade to version 1.13.4 (#10269, @yulng)
• [Cilium] Do not mount tls when 'cilium_hubble_tls_generate' is false (#10357, @charlychiu)
• [Cilium] Update cilium to 1.13.3 (#10158, @jcpunk)
• [flannel] Only create /var/lib/calico when needed (#10156, @jcpunk)
• [flannel] Bump flannel version to v0.22.0 and flannel-cni-plugin version to v1.1.2. Also, changes flannel repository from flannelcni to flannel (#10205, @eminaktas)
• [flannel] Remove unused flannel_cni_download_url (#10188, @oomichi)
• [kube-ovn]: update version v1.11.5 (#10125, @yankay)
• [multus] Fix loop_control template error when item is None (#10347, @nicolas-goudry)

API Change

• Unless the pod security standard versions are changed on intentionally, as default it will be the same major version with Kubernetes version. (#10210, @ugur99)
• Upgrade ansible to 7.0 and ansible-core to 2.14.x (#10190, @MrFreezeex) ⚠️ (See Notes 2)

Documentation

• Add github container registry (github_image_repo) to docs/offline-environment.md (#10265, @blackliner)
• Update doc for ansible-core 2.14 support and clarify issues running older python versions (#10261, @MrFreezeex)
• Update links for aws_alb_ingress_controller (#10264, @kundan2707)
• Update links in ingress-controller and kuberentes-apps (#10239, @vaibhav2107)
• Update Calico to lowercase and fix broken calico link in README (#10232, @Xieql)
• Document containerd command to restart nginx-proxy container when adding control plane node (#10406, @nicolas-goudry)

Failing Test

• Increase metallb wait timeout from 30sec to 2min (#10260, @MrFreezeex)
• Update CentOS 7 image and test fedora 37 and 38 instead of fedora 35 and 36 (#10108, @MrFreezeex)

Bug or Regression

• Fix Dockerfile for newest directory layout (#10128, @dabeck)
• Fix Flatcar bootstrap issues (yaml module missing and ntp issue) (#10363, @tenni-paws)
• Fix argocd install not working using the kubespray docker image (#10371, @cortex3)
• Fix correctly mount ssl ca directories (#9794, @maxime1907)
• Fix etcdctl copy operation (#10230, @ErikJiang)
• Fix gce-pd-csi driver (#10208, @ashishsinghdev)
• Fix grep command without -w option causing prefix matched while adding one etcd member (#10291, @yangsenzk)
• Fix hcloud-cloud-controller-manager not working in certain setups (#10297, @cortex3)
• Fix helm (kubelet-csr-approver) installation on redhat distro (#10204, @MrFreezeex)
• Fix kubelet-csr-approver usage with upgrade-cluster.yml and missing package with helm role (#10165, @j4m3s-s)
• Fix nginxingress-class template (missing newline) (#10174, @richard-fairthorne)
• Fix problem migration problem with k8s 1.27 (#10136, @batazor)
• Fix reset_confirmation not working when inputing correct value (#10288, @somewho)
• Fix wrong path in manage-offline-files script (#9886, @Medosopher)
• Fix an issue where using Rocky Linux 8 as OS for Vagrant for testing purposes causing etcd to fail on start. (#10252, @nltimv)
• Fix ansible-lint galaxy rule (#10277, @MrFreezeex)
• Fix ansible-lint key-order error (#10314, @MrFreezeex)
• Fix outdated tag and experimental ansible-lint rules (#10254, @MrFreezeex)
• Fix dockerfile build error (#10127, @yankay)
• Fix metrics-server deployment to run with kubernetes 1.26+ (#10183, @mzaian)
• Fix undefined reset_confirmation_prompt variable in reset play (#10303, @Mishavint)
• Fix CIS Kubernetes V1.23 Benchmark item number 4.1.9 to enhance security (Change kubelet-config.yaml and kubelet.env file permissions from 640 to 600) (#10304, @satandyh)
• Fix parsing of RHSM proxy configuration (#10228, @tmurakam)
• Fix var-spacing ansible rule (#10266, @MrFreezeex)
• Fix specify owner to kube_owner in task of copy cni plugins (#10407, @NierYYDS)
• Fix typo kubelet_topoloy_manager_policy => kubelet_topology_manager_policy (#10384, @hangscer8)
• Fix recover_control_plane playbook (also add debian 12 with cilium as a new nightly test) (#10411, @floryut)
• Fix nameserver inline comments in /etc/resolv.conf (#10415, @yankay)
• Added systemd_resolved_disable_stub_listener variable to disable systemd-resolved's stub listener, defaults to true on Flatcar. (#9875, @cosandr)
• Remove auto_attach and syspurpose in RHEL subscription Organization ID/Activation Key registration. (#10258, @yckaolalala)
• Replace "crio_packages" with "crio_bin_files" (#10182, @yckaolalala)
• Update MetalLB deployment, wait for resource. (#9995, @Jeroen0494)
• Upgrade ansible to 7.0 and ansible-core to 2.14.x in Dockerfile (#10259, @yckaolalala)
• Fix typo kubelet_topoloy_manager_policy => kubelet_topology_manager_policy (#10384, @hangscer8) ⚠️ (See Notes 1)
• Change maximal_ansible_version to 2.15(exclusive) (#10395, @yankay)
• Install etcdutl file by default (#10385, @liupeng0518)

Other (Cleanup or Flake)

• [CI] Add CI VM for debian12 (#10222, @yankay)
• [CI] Removes Ansible reinstall from build pipeline (#10032, @luksi1)
• [CI] cleanup stale packet namespace automatically (#10245, @MrFreezeex)
• [CI] fix tf-elastx_cleanup fail (#10133, @yankay)
• [CI] Sanitize branch name in testing before using it in kubernetes label for packet-ci (#10315, @MrFreezeex)
• Add an exception for youki in download_hash script (#10346, @ErikJiang)
• Drop support for Kubernetes 1.24.x (move min version to 1.25.x) (#10126, @yankay)
• Ensure host entries from /etc/host are absent when populate_inventory_to_hosts_file is false (#10144, @rptaylor)
• Exclude terraform.tfstate backups in .gitignore (#10216, @rptaylor)
• Ping is no longer reported as a changed task (#10160, @jcpunk)
• Reading mounted volumes no longer considered a changed task (#10161, @jcpunk)
• Resolve ansible-lint name errors (#10253, @MrFreezeex)
• Update KUBESPRAY_VERSION for v2.22.1 (#10201, @yankay)

Supported Components

• Core
  • kubernetes v1.27.5
  • etcd v3.5.7
  • docker v20.10 (see note)
  • containerd v1.7.5
  • cri-o v1.27 (experimental: see CRI-O Note. Only on fedora, ubuntu and centos based OS)
• Network Plugin
  • cni-plugins v1.2.0
  • calico v3.25.2
  • cilium v1.13.4
  • flannel v0.22.0
  • kube-ovn v1.11.5
  • kube-router v1.5.1
  • multus v3.8
  • weave v2.8.1
  • kube-vip v0.5.12
• Application
  • cert-manager v1.11.1
  • coredns v1.10.1
  • ingress-nginx v1.8.1
  • krew v0.4.4
  • argocd v2.8.0
  • helm v3.12.3
  • metallb v0.13.9
  • registry v2.8.1
• Storage Plugin
  • cephfs-provisioner v2.1.0-k8s1.11
  • rbd-provisioner v2.1.1-k8s1.11
  • aws-ebs-csi-plugin v0.5.0
  • azure-csi-plugin v1.10.0
  • cinder-csi-plugin v1.22.0
  • gcp-pd-csi-plugin v1.9.2
  • local-path-provisioner v0.0.24
  • local-volume-provisioner v2.5.0

Known issues

N/A

Notes

1. Variable kubelet_topoloy_manager_policy change to kubelet_topology_manager_policy, please update your inventory
2. Upgrade ansible to 7.0 and ansible-core to 2.14.x
3. ⚠️ breaking change : containerd config_path enable mirrors config using new variable containerd_registries_mirrors (#10196, @yckaolalala)