Deprecation / Removal

• Drop calico v3.21 support (#9515, @oomichi)

Feature / Major Changes

• Add Check resolv.conf is empty to avoid CoreDNS crash (#9502, @yankay)
• Add XDG related Helm paths to be removed from reset tasks (#9561, @emiran-orange)
• Add a parameter (disable_host_nameservers) to disable host nameservers (#9357, @eminaktas)
• Add an option (populate_loadbalancer_apiserver_to_hosts_file) to skip adding load balancer name in the hosts file (#9331, @JRaver)
• Add custom options to coredns kubernets plugin (coredns_kubernetes_extra_opts ) (#9608, @mvandergiesen)
• Add docker support for openEuler linux (#9498, @ErikJiang)
• Add support for the OpenEuler Linux (#9494, @ErikJiang)
• Add terraform script for Flatcar Linux on Hetzner (#9618, @florianow)
• Add the ability to define options for DNS upstream servers (using new variable dns_upstream_forward_extra_opts) (#9311, @emiran-orange)
• Add var (ingress_nginx_probe_initial_delay_seconds) for control initialDelaySeconds in ingress-nginx probes (#9405, @zvlb)
• Add variable condition snapshot in vSphere CSI (vsphere_csi_block_volume_snapshot) (#9429, @yanggangtony)
• Add variable in metrics_server deployment (metrics_server_replicas) to enable HA mode (#9539, @ugur99)
• Change dns upstream condition for nodelocaldns when using host_resolvconf (#9378, @unai-ttxu)
• Download coredns image to all hosts in k8s_cluster (#9316, @joes)
• Enable check mode in DNS Cleanup tasks (#9472, @emiran-orange)
• Etcd image has the same tag accross multiple archs (#9516, @hangscer8)
• Fix a pre-upgrade node drain rescue task failure when kube_override_hostname is set (#9556, @chadswen)
• Fix default value for kubelet_secure_addresses (#9355, @willtrnr)
• Provides <kubeadm_init_timeout> to change the timeout of first control-plane initialization (#9617, @tu1h)
• Remove PodSecurityPolicies in MetalLB for kubernetes 1.25 (#9442, @yanggangtony)
• Support Python 3.11 - ruamel.yaml.clib need to be updated to 0.2.7 (#9426, @olivierlemasle)
• Support customize the additional sysctl variables using additional_sysctl (#9351, @yankay)
• Support patches field in kubeadm v1beta3 in both InitConfiguration and JoinConfiguration (using new variable kubeadm_patches) (#9326, @titaneric)
• Switch helm install (from synchronize to copy) to support password authentication (#9343, @ghostloda)
• Update api version for pdb and batch (deprecated in 1.25) (#9369, @yankay)
• Update dashboard image repo to remove arch flag (#9530, @tu1h)
• Update etcd log-level parameter name (new name: ETCD_LOG_LEVEL) (#9540, @ErikJiang)
• Update local-volume-provisioner to 2.5.0 + add documentation (#9463, @olivierlemasle)
• Update the number of nofile limits in containerd to 65535 (#9507, @ErikJiang)
• Upgrade metrics server to v0.6.2 (#9554, @mzaian)
• Upgrade the load balancer ( nginx and haproxy ) image version. (#9506, @yankay)
• Use kube_apiserver_port variable instead of hard-coding 6443 (#9620, @huangkevin404)
• [etcd] Default version to 3.5.5 for k8s 1.25.x (#9419, @mzaian)
• Update CoreDNS version to v1.9.3 (#9503, @yankay)
• Add the possibility to specify extra domains for the coredns kubernets plugin (using coredns_kubernetes_extra_domains) (#9635, @mvandergiesen)
• Streamline ansible_default_ipv4 gathering loop (#9281, @rptaylor)
• Update kubernetes dashboard to 2.7.0 (k8s 1.25 support) (#9425, @mzaian)
• Skip retry operation with containerd when etcd installed on host VM (#9560, @JRaver)
• Update pause image version to v3.8 (#9668, @mzaian)
• Enable kubelet_authorization_mode_webhook back by default and remove extra role (#9662, @MrFreezeex)
• Terraform gcp can now have extra ingress firewall rules, using new variable extra_ingress_firewalls (#9658, @sathieu)
• kubeadm/etcd: use config to download certificate (#9609, @MrFreezeex)

Applications

• [argocd] update argocd to v2.5.5 (#9604, @mzaian)
• Upcloud: Reclaim policy for PV is now delete (#9574, @robinAwallace)
• [Exoscale] Add missing zone input variable (#9495, @ayoubeddafali)
• [MetalLB] Avoid MetalLB speaker image download when MetalLB speaker is disabled (#9248, @unai-ttxu)
• [Openstack] Replace deprecated "template" Terraform provider with supported "cloudinit" Terraform provider (#9536, @inflatador)
• [OpenStack] Updated openstack cloud controller to version v1.25.3 (#9500, @robinAwallace)
• [Openstack] Add bastion_allowed_ports to allow custom security group rules on bastion node (#9336, @bl0m1)
• [Openstack] Upgrade 1.22.0 to 1.23.4 (#9332, @QcFe) (See Notes 1)
• [Openstack] Added override variable, additional server groups and cloudinit config (#9452, @Xartos)
• [cinder-csi-nodeplugin] Remove the pods-cloud-data volume (delete upstream) (#9362, @huangkevin404)
• [vsphere-csi] Add missing defaults for external_vsphere_* variables in the csi_driver/vsphere role (#9664, @rlacko58)
• [hetzner] In config, rename ansible groups to use _ instead of - (#9569, @ym)
• [kube-vip] Minor changes on Kube VIP configuration parameters (and fix wrong properties) (#9414, @woutergd)
• [cert-manager] Upgrade to v1.10.1 (#9512, @rtsp) then v1.11.0 (#9661, @mzaian)
• [helm] upgrade to 3.10.3 (#9605, @mzaian)
• [ingress-nginx] upgrade to 1.5.1 (#9532, @mzaian)
• [vSphere] Removing unneeded terraform dependencie & mark vsphere_password as sensitive (#9672, @sathieu)

Container-Managers

• Optimize cgroups settings for node reserved (using new kube_reserved, see docs for more information) (#9209, @shelmingsong)
• [Docker] Update docker package to 20.10.20 (partial fix for CVE-2022-39253) (#9410, @floryut)
• [containerd] Add support for 1.6.11 (#9544, @yanggangtony)
• [containerd] Added variables for unpriviledged ports and icmp (#9517, @Xartos)
• [containerd] Allow containerd-common to execute multiple times per play (#9543, @chadswen)
• [containerd] Newly started containers will be limited to 16384 open files. To change this number, set containerd_base_runtime_spec_rlimit_nofile, or remove base_runtime_spec from runc runtime to revert to previous behaviour. (#9319, @fungusakafungus)
• [containerd] Support v1.6.13 and v1.6.14 (#9585, @yanggangtony)
• [containerd] Add config_path var in config.toml.j2 file (#9566, @lengrongfu)
• [containerd] Add hashes for containerd versions 1.5.14 , 1.5.15 , 1.5.16 (#9678, @yanggangtony)
• [cri-o] Use cri-o from upstream instead of kubic/OBS (#9374, @cristicalin)
• [nerdctl] upgrade to version 1.0.0 (#9424, @mzaian)

Network

• Bump cni-plugins version to v1.2.0 (#9671, @cyclinder)
• Fix remove Cilium CNI failed because the CNI bin dependency (#9563, @yankay)
• [Calico] Add cni bin when installing (#9367, @ErikJiang)
• [Calico] Add retry for start calico kube controller (#9450, @cleverhu)
• [Calico] Adjust calico-kube-controller pod to non hostNetwork pod (#9465, @cyclinder)
• [Calico] Adjust calico-kube-controller pod to use hostnetwork if using etcd (#9573, @JSpon)
• [Calico] Disable 'Check that IP range is enough for the nodes' (#9491, @mzaian)
• [Calico] Update the tag image to support multiple architectures with the same tag (#9529, @ErikJiang)
• [Calico] remove deprecated PodSecurityPolicy (removed in Kubernetes in v1.25) (#9395, @yankay)
• |Calico] Allow user to set env: FELIX_MTUIFACEPATTERN in calico-node.yml (using calico_felix_mtu_iface_pattern) (#9330, @shelmingsong)
• [Calico] Replace node-role.kubernetes.io/master with control-plane (#9627, @my-git9)
• [Calico] upgrade default calico version to v3.24.5 (#9580, @yankay)
• [Calico] Add vxlan-v6.calico to the list of NetworkManager unmanaged interfaces (#9631, @cyclinder)
• [Calico] Add retry to avoid 'unknown' state for calicoctl (#9633, @tu1h)
• [Calico] Update Calico VXLAN offload docs because Calico changed the default value (#9639, @yankay)
• [Calico] Add possibility to enable calico floatingIPs feature (using calico_felix_floating_ips) (#9680, @MatthieuFin)
• [Cilium] Add download configuration for cilium hubble images (using cilium_enable_hubble variable) (#9376, @ErikJiang)
• [Cilium] Add switch cilium_enable_bandwidth_manager (#9441, @dcwbq)
• [Cilium] Cleanup cilium-init image from cilium template (#9508, @ErikJiang)
• [Cilium] update cilium cli offline download url example (#9458, @cleverhu)
• [Cilium] Install Cilium CLI alongside Cilium (#9436, @dcwbq)
• [flannel] Initcontainer image now correctly support architecture suffix (#9461, @rollandf)
• [flannel] Upgrade version to v0.20.1 (#9528, @ErikJiang)
• [flannel] remove deprecated PodSecurityPolicy (removed in Kubernetes in v1.25) (#9365, @yankay)
• [flannel] Add wireguard encryption backend as option (#9583, @janaurka)
• [flannel] Support dual stack IPv4 & IPv6 networking (#9564, @styshoo)
• [flannel] Allow setting the DirectRouting option on VXLAN (#9438, @willtrnr)
• [flannel] update to v0.20.2 & make it default (#9675, @mzaian)
• [kube-ovn] Update version to v1.10.7 (#9527, @liupeng0518)
• [kube-ovn] Remove kube-ovn log directories when reseting (#9625, @JochenFriedrich)
• [kube-ovn] Remove ovn.kubernetes.io/ovs_dp_type from nodeSelector (#9594, @JochenFriedrich)
• [kube-ovn] Support OVN Interconnect (#9599, @JochenFriedrich)
• [multus] added support for mixed type of container engine (#9224, @mr-yaky)

Bug or Regression

• Change include to import_playbook in recover_control_plane playbook, to support ansible 2.12+ (#9576, @floryut)
• Corrected vsphere directory in docs (#9534, @wojciehm)
• Deleting worker nodes is now skipped if there is no kube_control_plane node. (#9430, @kerryeon)
• Etcd arch can now support arm64 and amd64 (#9421, @yanggangtony)
• Fix cert-manager deployment on hardening environments (#9404, @oomichi)
• Fix checksum of ciliumcli v0.12.5 for arm64 (#9614, @oomichi)
• Fix inconsistent handling of admission plugin list (kube_apiserver_enable_admission_plugins must be specified as a list of individual plugin names instead of a single item comma-separated list) (#9407, @willtrnr)
• Fix kube token dir permissions (#9590, @C-Romeo)
• Fix missing control plane taint in kubeadm (#9592, @yankay)
• Fix regex for comments nameserver in resolv.conf (#9523, @yankay)
• Fix reset for RedHat based distro with major version >=8 (#9537, @dougsland)
• Fix wrong cri_socket path for containerd (#9401, @maxime1907)
• Fix wrong rbac of the ClusterRole csi-snapshotter-role (#9610, @maxime1907)
• Remove coredns_server from supersede_nameserver in dhclient.conf if nodelocaldns is enabled. (#9392, @JiffsMaverick)
• Remove immutable flag from /var/lib/kubelet subdirs (#9597, @emiran-orange)
• Skip the install of ping package in Fedora CoreOS & Flatcar (#9370, @yankay)
• Fix OL9 setup - disable Centos Extras repo creation (#9483, @psvmcc)
• Use hostname override in post-remove role, just as pre-remove role does (#9360, @JSpon)
• [Calico] Install calico-kube-controller also when using kdd datastore (#9358, @wayfrro)
• [Cilium] Fix the Hubble certificate being faulty because the cluster name has an hard coded value (#9340, @dcwbq)
• [Cilium] Fix tls settings not being properly set (#9457, @charlychiu)
• [Cilium] Remove trailing backslash and fix yaml indent (#9339, @reneluria)
• [Openstack] Fix a race condition in terraform causing ports to not get an IP (#9345, @bl0m1)
• [Openstack] Fix missing permissions for Openstack cloud-controller-manager (#9335, @bl0m1)
• [gVisor] Allow installation on arm architecture systems (#9493, @ErikJiang)
• [kube-ovn] Cluster support for ovn-central (#9596, @JochenFriedrich)
• [upcloud] Fixed issue where DNS would be blocked while using allowlist (#9510, @Xartos)

Other (Cleanup or Flake)

• Use the correct api version and resource type in secrets_encryption.yaml.j2 (#9575, @LukasNajman)
• Minor cleanup of docs by rephrasing some unclear documentation (#9621, @anthonyeleven)
• Add mirror doc to support mirror usage. (#9396, @yankay)
• [CI] Add check_typo job (and fix a bunch of typos) (#9361, @oomichi)
• [CI] Stop using python 'test' internal package (#9454, @olivierlemasle)
• [CI] Update securityContext of netchecker (#9398, @oomichi)
• [CI] Use agnhost instead of busybox for network test (#9390, @oomichi)
• [CI] Add ubuntu20 hardening job (#9359, @oomichi)
• [CI] Fix YAML format in hardening.md file (#9387, @oomichi)
• [CI] Make vagrant-ubuntu20-flannel voting (by removing allow failure) (#9469, @oomichi)
• [CI] Update sonobuoy version to a more recent one (#9485, @oomichi)
• [CI] Increase the fedora memory at CI to fix the CI broken (#9640, @yankay)
• [CI] Add CI for rockylinux9 and cilium (#9562, @yankay)

Component versions

• Core
  • kubernetes v1.25.6
  • etcd v3.5.6
  • docker v20.10 (cri_dockerd: v0.3.0)
  • containerd v1.6.15
  • cri-o v1.24
• Network Plugin
  • cni-plugins v1.2.0
  • calico v3.24.5
  • cilium v1.12.1
  • flannel v0.20.2
  • kube-ovn v1.10.7
  • kube-router v1.5.1
  • multus v3.8
  • weave v2.8.1
  • kube-vip v0.5.5
• Application
  • cert-manager v1.11.0
  • coredns v1.9.3
  • ingress-nginx v1.5.1
  • krew v0.4.3
  • argocd v2.5.7
  • helm v3.10.3
  • metallb v0.12.1
  • registry v2.8.1
• Storage Plugin
  • cephfs-provisioner v2.1.0-k8s1.11
  • rbd-provisioner v2.1.1-k8s1.11
  • aws-ebs-csi-plugin v0.5.0
  • azure-csi-plugin v1.10.0
  • cinder-csi-plugin v1.22.0
  • gcp-pd-csi-plugin v1.4.0
  • local-path-provisioner v0.0.22
  • local-volume-provisioner v2.5.0

Known issues

N/A

Notes

1. As stated in cloud-provider-openstack:1.23.0: Load balancers don't relate to a dedicated Service anymore, any scripts relying on that relationship previously need to change to use the load balancer tags instead