Deprecation / Removal

• Drop Ansible support for v2.9 and v2.10 (#8925, @oomichi)
• Drop support for Fedora 34 (#8967, @floryut)

Feature / Major changes

• Add Rocky Linux 8 support (#8905, @oomichi)
• Add Kylin Linux support. (#9078, @ErikJiang)
• Add Fedora36 support (#8967, @floryut)
• Add 'flush ip6tables' task in reset role (#9168, @GreatLazyMan)
• Add tar in common required package (#9184, @yankay)
• Add support for NTP configuration. (#9027, @yankay)
• Increase ansible fact_caching_timeout (from 2 to 24 hours) (#9059, @rptaylor)
• Add kubelet systemd service hardening option kubelet_systemd_hardening: [true|false] (#9194, @alegrey91)
• Support timezone setting (#9263, @yankay)
• Update deprecated ansible include syntax (#9040, @boeto)
• Update etcd download url in offline.yml to use arch (#8943, @ErikJiang)
• Add Support for Rewrite Plugin to CoreDNS/NodelocalDNS (#9245, @eifelmicha)
• Add SeccompDefault admission plugin for kubelet (using new variable kubelet_seccomp_default) (#9074, @alegrey91)
• Add an optional extra_groups parameter for k8s_nodes (e.g. to configure calico route reflector nodes on Openstack using the calico_rr group) (#9211, @rptaylor)
• Add arm64 Flatcar OS's pypy bootstrapping support (#8959, @kerryeon) (see Notes 1)
• Add docker support for Kylin distributions (#9144, @ErikJiang)
• Add hashes for Kubernetes 1.24.3 , v1.22.12, v1.23.9 (#9092, @marcofortina)
• Add ingress nginx webhook (#9033, @liupeng0518)
• Add manage-offline-files.sh to collect necessary files and provides http file download service for offline deployment. (#8956, @ErikJiang)
• Add missing configuration for extra tolerations (#8908, @smasset)
• Add support for node & pod pid limits (in kubelet-config file) (#9038, @h9-HSFRQDH)
• Add the option to enable default Pod Security Configuration (#9017, @Foxlik)
• Add unsafe_show_logs switch to show more log details (default to false, same as previous behavior) (#9164, @ErikJiang)
• Add variables (delete_node_retries,delete_node_delay_seconds) to tweak remove node process (#9096, @ydFu)
• Added 'avoid-buggy-ips' support of MetalLB (metallb_avoid_buggy_ips for default IP address pool and avoid_buggy_ips for additional IP address pools defined in metallb_additional_address_pools) (#9166, @kerryeon) (see Notes 2)
• Adjust the default value of calico blockSize ipv4 to 26, and ipv6 to 122. (#9055, @cyclinder)
• Make kubernetes owner parametrized (using kube_owner/kube_cert_group/etcd_owner variables) (#8952, @alegrey91)
• Move old etcd backup removal after etcd restart, to prevent removing backup if etcd fail (#9147, @emiran-orange)
• Supports reserve ephemeral-storage (#8895, @Thearas)
• [dev/docs] add support for pre-commit hook (#9158, @cristicalin)
• [etcd] Etcd role won't run on all nodes everytime. (#9173, @liupeng0518)
• [etcd] add 3.5.4 and drop 3.5.1 and 3.5.2 (#9021, @cristicalin)
• [infra] bump pause container to 3.6 (#9024, @cristicalin)
• Update Kubernetes dashboard to 2.6.0 (k8s 1.24 support) (#8906, @floryut)
• [kubernetes] make 1.24.x the new default (#8935, @cristicalin)
• [kubernetes] drop support for 1.21.x (#8935, @cristicalin)
• [kubernetes] drop support for deprecated dynamic_kubelet_configuration (#8935, @cristicalin)
• [offline] Archive offline-files and env NO_HTTP_SERVER to skip Nginx container running. (#9068, @yjqg6666)
• Adds support for multiple architectures to yq (#9288, @ErmalKristo)
• Add variable to tweak the vsphere-csi namespace (vsphere_csi_namespace) (#9278, @MahdiAbbasi95)
• Ensure ping package is installed on the system (#9284, @yankay)
• Add more functionalty to DNS configuration (#9270, @eminaktas)
• Ensure ostree variable has been defined for fcos (#9321, @electrocucaracha)
• Support removing options in resolvconf with tab separator (#9304, @2k0ri)
• preinstall: Add nodelocaldns to supersede_nameserver if enabled (#9282, @azuwis)

Network

• [Calico] calico rr now supports multiple groups (#9134, @liupeng0518)
• [Calico] drop support for 3.19.x and 3.20.x
• [Calico] Make Calico CNI log path configurable and allow disabling this log (#8921, @fungusakafungus)
• [Calico] The NAT (nat_outgoing) would not be disabled automatically when enabling peer_with_router. (#9255, @kerryeon)
• [Calico] The variable calcio_ipam_autoallocateblocks has been renamed to calico_ipam_autoallocateblocks (#9056, @liupeng0518)
• [Calico] calico-typha metrics port are now exposed when metrics are enabled (#8855, @vjacynycz)
• [Calico] Add Wireguard support for Rocky Linux 9 (#9287, @krystianmlynek)
• [Calico] The parameter name calcio_rr_id Is renamed to calico_rr_id for fixing a typo ⚠️ (#9327, @kerryeon)
• [Canal] update templates to work again with both etcd and k8s datastore (#9113, @floryut)
• [Cilium] Add list/watch nodes rules to cilium-operator clusterrole. (#9178, @Thearas)
• [Cilium] Add support for the updated (startup|liveness|readiness)Probe.Port numbers (#9031, @tomberget)
• [Cilium] Update cilium to v1.11.7 (#9119, @dkhachyan)
• [Cilium] Make rolling-restart readiness wait delay and count configurable via cilium_rolling_restart_wait_retries_{count, delay_seconds} (#9176, @Tristan971)
• [Cilium] Upgrades cilium to 1.11.6 and add some default variables. (#9065, @eminaktas) (See Notes 3)
• [Cilium] Update Cilium default to 1.12.x (#9225, @necatican) (See Notes 5)
• [Cilium] Dropped support for < v1.10.0 (#9225, @necatican)
• [Cilium] cilium_ip_masq_agent_enable variable no longer exists. Use enable-ipv4-masquerade and enable-ipv4-masquerade to enable masquerade. (#9225, @necatican)
• [flannel] update to v1.18.1 & make it default (#9104, @mzaian)
• [flannel] update to v1.19.2 & make it default (#9296, @mzaian)
• [Kube-vip] Fail if kube_proxy_strict_arp is set to false in arp mode (#9223, @yankay)
• [Multus] Support multi-architecture installation (#9012, @cyclinder)

Applications

• [Openstack] Add option to use default deny firewall policy and port allowlisting on UpCloud (#9058, @Ajarmar)
• [Openstack] Fix subnet order and number of master nodes (#9159, @robinelastisys)
• [Metallb] Renamed matallb_auto_assign variable to metallb_auto_assign (users disabling 'auto-assign' in metallb must update the variable name) (#8949, @orange-llajeanne)
• [vSphere-csi] Add nodeAffinity to daemonset using vsphere_csi_node_affinity variable (#9293, @dmitrytretyakov)
• [upcload-csi] Bump driver version to v0.3.3 (#9317, @robinAwallace)

Container-Managers

• [containerd] add hashes for 1.5.12, 1.5.13, 1.6.5 and 1.6.6, make 1.6.6 the new default (#8980, @cristicalin)
• [containerd] Add LimitMEMLOCK parameter configuration in containerd.service (using containerd_limit_[proc_num/core/open_file_num/mem_lock) (#9269, @ErikJiang)
• [containerd] Remove duplication in containerd template (#9301, @fungusakafungus)
• [containerd] Allow configuring base_runtime_spec per containerd runtime and supply a default runtime spec (#9302, @fungusakafungus)
• [Docker] use cri-dockerd instead of dockershim by default
• [Docker] Enable cri-dockerd service to prevent issue with reboot (#9201, @mostafaghadimi)
• [cri-o] Add dpkg hold for apt installs (#9075, @SamuelBECK1)
• [cri-o] add support for 1.24.x required by kubernetes 1.24.x (#8935, @cristicalin)
• [runc] update versions for 1.1.x and drop 1.0.x (#9022, @cristicalin)
• [runc] Variable containerd_default_runtime is now undifined by default (but default to runc) (#9026, @rptaylor)
• [crun] add 1.4.5 and drop 1.2 and 1.3 (#9023, @cristicalin)
• [nerdctl] upgrade to 0.20.0 (#8980, @cristicalin) then 0.22.2 (#9180, @panpan0000)

Bug or Regression

• Fix failure to look up user etcd when adding a user (#9016, @yankay)
• Fixing setting up kubespray on Azure with CSI drivers. (#9153, @wayfrro)
• Add --supervisor-fss-namespace=kube-system flag to vcloud-csi installation (#9066, @yasintahaerol)
• Add assertion for IPv4 check in verify settings (to allow IPv6 deployments) (#8946, @Citrullin)
• Add calico-kube-controllers missing verbs (#9032, @ghostloda)
• Allow "openSUSE Tumbleweed" to be run (again) (#9072, @oomichi)
• Apply calico bgp peer definition task to all nodes (#8974, @orange-llajeanne)
• Create snapshot namespace only when needed (#9014, @robinAwallace)
• Disable kubelet_authorization_mode_webhook by default (#9238, @cristicalin)
• Disabled DNSStubListener for Flatcar Linux (#9160, @kerryeon)
• Do not run etcd role in scale.yml playbook when etcd installed by kubeadm (#9210, @LuckySB)
• Fix Hetzner CCM cluster-cidr (wrongly set to a static value) (#9127, @ym)
• Fix calicoctl.sh path error when getting calico configuration (#9217, @tasekida)
• Fix failing tasks when calico_datastore is set to etcd (#9228, @chadswen)
• Fix missing quote in task "See if node is schedulable" (#9146, @emiran-orange)
• Fix number node name can't be added. (#9266, @cleverhu)
• Fix regex for replacing http_proxy host in RedHat Subscription Manager (#8957, @dicksontung)
• Fix some docker reset task (don't remove already uninstalled packages, ignore error on remove docker config files if already removed) (#8966, @orange-llajeanne)
• Fix the Centos/RHEL docker installation issue in ARM64 (#9047, @yankay)
• Fix the kube-vip missed SAN issue (#9099, @yankay)
• Fixed concatenate str & int in auto_renew_certificates_systemd_calendar (#8979, @floryut)
• Fixes the issue when it cannot correctly set the namespace for vphere-csi-driver (#9046, @eminaktas)
• Fixes vSphere CSI for vSphere CSI >= 2.4.0 on vSphere 6.7U3 (#8944, @snowball77)
• No more errors are emitted when attempting to delete worker nodes that do not exist. (#9244, @kerryeon)
• Optimize the format of evictionHard in kubelet-config.yaml template (#9204, @shelmingsong)
• Remove kubeowner different than root condition for user creation (#9125, @alegrey91)
• Remove unneeded socat wrapper installation for Flatcar (#8970, @kerryeon) (See Notes 4)
• Set fallback value of kubelet ip6 (#8926, @kerryeon)
• Swap calico download url, as the old primary url was deprecated and artefact no longer published (#8920, @sathieu)
• Upgrade the nginx-proxy and haproxy image version , and use the alpine base image (#9100, @yankay)
• Variable kube_pid_reserved must be a string (#9124, @liupeng0518)
• [Docker] Add restart of docker.service during install (#9205, @krystianmlynek)
• [Kube-ovn] Value check for HW_OFFLOAD is now correctly handle (and will no longer always be false) (#9218, @floryut)
• [ingress-nginx] Fix ingress-nginx RBAC rules when deployed classless (#9156, @cristicalin)
• Remove the 'etcd-unsupported-arch' args to fix the etcd issue in arm64 (#9049, @yankay)
• Fix duplicate field in ingress-nginx template (#9285, @cloud-66)
• Fix CoreDNS memory leak issue by adding max_concurrent=1000 in the CoreDNS config (#9307, @yankay)
• Fix ansible user module create_home property (erroneously written as createhome) (#9314, @liupeng0518)
• Hotfix containerd restart not properly restarting (#9322, @fungusakafungus)

Other (Cleanup or Flake)

• [CI] upgrade vagrant image for opensuse leap to 15.4 (#9175, @cristicalin)
• [CI] test upgrade with defaults (containerd) instead of docker (#8980, @cristicalin)
• [CI] Fix cloud_init files for different distros (#9232, @floryut)
• git ignore .terraform.lock.hcl in all folders (#9109, @rptaylor)

Component versions:

• Core
  • Kubernetes v1.24.6
  • Etcd v3.5.4
  • Docker v20.10
  • Containerd v1.6.8
  • CRI-O v1.24
• Network
  • CNI-plugins v1.1.1
  • Calico v3.23.3
  • Cilium v1.12.1
  • Flannel v0.19.2
  • Kube-ovn v1.9.7
  • Kube-Router v1.5.1
  • Multus v3.8
  • Weave v2.8.1
  • kube-vip v0.4.2
• App
  • Cert-manager v1.9.1
  • CoreDNS v1.8.6
  • Nginx-ingress v1.3.1
  • krew v0.4.3
  • argocd v2.4.12
  • helm v3.9.4
  • metallb v0.12.1
  • registry v2.8.1

Known issues

• Host network might broke when an interface goes down (Cilium 1.12/Ubuntu 22.04), please read Note 5.
• If bin_dir value is changed to something other than /usr/local/bin, containerd configuration might need to be tweak, please check #9243

Notes

1. Upgrading the bootstrap pypy may cause some unexpected behaviors for Flatcar use-cases)
2. As the newly added feature uses the default value of MetalLB as same, there is no side effect for users who do not change it's value
3. This PR also implements cgroup auto-mount. By default, it is enabled. You can disable it by adding cgroup_auto_mount: false. Moreover, you can enable or disable BPF with these variables cilium_enable_bpf_masquerade and cilium_enable_host_legacy_routing
4. Some old (<2020Y) 'Flatcar Container Linux by Kinvolk' may not be supported.
5. With Cilium 1.12/Ubuntu 22.04, you might run into this issue, workaround are available while the issue is resolved on cilium end.