kata-containers Changes

Feature highlights include:

• Firecracker has been updated to v1.1.0
• Nydus has been updated to v2.1.0-alpha.4
• Cloud Hypervisor has been updated to v25.0
• Support containerd shimv2 logging plugin
• Support virtio-block multiqueue
• Support QEMU sandbox feature
• Switch to rust version virtiofsd
• Support core scheduling with containerd
• kata-runtime iptables subcommand to manipulate iptables in the guest
• A few new subcommands for runk
• Support direct-assigned volumes
• Many bugfix, CI and packaging improvements.

Shortlog

da875e747 release: Kata Containers 2.5.0 05b2096c0 release: Adapt kata-deploy for 2.5.0 1b930156c build: Fix clh source build as normal user 01c889fb6 runtime: Fix DisableSelinux config 59bd5c2e0 container: kill all of the processes in this container 22c005f55 nydus: upgrade nydus/nydus-snapshotter version 8220e5478 runtime: add unlock before return in sendReq 4f0ca40e0 versions: Update Firecracker version to v1.1.0 da24fd88e clh: Don't crash if no network device is set by the upper layer ed25d2cf5 versions: Update Cloud Hypervisor to v25.0 dfc1413e4 action: extend commit message line limit to 150 bytes 2d29791c1 release: Kata Containers 2.5.0-rc0 f4eea832a release: Adapt kata-deploy for 2.5.0-rc0 96553e8bd runtime: Add documentation of drop-in config file fragments c656457e9 runtime: Add tests of drop-in config file decoding 99f5ca80f runtime: Plug drop-in decoding into decodeConfig() 0f9856c46 runtime: Scan drop-in directory, read files and decode them 2c1efcc69 runtime: Add helpers to copy fields between tomlConfig instances 20f11877b runtime: Add framework to manipulate config structs via reflection 2a4fbd6d8 agent: enhance get handled signal 0ddb34a38 oci: fix serde skip serializing condition acd3302be agent: Run OCI poststart hooks after a container is launched fbb2e9bce agent: Replace some libc functions with nix ones 1f363a386 runtime: overwrite mount type to bind for bind mounts 4e48509ed build: Set safe.directory for runtime repo 433816cca ci/cd: update check-commit-message 48ccd4233 ci: Set safe.directory against tests repository a5a25ed13 runtime: delete Console from Cmd type 323271403 virtcontainers: Remove unused function 0939f5181 config: Expose default_maxmemory 58ff2bd5c clh,qemu: Adapt to using default_maxmemory afdc96042 hypervisor: Add default_maxmemory configuration ab5f1c956 shim: set a non-zero return code if the wait process call failed. e5be5cb08 runtime: device: cleanup outdated comments 5f936f268 virtcontainers: config validation is host specific bdf5e5229 virtcontainers: validate hypervisor config outside of hypervisor itself 469e09854 katautils: don't do validation when loading hypervisor config 1a78c3df2 packaging: Remove unused kata docker configure script 0e2459d13 docs: Add cgroupDriver for containerd 4e30e11b3 shim: support shim v2 logging plugin e32bf5331 device: deduplicate state structures f97d9b45c runtime: device/persist: drop persist dependency from device pkgs f9e96c650 runtime: device: move to top level package 3880e0c07 agent: refactor reading file timing for debugging 93874cb3b packaging: Restrict kernel patches applied to top-level dir 07b1367c2 versions: Update kernel to latest LTS version 5.15.48 1b7d36fdb agent: Allow BUILD_TYPE=debug c70d3a2c3 agent: Update the dependencies 612fd79ba random: Fix "nonminimal-bool" clippy warning d4417f210 netlink: Fix "or-fun-call" clippy warnings e227b4c40 block: Leverage multiqueue for virtio-block 9ff10c083 kernel: Add CONFIG_EFI=y as part of the TDX fragments e7e7dc9df runtime: Add heuristic to get the right value(s) for mem-reserve ef925d40c runtime: enable sandbox feature on qemu 0bbbe7068 snap: fix snap build on ppc64le c7dd10e5e packaging: Remove unused publish kata image script 1b7fd19ac rootfs: Fix chronyd.service failing on boot 28995301b tracing: Remove whitespace from root span 9941588c0 workflow: Removing man-db, workflow kept failing a305bafee docs: Update outdated URLs and keep them available 721ca72a6 runtime: fix error when trying to parse sandbox sizing annotations 90a7763ac snap: Fix debug cli option 5d7fb7b7b build(deps): bump github.com/containerd/containerd in /src/runtime d0ca2fcbb build(deps): bump crossbeam-utils in /src/tools/trace-forwarder a60dcff4d build(deps): bump regex from 1.5.4 to 1.5.6 in /src/tools/agent-ctl dbf50672e build(deps): bump crossbeam-utils in /src/tools/agent-ctl 8e2847bd5 build(deps): bump crossbeam-utils from 0.8.6 to 0.8.8 in /src/libs e9ada165f build(deps): bump regex from 1.5.4 to 1.5.5 in /src/agent adad9cef1 build(deps): bump crossbeam-utils from 0.8.5 to 0.8.8 in /src/agent ac5dbd859 clh: Improve logging related to the net dev addition 0b75522e1 network: Set queues to 1 to ensure we get the network fds 93b61e0f0 network: Add FFI_NO_PI to the netlink flags bf3ddc125 clh: Pass the tuntap fds down to Cloud Hypervisor 55ed32e92 clh: Take care of the VmAdNetdPut request ourselves 01fe09a4e clh: Hotplug the network devices 2e0753833 clh: Expose VmAddNetPut bee770343 docs: Update containerd url link 1a5ba31cb agent: refactor reading file timing for debugging bb26bd73b safe-path: fix clippy warning db5048d52 kernel: build efi_secret module for SEV 1ef0b7ded runtime: Switch to using the rust version of virtiofsd (all but power) 9773838c0 virtiofsd: export env vars needed for building it eff4e1017 shim: change the log level for GetOOMEvent call failures 412441308 docs: Add more kata monitor details 8f10e13e0 config: Allow enable_iommu pod annotation by default b0e090f40 versions: Bump virtiofsd to v1.3.0 1b845978f docs: Add storage limits to arch doc 7ae11cad6 docs: Update source for cri-tools f5099620f tools: Enable extra detail on error 34bcef884 docs: Add agent-ctl examples section 815157bf0 docs: Remove erroneous whitespace eb24e9715 release: Kata Containers 2.5.0-alpha2 d2df1209a docs: describe kata handling for core-scheduling 22b6a94a8 shim: add support for core scheduling fe3c1d9cd docs: Update storage documentation link 6ecea84bc rustjail: get home dir using nix crate 38a318820 runk: Support list sub-command 6d0ff901a docs: Update vGPU use-case 9d27c1fce agent: ignore ESRCH error when destroying containers 9726f56fd runtime: force stop container after the container process exits 168f325c4 docs: Update configuration reference for snap documentation b9fc24ff3 docs: update release process github token instructions c1476a174 docs: update release process with latest workflow triggering 8b57bf97a workflows: add workflow_dispatch triggering to test-kata-deploy 002f2cd10 snap: Use helper script and cleanup 9b108d993 docs: Improve snap formatting 894f661cc docs: Add warning to snap build d759f6c3e snap: Fix CH architecture check 56591804b docs: Improve snap build instructions cb2b30970 snap: Build using destructive mode 60823abb9 docs: Move snap README af2ef3f7a agent-ctl: introduce handle for iptables get/set 65f0cef16 kata-runtime: add iptables CLI to test http endpoint 3201ad083 shim-client: ensure we check resp status for Put/Post 0706fb28a kata-runtime: shmgmt: make url usage consistent 2a09378dd shim-client: add support for DoPut 640173cfc shim-mgmt: Add endpoint handler for interacting with iptables 0136be22c virtcontainers: plumb iptable set/get from sandbox to agent bd50d463b agent: iptables: get/set handling for iptables 03176a9e0 proto: update generated code based on proto update 38ebbc705 proto: update to add set/get iptables 78d45b434 agent: return mount file content if parse mountinfo failed 2e04833fb docs: Update Intel QAT documentation links 7c4049aab osbuilder: add iptables package 648b8d0ae runk: Return error when tty is used without console socket 5205efd9b runk: Add Podman guide in README 590381574 agent: Pass standard I/O to container launched by runk c7b3941c9 runk: Enable test for the agent built with standard-oci-runtime feature 6dbce7c3d agent: Remove unused import in console test d862ca059 runk: Handle rootfs path in config.json properly c95ba63c0 docs: Remove information related to Kata 1.x 34b80382b docs: Get rid of note related to networking. dfad5728a docs: Mention --cni flag while invoking ctr fff832874 clh: Update to v24.0 49361749e snap: Build and package rust version of virtiofsd 27d903b76 snap: Put the yq binary in the staging bin directory d7b4ce049 snap: Remove unused variable 43de5440e snap: Fix unbound variable error c9b291509 snap: Fix whitespace 122a85e22 agent: remove bin oci-kata-agent 35619b45a runk: merge oci-kata-agent into runk 10c13d719 qemu: remove virtiofsd option in qemu config d20bc5a4d virtiofsd: build rust based virtiofsd from source for non-x86_64 8e7c5975c agent: fix direct-assigned volume stats 4428ceae1 runtime: direct-volume stats use correct name ffdc065b4 runtime: direct-volume stats update to use GET parameter f29595318 runtime: fix incorrect Action function for direct-volume stats 2a1d39414 runtime: Adding the correct detection of mediated PCIe devices ce2e521a0 runtime: remove duplicate 'types' import 7a5ccd126 runtime: sync docstrings with function names 834f93ce8 docs: fix annotations example f4994e486 runtime: allow annotation configuration to use_legacy_serial c67b9d297 qemu: allow using legacy serial device for the console 44814dce1 qemu: treat console kernel params within appendConsole 24a2b0f6a docs: Remove clear containers reference in README 8052fe62f runtime: do not check for EOF error in console watcher abad33eba kernel: Remove nemu.conf from packaging e87eb13c4 tools: delete unused param from get_from_kata_deps callers 4b437d91f agent: Fix is_signal_handled failing parsing str to u64 e73b70baf runtime: Don't run unit tests verbose by default f24a6e761 runtime: Consolidate flags setting in unit tests script cf465feb0 runtime: Don't change test behaviour based on $CI or $KATA_DEV_MODE 34c4ac599 runtime: Remove redundant subcommands from go-test.sh 0aff5aaa3 runtime: Simplify package listing in go-test.sh 557c4cfd0 runtime: Don't chmod coverage files in Go tests 04c8b52e0 runtime: Remove HTML coverage option from go-test.sh 7f7691442 runtime: Add coverage.txt.tmp to gitignore 13c257700 runtime: Move go testing script locally 4f586d2a9 packaging: Add kernel config option for SGX in Gramine 7bc4ab68c ci: Don't run Docs URL Alive Check workflow on forks b4b9068cb tools: Add QEMU patches for SGX numa support 88fb9b72e docs: Update runc containerd runtime a475956ab workflows: Add support for building virtiofsd 71f59f3a7 local-build: Add support for building virtiofsd c7ac55b6d dockerbuild: Install unzip 8e2042d05 tools: add script to pull virtiofsd dbedea508 versions: Add virtiofsd entry 421064680 doc: Update log parser link 271933fec log-parser: fix some of the documentation c7dacb121 log-parser: move the kata-log-parser from the tests repo 82ea01828 versions: Upgrade to Cloud Hypervisor v23.1 383be2203 agent: Add a macro to skip a loop easier 97d7b1845 runk: use custom Kill command to support --all option 475e3bf38 agent: add test coverage for functions find_process and online_resources 4a1e13bd8 rustjail: Add tests for hook_grpc_to_oci 9b863b0e0 release: Kata Containers 2.5.0-alpha1 70eda2fa6 agent: watchers: ensure uid/gid is preserved on copy/mkdir 33a8b7055 clh: Rely on Cloud Hypervisor for generating the device ID 81f6b4862 agent: add tests for create_logger_task function 7772f7dd9 runk: set BinaryName for runk for containerd b221a2590 tools: Add runk 2c218a07b agent: Modify Kata agent for runk b0e439cb6 rustjail: add tests for parse_mount_table b975f2e8d Virtcontainers: Enable hot plugging vhost-user-blk device on ARM 7ffe5a16f docs: Direct-assigned volume design 081f6de87 versions: change qemu tdx url and tag dd4bd7f47 doc: Added initial doc update for NV GPUs 666aee54d docs: Add VSOCK localhost example for agent-ctl 86d348e06 docs: Use VM term in agent-ctl doc 4b9b62bb3 agent-ctl: Fix abstract socket connections b6467ddd7 clh: Expose disk rate limiter config 7580bb5a7 clh: Expose net rate limiter config a88adabaa clh: Cloud Hypervisor has a built-in Rate Limiter 63c4da03a clh: Implement the Disk RateLimiter logic 511f7f822 config: Add DiskRateLimiter* to Cloud Hypervisor 5b18575df hypervisor: Add disk bandwidth and operations rate limiters 1cf946929 clh: Implement the Network RateLimiter logic 00a5b1bda utils: Define DefaultRateLimiterRefillTimeMilliSecs be1bb7e39 utils: Move FC's function to revert bytes to utils c9f6496d6 config: Add NetRateLimiter* to Cloud Hypervisor 2d35e6066 hypervisor: Add network bandwidth and operations rate limiters ccb018393 kata-deploy: Add support to RKE2 9d39362e3 kata-deploy: Reestructure the installing section 18d27f794 kata-deploy: Add a missing $ prefix in the README 6948b4b36 docs: Update containerd link to installation guide 832c33d5b docs: remove pc machine type supports 1cad3a469 agent/random: Ensure data.len > 0 33c953ace agent: Add test_ressed_rng_not_root 39a35b693 agent: Add test to random::reseed_rng() d8f39fb26 agent/random: Rename RNDRESEEDRNG to RNDRESEEDCRNG 4b9e78b83 rustjail: Add tests for mount_grpc_to_oci b658dccc5 tools: fix typo in clh directory name afbd60da2 packaging: Fix clh build from source fall-back 1b931f420 runtime: Allock mockfs storage to be placed in any directory ef6d54a78 runtime: Let MockFSInit create a mock fs driver at any path 5d8438e93 runtime: Move mockfs control global into mockfs.go 963d03ea8 runtime: Export StoragePathSuffix 1719a8b49 runtime: Don't abuse MockStorageRootPath() for factory tests bec59f9e3 runtime: Make bind mount tests better clean up after themselves f7ba21c86 runtime: Clean up mock hook logs in tests 90b2f5b77 runtime: Make SetupOCIConfigFile clean up after itself 2eeb5dc22 runtime: Don't use fixed /tmp/mountPoint path f385b21b0 rustjail: add tests for mount_from function 96bc3ec2e rustjail: Add tests for hooks_grpc_to_oci 023950278 agent: modify the type of swappiness to u64 0ad89ebd7 safe-path: add more unit test cases b63774ec6 libs/safe-path: add crate to safely resolve fs paths 0e7f1a5e3 agent: move assert_result macro to test_utils file 2256bcb6a rustjail: Add tests for root_grpc_to_oci 9b6f24b2e agent: add tests for mount_to_rootfs function 9c22d9554 agent: add tests for update_container_namespaces c3776b179 agent: add tests for is_signal_handled function 29e569aa9 virtcontainers: clh: Re-generate the client code 6012c1970 versions: Upgrade to Cloud Hypervisor v23.0 aabcebbf5 agent: best-effort removing mount point d136c9c24 test: Fix golangci-lint error for s390x 92c00c7e8 agent: fsGroup support for direct-assigned volume 532d53977 runtime: fsGroup support for direct-assigned volume 6a47b82c8 proto: fsGroup support for direct-assigned volume 7b2ff0264 kata-monitor: add a README file 86977ff78 kata-monitor: update the hrefs in the debug/pprof index page 354cd3b9b runtime: Base64 encode the direct volume mountInfo path 6e79042aa runtime: no need to write virtiofsd error to log f8cc5d1ad kata-monitor: add some links when generating pages for browsers 78f30c33c agent: Avoid agent panic when reading empty stats 6e9e4e8ce docs: Update link to contributions guide 9d5e7ee0d agent: add tests for mount_storage 1118a3d2d agent: add test coverage for parse_mount_flags_and_options function 485aeabb6 agent: add tests for do_write_stream function 9d5b03a1b runtime: delete debug option in virtiofsd c31cd0e81 rustjail: add test coverage for process_grpc_to_oci function eff7c7e0f agent: Allow the agent to be rebuilt with the change of Cargo features 962d05ec8 protocols: add src/csi.rs to .gitignore a2f5c1768 runtime/virtcontainers: Pass the hugepages resources to agent 4405b188e docs: Add a firecracker installation guide ff17c756d runtime: Allow and require no initrd for SE 59c7165ee test: use T.TempDir to create temporary test directory 98750d792 clh: Expose service offload configuration c9e24433d release: Kata Containers 2.5.0-alpha0 0d5f80b80 versions: Bump firecracker to v0.23.4 800e4a9cf agent: use ms as unit of cputime instead of ticks 0d765bd08 agent: fix container stop error with signal SIGRTMIN+3 9e4ca0c4f doc: Improve kata-deploy README.md by changing sh blocks to bash blocks 2b91dcfee docs: Remove kata-proxy reference a63bbf979 kata-monitor: fix duplicated output when printing usage 5e1c30d48 runtime: add logs around sandbox monitor fb8be9619 runtime: stop getting OOM events when ttrpc: closed error a779e19be tools/packaging: Fix error path in 'kata-deploy-binaries.sh -s' 0baebd2b3 tools/packaging: Fix usage of kata-deploy-binaries.sh 93d03cc06 kata-deploy: fix version bump from -rc to stable 3606923ac workflows,release: Ship all the rust vendored code 2eb07455d tools: Add a generate_vendor.sh script ecf71d6dd docs: Remove VPP documentation 66f05c5bc runtime: Remove the explicit VirtioMem set and fix the comment 154c8b03d tools/packaging/kata-deploy: Copy install_yq.sh in a dedicated script 1ed7da8fc packaging: Eliminate TTY_OPT and NO_TTY variables in kata-deploy bad859d2f tools/packaging/kata-deploy/local-build: Add build to gitignore a93140237 docs: Remove kata-proxy references in documentation 0928eb9f4 agent: Kill the all the container processes of the same cgroup 19f372b5f runtime: Add more debug logs for container io stream copy c27963276 osbuilder/qat: don't pull kata sources if exist 774348641 docs: fix markdown issues in how-to-run-docker-with-kata.md 459f4bfed osbuilder/qat: use centos as base OS 9a5b47706 docs: Update vcpu handling document 32131cb8b Agent: fix unneeded late initialization lint ebec6903b static-build,clh: Add the ability to build from a PR c77e34de3 runtime: Move mock hook source 86723b51a virtcontainers: Remove unused install/uninstall targets 0e83c95fa virtcontainers: Run mock hook from build tree rather than system bin dir e65db838f virtcontainers: Remove VC_BIN_DIR c20ad2836 virtcontainers: Remove unused Makefile defines c776bdf4a virtcontainers: Remove unused parameter from go-test.sh 168fadf1d ci: Weekly check whether the docs url is alive 72f7e9e30 osbuilder: Multistrap Ubuntu df511bf17 packaging: Enable cross-building agent 0a313eda1 osbuilder: Fix use of LIBC in rootfs.sh 2c86b956f osbuilder: Simplify Rust installation 0072cc2b6 osbuilder: Remove musl installations 5c3e55362 osbuilder: apk add --no-cache efa19c41e device: use const strings for block-driver option instead of hard coding 24b29310b doc: update Intel SGX use cases document 18d4d7fb1 tools: update QEMU to 6.2 62351637d action: Update link for format patch documentation aa5ae6b17 runtime: Properly handle ESRCH error when signaling container 5c434270d docs: Update k8s documentation 92ce5e2dc rustjail: optimization, merged several writelns into one dacf6e395 doc: fix filename typo 7a18e32fa versions: Upgrade to Cloud Hypervisor v22.1 be12baf3c manager: Change here documents to use standard delimiter 9576a7da5 manager: Add options to change self test behaviour d4d65bed3 manager: Add option to enable component debug 019da91d7 manager: Whitespace fix d234cb76b manager: Create containerd link 5d6d39be4 scripts: Change here document delimiters c088a3f3a agent: add tests for get_memory_info function 4b1e2f527 CI: Update GHA secret name 4adf93ef2 tools: release: Do not consider release candidates as stable releases 5ec7592df kernel: fix cve-2022-0847 ffdf961ae docs: Update contact link in runtime README 42e35505b agent: Verify that we allocated as many hugepages as we need 608e003ab agent: Don't attempt to create directories for hugepage configuration 6a850899c CI: Create GHA to add PR sizing label 2b41d275a release: Revert kata-deploy changes after 2.4.0-rc0 release

Compatibility with CRI-O

Kata Containers 2.5.0 is compatible with CRI-O

Compatibility with containerd

Kata Containers 2.5.0 is compatible with contaienrd v1.5.2

OCI Runtime Specification

Kata Containers 2.5.0 support the OCI Runtime Specification v1.0.0-rc5

Compatibility with Kubernetes

Kata Containers 2.5.0 is compatible with Kubernetes 1.23.1-00

Libseccomp Notices

The kata-agent binaries inside the Kata Containers images provided with this release are statically linked with the following GNU LGPL-2.1 licensed libseccomp library.

• libseccomp

The kata-agent uses the libseccomp v2.5.1 which is not modified from the upstream version. However, in order to comply with the LGPL-2.1 (§6(a)), we attach the complete source code for the library.

If you want to use the kata-agent which is not statically linked with the library, you can build a custom kata-agent that does not use the library from sources. For the details, please check the developer guide.

Kata Linux Containers image

Agent version: 2.5.0

Default Image Guest OS:

description: | Root filesystem disk image used to boot the guest virtual machine. url: "https://github.com/kata-containers/kata-containers/tools/osbuilder" architecture: aarch64: name: "ubuntu" version: "latest" ppc64le: name: "ubuntu" version: "latest" s390x: name: "ubuntu" version: "latest" x86_64: name: "clearlinux" version: "latest" meta: image-type: "clearlinux"

Default Initrd Guest OS:

description: | Root filesystem initrd used to boot the guest virtual machine. url: "https://github.com/kata-containers/kata-containers/tools/osbuilder" architecture: aarch64: name: "alpine" version: "3.15"

Do not use Alpine on ppc64le & s390x, the agent cannot use musl because

there is no such Rust target

ppc64le: name: "ubuntu" version: "20.04" s390x: name: "ubuntu" version: "20.04" x86_64: name: "alpine" version: "3.15"

Kata Linux Containers Kernel

Kata Containers 2.5.0 suggest to use the Linux kernel v5.15.48 See the kernel suggested Guest Kernel patches See the kernel suggested Guest Kernel config

Installation

Follow the Kata installation instructions.

Issues & limitations

More information Limitations