Introduced estargz library for manipulating eStargz layer (#195, #196)

estargz is the Go library for manipulating eStargz layers, based on stargz library by CRFS project (https://github.com/google/crfs/tree/master/stargz). Applications can convert tar blob into eStargz layer with specifying prioritized files and also can unmarshal it into the structured tree format. Because estargz package has its own go.mod, applications outside of this project can import this package with minimal dependencies (i.e. without snapshotter-related nor optimizer-related dependencies).

Leveraging this package, go-containerregistry and crane CLI, Kaniko, ko and nerdctl start to experimentally support creating eStargz recently.

Big thanks to @mattmoor and go-containerregistry community for discussion and contribution for designing this package and expanding its adoption in community.

Introduced nativeconverter library and ctr-remote image convert CLI as containerd-native image converter (#224, #234, #236)

This release introduces brand-new containerd-native image converter library nativeconverter and ctr-remote image convert CLI.

nativeconverter is a high-level image converter library based on estargz package. This package works on containerd and can convert an image stored in containerd content store into eStargz or other formats (e.g. tar). The resulting image will also be stored to containerd content store.

nerdctl supports converting images into eStargz leveraging nativeconverter library.

nerdctl image convert --estargz --oci <SRC> <DST>

ctr-remote image convert is our CLI implementation based on nativeconverter. An image stored in containerd can be converted to eStargz using the following command.

ctr-remote images convert --oci --estargz <SRC> <DST>

Big thanks to @AkihiroSuda for introducing this feature and integrating this to nerdctl.

Note:

• nativeconverter and ctr-remote image convert don't support workload-based optimization which ctr-remote image optimize supports.
• In the futural release, ctr-remote image optimize (currently based on runc) will also be refactored based on nativeconverter and containerd.

Notable Changes

• Stargz Snapshotter

  • Added default path to load config (/etc/containerd-stargz-grpc/config.toml) (#214, #229) (thanks to @ohkinozomu)
  • Added fetching_timeout config option for making timeout for fetching chunks configurable (#228)
  • Added max_concurrency config option for making the concurrency of background fetching configurable (#237) (thanks to @mc256)
  • Fixed the order of directory entries returned from getdents was non-deterministic (#242)

• estargz library

  • Added WithCompressionLevel option for specifying gzip compression level of eStargz (#201)
  • Added WithPrioritizedFiles option for specifying prioritized files (#206, #212, #216)
  • Replaced the multierror dependency with a simple error aggregator (#207) (thanks to @mattmoor)
  • Added DiffID() API for exposing uncompressed digest (DiffID) of built eStargz (#208)
  • Added flexibility for manipulating paths formatted in various ways (e.g. paths prefixed with "/", "./" and "" are handled correctly) (#213, #218)
  • Allowed duplicated entries of input tar (#219)
  • Fixed stack overflow bug because of the infinite loop (#221)
  • Fixed paths aren't handled correctly on Windows (#225) (thanks to @mattmoor)

• ctr-remote image optimize

  • Refactored the structure of ctr-remote i optimize (#198, #199)
  • Added --wait-on-signal option for enabling the user to fully control container termination during optimization (#192) (thanks to @mc256)
  • Added --record-out option for dumping out monitor log to a file (#222)

• Docs and dependencies

  • Clarified the location of TOC digest annotation in eStargz spec (#211)
  • Allowed KinD image built by non- BuildKit builders (#232)
  • Bumping up dependencies (#197)
  • README.md: fix a wrong image tag (#191)
  • Fixed typos on docs (#233) (thanks to @fahedouch)