v0.17.0 (2021-11-05)

v0.17.0 is a major feature release.

Major features in this release

1. New concretizer is now default The new concretizer introduced as an experimental feature in v0.16.0 is now the default (#25502). The new concretizer is based on the clingo logic programming system, and it enables us to do much higher quality and faster dependency solving The old concretizer is still available via the concretizer: original setting, but it is deprecated and will be removed in v0.18.0.

2. Binary Bootstrapping To make it easier to use the new concretizer and binary packages, Spack now bootstraps clingo and GnuPG from public binaries. If it is not able to bootstrap them from binaries, it installs them from source code. With these changes, you should still be able to clone Spack and start using it almost immediately. (#21446, #22354, #22489, #22606, #22720, #22720, #23677, #23946, #24003, #25138, #25607, #25964, #26029, #26399, #26599).

3. Reuse existing packages (experimental) The most wanted feature from our 2020 user survey and the most wanted Spack feature of all time (#25310). spack install, spack spec, and spack concretize now have a --reuse option, which causes Spack to minimize the number of rebuilds it does. The --reuse option will try to find existing installations and binary packages locally and in registered mirrors, and will prefer to use them over building new versions. This will allow users to build from source far less than in prior versions of Spack. This feature will continue to be improved, with configuration options and better CLI expected in v0.17.1. It will become the default concretization mode in v0.18.0.

4. Better error messages We have improved the error messages generated by the new concretizer by using unsatisfiable cores. Spack will now print a summary of the types of constraints that were violated to make a spec unsatisfiable (#26719).

5. Conditional variants Variants can now have a when="<spec>" clause, allowing them to be conditional based on the version or other attributes of a package (#24858).

6. Git commit versions In an environment and on the command-line, you can now provide a full, 40-character git commit as a version for any package with a top-level git URL. e.g., spack install hdf5@45bb27f58240a8da7ebb4efc821a1a964d7712a8. Spack will compare the commit to tags in the git repository to understand what versions it is ahead of or behind.

7. Override local config and cache directories You can now set SPACK_DISABLE_LOCAL_CONFIG to disable the ~/.spack and /etc/spack configuration scopes. SPACK_USER_CACHE_PATH allows you to move caches out of ~/.spack, as well (#27022, #26735). This addresses common problems where users could not isolate CI environments from local configuration.

8. Improvements to Spack Containerize For added reproducibility, you can now pin the Spack version used by spack containerize (#21910). The container build will only build with the Spack version pinned at build recipe creation instead of the latest Spack version.

9. New commands for dealing with tags The spack tags command allows you to list tags on packages (#26136), and you can list tests and filter tags with spack test list (#26842).

Other new features of note

• Copy and relocate environment views as stand-alone installations (#24832)
• spack diff command can diff two installed specs (#22283, #25169)
• spack -c <config> can set one-off config parameters on CLI (#22251)
• spack load --list is an alias for spack find --loaded (#27184)
• spack gpg can export private key with --secret (#22557)
• spack style automatically bootstraps dependencies (#24819)
• spack style --fix automatically invokes isort (#24071)
• build dependencies can be installed from build caches with --include-build-deps (#19955)
• spack audit command for checking package constraints (#23053)
• Spack can now fetch from CVS repositories (yep, really) (#23212)
• spack monitor lets you upload analysis about installations to a spack monitor server (#23804, #24321, #23777, #25928))
• spack python --path shows which python Spack is using (#22006)
• spack env activate --temp can create temporary environments (#25388)
• --preferred and --latest options for spack checksum (#25830)
• cc is now pure posix and runs on Alpine (#26259)
• SPACK_PYTHON environment variable sets which python spack uses (#21222)
• SPACK_SKIP_MODULES lets you source setup-env.sh faster if you don't need modules (#24545)

Major internal refactors

• spec.yaml files are now spec.json, yielding a large speed improvement (#22845)
• Splicing allows Spack specs to store mixed build provenance (#20262)
• More extensive hooks API for installations (#21930)
• New internal API for getting the active environment (#25439)

Performance Improvements

• Parallelize separate concretization in environments; Previously 55 min E4S solve now takes 2.5 min (#26264)
• Drastically improve YamlFilesystemView file removal performance via batching (#24355)
• Speed up spec comparison (#21618)
• Speed up environment activation (#25633)

Archspec improvements

• support for new generic x86_64_v2, x86_64_v3, x86_64_v4 targets (see archspec#31)
• spack arch --generic lets you get the best generic architecture for your node (#27061)
• added support for aocc (#20124), arm compiler on graviton2 (#24904) and on a64fx (#24524),

Infrastructure, buildcaches, and services

• Add support for GCS Bucket Mirrors (#26382)
• Add spackbot to help package maintainers with notifications. See spack.github.io/spackbot
• Reproducible pipeline builds with spack ci rebuild (#22887)
• Removed redundant concretizations from GitLab pipeline generation (#26622)
• Spack CI no longer generates jobs for unbuilt specs (#20435)
• Every pull request pipeline has its own buildcache (#25529)
• --no-add installs only specified specs and only if already present in… (#22657)
• Add environment-aware spack buildcache sync command (#25470)
• Binary cache installation speedups and improvements (#19690, #20768)

Deprecations and Removals

• 0.17.x is the last Spack release series that will support Python 2.6. We will no longer retain 2.6 compatibility in develop and it will be unsupported in 0.18.
• spack setup was deprecated in v0.16.0, and has now been removed. Use spack develop and spack dev-build.
• Remove unused --dependencies flag from spack load (#25731)
• Remove stubs for spack module [refresh|find|rm|loads], all of which were deprecated in 2018.

Notable Bugfixes

• Deactivate previous env before activating new one (#25409)
• Many fixes to error codes from spack install (#21319, #27012, #25314)
• config add: infer type based on JSON schema validation errors (#27035)
• spack config edit now works even if spack.yaml is broken (#24689)

Packages

• Allow non-empty version ranges like 1.1.0:1.1 (#26402)
• Remove .99's from many version ranges (#26422)
• Python: use platform-specific site packages dir (#25998)
• CachedCMakePackage for using *.cmake initial config files (#19316)
• lua-lang allows swapping lua and luajit (#22492)
• Better support for ld.gold and ld.lld (#25626)
• build times are now stored as metadata in $prefix/.spack (#21179)
• post-install tests can be reused in smoke tests (#20298)
• Packages can use pypi attribute to infer homepage/url/list_url (#17587)
• Use gnuconfig package for config.guess file replacement (#26035)
• patches: make re-applied patches idempotent (#26784)

Spack community stats

• 5969 total packages, 920 new since v0.16.0
  • 358 new Python packages, 175 new R packages
• 513 people contributed to this release
  • 490 committers to packages
  • 105 committers to core
• Lots of GPU updates:
  • ~77 CUDA-related commits
  • ~66 AMD-related updates
  • ~27 OneAPI-related commits
  • 30 commits from AMD toolchain support
• spack test usage in packages is increasing
  • 1669 packages with tests (mostly generic python tests)
  • 93 packages with their own tests