We're very pleased to present you restic 0.14.0! This is a great release which introduces a new repository version that adds the most requested feature: compression!

restic is distributed as a standalone binary: download the correct file for your operating system and architecture, extract the file and just run it. If you run into any issues, please report them at the GitHub issue tracker or visit the forum. If you already have restic >= 0.9.4, you can use restic self-update to get the latest version in a secure way.

The binaries released with each restic version are reproducible, which means that you can reproduce a byte identical version from the source code for that release. Instructions on how to do that in the Developer Documentation.

Changelog for restic 0.14.0 (2022-08-25)

The following sections list the changes in restic 0.14.0 relevant to restic users. The changes are ordered by importance.

Summary

• Fix #2248: Support self-update on Windows
• Fix #3428: List snapshots in backend at most once to resolve snapshot IDs
• Fix #3432: Fix rare 'not found in repository' error for copy command
• Fix #3685: The diff command incorrectly listed some files as added
• Fix #3681: Fix rclone (shimmed by Scoop) and sftp not working on Windows
• Fix #3720: Directory sync errors for repositories accessed via SMB
• Fix #3736: The stats command miscalculated restore size for multiple snapshots
• Fix #3861: Yield error on invalid policy to forget
• Fix #3716: Print "wrong password" to stderr instead of stdout
• Fix #3772: Correctly rebuild index for legacy repositories
• Fix #3776: Limit number of key files tested while opening a repository
• Chg #1842: Support debug log creation in release builds
• Chg #3295: Deprecate check --check-unused and add further checks
• Chg #3680: Update dependencies and require Go 1.15 or newer
• Chg #3742: Replace --repo2 option used by init/copy with --from-repo
• Enh #1153: Support pruning even when the disk is full
• Enh #21: Add compression support
• Enh #2162: Adaptive IO concurrency based on backend connections
• Enh #2291: Allow pack size customization
• Enh #2295: Allow use of SAS token to authenticate to Azure
• Enh #2696: Improve backup speed with many small files
• Enh #2907: Make snapshot directory structure of mount command customizable
• Enh #3114: Optimize handling of duplicate blobs in prune
• Enh #3465: Improve handling of temporary files on Windows
• Enh #3709: Validate exclude patterns before backing up
• Enh #3837: Improve SFTP repository initialization over slow links
• Enh #2351: Use config file permissions to control file group access
• Enh #3475: Allow limiting IO concurrency for local and SFTP backend
• Enh #3484: Stream data in check and prune commands
• Enh #2923: Improve speed of copy command
• Enh #3729: Display full IDs in check warnings
• Enh #3773: Optimize memory usage for directories with many files
• Enh #3819: Validate include/exclude patterns before restoring

Details

• Bugfix #2248: Support self-update on Windows

  Restic self-update would fail in situations where the operating system locks running binaries, including Windows. The new behavior works around this by renaming the running file and swapping the updated file in place.

  #2248 #3675

• Bugfix #3428: List snapshots in backend at most once to resolve snapshot IDs

  Many commands support specifying a list of snapshot IDs which are then used to determine the snapshots to be processed by the command. To resolve snapshot IDs or latest, and check that these exist, restic previously listed all snapshots stored in the repository. Depending on the backend this could be a slow and/or expensive operation.

  Restic now lists the snapshots only once and remembers the result in order to resolve all further snapshot IDs swiftly.

  #3428 #3570 #3395

• Bugfix #3432: Fix rare 'not found in repository' error for copy command

  In rare cases copy (and other commands) would report that LoadTree(...) returned an id [...] not found in repository error. This could be caused by a backup or copy command running concurrently. The error was only temporary; running the failed restic command a second time as a workaround did resolve the error.

  This issue has now been fixed by correcting the order in which restic reads data from the repository. It is now guaranteed that restic only loads snapshots for which all necessary data is already available.

  #3432 #3570

• Bugfix #3685: The diff command incorrectly listed some files as added

  There was a bug in the diff command, causing it to always show files in a removed directory as added. This has now been fixed.

  #3685 #3686

• Bugfix #3681: Fix rclone (shimmed by Scoop) and sftp not working on Windows

  In #3602 a fix was introduced to address the problem of rclone prematurely exiting when Ctrl+C is pressed on Windows. The solution was to create the subprocess with its console detached from the restic console.

  However, this solution failed when using rclone installed by Scoop or using sftp with a passphrase-protected private key. We've now fixed this by using a different approach to prevent Ctrl-C from passing down too early.

  #3681 #3692 #3696

• Bugfix #3720: Directory sync errors for repositories accessed via SMB

  On Linux and macOS, accessing a repository via a SMB/CIFS mount resulted in restic failing to save the lock file, yielding the following errors:

  Save(<lock/071fe833f0>) returned error, retrying after 552.330144ms: sync /repo/locks: no such file or directory Save(<lock/bf789d7343>) returned error, retrying after 552.330144ms: sync /repo/locks: invalid argument

  This has now been fixed by ignoring the relevant error codes.

  #3720 #3751 #3752

• Bugfix #3736: The stats command miscalculated restore size for multiple snapshots

  Since restic 0.10.0 the restore size calculated by the stats command for multiple snapshots was too low. The hardlink detection was accidentally applied across multiple snapshots and thus ignored many files. This has now been fixed.

  #3736 #3740

• Bugfix #3861: Yield error on invalid policy to forget

  The forget command previously silently ignored invalid/unsupported units in the duration options, such as e.g. --keep-within-daily 2w.

  Specifying an invalid/unsupported duration unit now results in an error.

  #3861 #3862

• Bugfix #3716: Print "wrong password" to stderr instead of stdout

  If an invalid password was entered, the error message was printed on stdout and not on stderr as intended. This has now been fixed.

  #3716 https://forum.restic.net/t/4965

• Bugfix #3772: Correctly rebuild index for legacy repositories

  After running rebuild-index on a legacy repository containing mixed pack files (that is, pack files which store both metadata and file data), check printed warnings like pack 12345678 contained in several indexes: .... This warning was not critical, but has now nonetheless been fixed by properly handling mixed pack files while rebuilding the index.

  Running prune for such legacy repositories will also fix the warning by reorganizing the pack files which caused it.

  #3772 #3884 https://forum.restic.net/t/5044/13

• Bugfix #3776: Limit number of key files tested while opening a repository

  Previously, restic tested the password against every key in the repository when opening a repository. The more keys there were in the repository, the slower this operation became.

  Restic now tests the password against up to 20 key files in the repository. Alternatively, you can use the --key-hint=<key ID> option to specify a specific key file to use instead.

  #3776

• Change #1842: Support debug log creation in release builds

  Creating a debug log was only possible in debug builds which required users to manually build restic. We changed the release builds to allow creating debug logs by simply setting the environment variable DEBUG_LOG=logname.log.

  #1842 #3826

• Change #3295: Deprecate check --check-unused and add further checks

  Since restic 0.12.0, it is expected to still have unused blobs after running prune. This made the --check-unused option of the check command rather useless and tended to confuse users. This option has been deprecated and is now ignored.

  The check command now also warns if a repository is using either the legacy S3 layout or mixed pack files with both tree and data blobs. The latter is known to cause performance problems.

  #3295 #3730

• Change #3680: Update dependencies and require Go 1.15 or newer

  We've updated most dependencies. Since some libraries require newer language features we're dropping support for Go 1.14, which means that restic now requires at least Go 1.15 to build.

  #3680 #3883

• Change #3742: Replace --repo2 option used by init/copy with --from-repo

  The init and copy commands can read data from another repository. However, confusingly --repo2 referred to the repository from which the init command copies parameters, but for the copy command --repo2 referred to the copy destination.

  We've introduced a new option, --from-repo, which always refers to the source repository for both commands. The old parameter names have been deprecated but still work. To create a new repository and copy all snapshots to it, the commands are now as follows:

  restic -r /srv/restic-repo-copy init --from-repo /srv/restic-repo --copy-chunker-params restic -r /srv/restic-repo-copy copy --from-repo /srv/restic-repo

  #3742 https://forum.restic.net/t/5017

• Enhancement #1153: Support pruning even when the disk is full

  When running out of disk space it was no longer possible to add or remove data from a repository. To help with recovering from such a deadlock, the prune command now supports an --unsafe-recover-no-free-space option to recover from these situations. Make sure to read the documentation first!

  #1153 #3481

• Enhancement #21: Add compression support

  We've added compression support to the restic repository format. To create a repository using the new format run init --repository-version 2. Please note that the repository cannot be read by restic versions prior to 0.14.0.

  You can configure whether data is compressed with the option --compression. It can be set to auto (the default, which will compress very fast), max (which will trade backup speed and CPU usage for better compression), or off (which disables compression). Each setting is only applied for the current run of restic and does not apply to future runs. The option can also be set via the environment variable RESTIC_COMPRESSION.

  To upgrade in place run migrate upgrade_repo_v2 followed by prune. See the documentation for more details. The migration checks the repository integrity and upgrades the repository format, but will not change any data. Afterwards, prune will rewrite the metadata to make use of compression.

  As an alternative you can use the copy command to migrate snapshots; First create a new repository using init --repository-version 2 --copy-chunker-params --repo2 path/to/old/repo, and then use the copy command to copy all snapshots to the new repository.

  #21 #3779 #3666 #3704 #3733

• Enhancement #2162: Adaptive IO concurrency based on backend connections

  Many commands used hard-coded limits for the number of concurrent operations. This prevented speed improvements by increasing the number of connections used by a backend.

  These limits have now been replaced by using the configured number of backend connections instead, which can be controlled using the -o <backend-name>.connections=5 option. Commands will then automatically scale their parallelism accordingly.

  To limit the number of CPU cores used by restic, you can set the environment variable GOMAXPROCS accordingly. For example to use a single CPU core, use GOMAXPROCS=1.

  #2162 #1467 #3611

• Enhancement #2291: Allow pack size customization

  Restic now uses a target pack size of 16 MiB by default. This can be customized using the --pack-size size option. Supported pack sizes range between 4 and 128 MiB.

  It is possible to migrate an existing repository to larger pack files using prune --repack-small. This will rewrite every pack file which is significantly smaller than the target size.

  #2291 #3731

• Enhancement #2295: Allow use of SAS token to authenticate to Azure

  Previously restic only supported AccountKeys to authenticate to Azure storage accounts, which necessitates giving a significant amount of access.

  We added support for Azure SAS tokens which are a more fine-grained and time-limited manner of granting access. Set the AZURE_ACCOUNT_NAME and AZURE_ACCOUNT_SAS environment variables to use a SAS token for authentication. Note that if AZURE_ACCOUNT_KEY is set, it will take precedence.

  #2295 #3661

• Enhancement #2696: Improve backup speed with many small files

  We have restructured the backup pipeline to continue reading files while all upload connections are busy. This allows the backup to already prepare the next data file such that the upload can continue as soon as a connection becomes available. This can especially improve the backup performance for high latency backends.

  The upload concurrency is now controlled using the -o <backend-name>.connections=5 option.

  #2696 #3489

• Enhancement #2907: Make snapshot directory structure of mount command customizable

  We've added the possibility to customize the snapshot directory structure of the mount command using templates passed to the --snapshot-template option. The formatting of snapshots' timestamps is now controlled using --time-template and supports subdirectories to for example group snapshots by year. Please see restic help mount for further details.

  Characters in tag names which are not allowed in a filename are replaced by underscores _. For example a tag foo/bar will result in a directory name of foo_bar.

  #2907 #2913 #3691

• Enhancement #3114: Optimize handling of duplicate blobs in prune

  Restic prune always used to repack all data files containing duplicate blobs. This effectively removed all duplicates during prune. However, as a consequence all these data files were repacked even if the unused repository space threshold could be reached with less work.

  This is now changed and prune works nice and fast even when there are lots of duplicate blobs.

  #3114 #3290

• Enhancement #3465: Improve handling of temporary files on Windows

  In some cases restic failed to delete temporary files, causing the current command to fail. This has now been fixed by ensuring that Windows automatically deletes the file. In addition, temporary files are only written to disk when necessary, reducing disk writes.

  #3465 #1551 #3610

• Enhancement #3709: Validate exclude patterns before backing up

  Exclude patterns provided via --exclude, --iexclude, --exclude-file or --iexclude-file previously weren't validated. As a consequence, invalid patterns resulted in files that were meant to be excluded being backed up.

  Restic now validates all patterns before running the backup and aborts with a fatal error if an invalid pattern is detected.

  #3709 #3734

• Enhancement #3837: Improve SFTP repository initialization over slow links

  The init command, when used on an SFTP backend, now sends multiple mkdir commands to the backend concurrently. This reduces the waiting times when creating a repository over a very slow connection.

  #3837 #3840

• Enhancement #2351: Use config file permissions to control file group access

  Previously files in a local/SFTP repository would always end up with very restrictive access permissions, allowing access only to the owner. This prevented a number of valid use-cases involving groups and ACLs.

  We now use the permissions of the config file in the repository to decide whether group access should be given to newly created repository files or not. We arrange for repository files to be created group readable exactly when the repository config file is group readable.

  To opt-in to group readable repositories, a simple chmod -R g+r or equivalent on the config file can be used. For repositories that should be writable by group members a tad more setup is required, see the docs.

  Posix ACLs can also be used now that the group permissions being forced to zero no longer masks the effect of ACL entries.

  #2351 #3419 https://forum.restic.net/t/1391

• Enhancement #3475: Allow limiting IO concurrency for local and SFTP backend

  Restic did not support limiting the IO concurrency / number of connections for accessing repositories stored using the local or SFTP backends. The number of connections is now limited as for other backends, and can be configured via the -o local.connections=2 and -o sftp.connections=5 options. This ensures that restic does not overwhelm the backend with concurrent IO operations.

  #3475

• Enhancement #3484: Stream data in check and prune commands

  The commands check --read-data and prune previously downloaded data files into temporary files which could end up being written to disk. This could cause a large amount of data being written to disk.

  The pack files are now instead streamed, which removes the need for temporary files. Please note that uploads during backup and prune still require temporary files.

  #3710 #3484 #3717

• Enhancement #2923: Improve speed of copy command

  The copy command could require a long time to copy snapshots for non-local backends. This has been improved to provide a throughput comparable to the restore command.

  Additionally, copy now displays a progress bar.

  #2923 #3513

• Enhancement #3729: Display full IDs in check warnings

  When running commands to inspect or repair a damaged repository, it is often necessary to supply the full IDs of objects stored in the repository.

  The output of check now includes full IDs instead of their shortened variant.

  #3729

• Enhancement #3773: Optimize memory usage for directories with many files

  Backing up a directory with hundreds of thousands or more files caused restic to require large amounts of memory. We've now optimized the backup command such that it requires up to 30% less memory.

  #3773

• Enhancement #3819: Validate include/exclude patterns before restoring

  Patterns provided to restore via --exclude, --iexclude, --include and --iinclude weren't validated before running the restore. Invalid patterns would result in error messages being printed repeatedly, and possibly unwanted files being restored.

  Restic now validates all patterns before running the restore, and aborts with a fatal error if an invalid pattern is detected.

  #3819