google/cdc-file-transfer
Fork: 85 Star: 2959 (更新于 2024-10-21 09:46:58)
license: Apache-2.0
Language: C++ .
Tools for synching and streaming files from Windows to Linux
最后发布版本: latest ( 2023-06-23 18:31:49)
CDC File Transfer
Born from the ashes of Stadia, this repository contains tools for syncing and streaming files from Windows to Windows or Linux. The tools are based on Content Defined Chunking (CDC), in particular FastCDC, to split up files into chunks.
History
At Stadia, game developers had access to Linux cloud instances to run games. Most developers wrote their games on Windows, though. Therefore, they needed a way to make them available on the remote Linux instance.
As developers had SSH access to those instances, they could use scp
to copy
the game content. However, this was impractical, especially with the shift to
working from home during the pandemic with sub-par internet connections. scp
always copies full files, there is no "delta mode" to copy only the things that
changed, it is slow for many small files, and there is no fast compression.
To help this situation, we developed two tools, cdc_rsync
and cdc_stream
,
which enable developers to quickly iterate on their games without repeatedly
incurring the cost of transmitting dozens of GBs.
CDC RSync
cdc_rsync
is a tool to sync files from a Windows machine to a Linux device,
similar to the standard Linux rsync. It is
basically a copy tool, but optimized for the case where there is already an old
version of the files available in the target directory.
- It quickly skips files if timestamp and file size match.
- It uses fast compression for all data transfer.
- If a file changed, it determines which parts changed and only transfers the differences.
The remote diffing algorithm is based on CDC. In our tests, it is up to 30x
faster than the one used in rsync
(1500 MB/s vs 50 MB/s).
The following chart shows a comparison of cdc_rsync
and Linux rsync
running
under Cygwin on Windows. The test data consists of 58 development builds
of some game provided to us for evaluation purposes. The builds are 40-45 GB
large. For this experiment, we uploaded the first build, then synced the second
build with each of the two tools and measured the time. For example, syncing
from build 1 to build 2 took 210 seconds with the Cygwin rsync
, but only 75
seconds with cdc_rsync
. The three outliers are probably feature drops from
another development branch, where the delta was much higher. Overall,
cdc_rsync
syncs files about 3 times faster than Cygwin rsync
.
We also ran the experiment with the native Linux rsync
, i.e syncing Linux to
Linux, to rule out issues with Cygwin. Linux rsync
performed on average 35%
worse than Cygwin rsync
, which can be attributed to CPU differences. We did
not include it in the figure because of this, but you can find it
here.
How does it work and why is it faster?
The standard Linux rsync
splits a file into fixed-size chunks of typically
several KB.
If the file is modified in the middle, e.g. by inserting xxxx
after 567
,
this usually means that the modified chunks as well as
all subsequent chunks change.
The standard rsync
algorithm hashes the chunks of the remote "old" file
and sends the hashes to the local device. The local device then figures out
which parts of the "new" file matches known chunks.
Standard rsync algorithm
This is a simplification. The actual algorithm is more complicated and uses
two hashes, a weak rolling hash and a strong hash, see
here for a great overview. What makes
rsync
relatively slow is the "no match" situation where the rolling hash does
not match any remote hash, and the algorithm has to roll the hash forward and
perform a hash map lookup for each byte. rsync
goes to
great lengths
optimizing lookups.
cdc_rsync
does not use fixed-size chunks, but instead variable-size,
content-defined chunks. That means, chunk boundaries are determined by the
local content of the file, in practice a 64 byte sliding window. For more
details, see
the FastCDC paper
or take a look at our implementation.
If the file is modified in the middle, only the modified chunks, but not subsequent chunks change (unless they are less than 64 bytes away from the modifications).
Computing the chunk boundaries is cheap and involves only a left-shift, a memory
lookup, an add
and an and
operation for each input byte. This is cheaper
than the hash map lookup for the standard rsync
algorithm.
Because of this, the cdc_rsync
algorithm is faster than the standard
rsync
. It is also simpler. Since chunk boundaries move along with insertions
or deletions, the task to match local and remote hashes is a trivial set
difference operation. It does not involve a per-byte hash map lookup.
cdc_rsync algorithm
CDC Stream
cdc_stream
is a tool to stream files and directories from a Windows machine to
a Linux device. Conceptually, it is similar to
sshfs, but it is optimized for read speed.
- It caches streamed data on the Linux device.
- If a file is re-read on Linux after it changed on Windows, only the differences are streamed again. The rest is read from the cache.
- Stat operations are very fast since the directory metadata (filenames, permissions etc.) is provided in a streaming-friendly way.
To efficiently determine which parts of a file changed, the tool uses the same
CDC-based diffing algorithm as cdc_rsync
. Changes to Windows files are almost
immediately reflected on Linux, with a delay of roughly (0.5s + 0.7s x total
size of changed files in GB).
The tool does not support writing files back from Linux to Windows; the Linux directory is readonly.
The following chart compares times from starting a game to reaching the menu.
In one case, the game is streamed via sshfs
, in the other case we use
cdc_stream
. Overall, we see a 2x to 5x speedup.
Supported Platforms
cdc_rsync |
From | To |
---|---|---|
Windows x86_64 | ✓ | ✓ 1 |
Ubuntu 22.04 x86_64 | ✗ 2 | ✓ |
Ubuntu 22.04 aarch64 | ✗ | ✗ |
macOS 13 x86_64 3 | ✗ | ✗ |
macOS 13 aarch64 3 | ✗ | ✗ |
cdc_stream |
From | To |
---|---|---|
Windows x86_64 | ✓ | ✗ |
Ubuntu 22.04 x86_64 | ✗ | ✓ |
Ubuntu 22.04 aarch64 | ✗ | ✗ |
macOS 13 x86_64 3 | ✗ | ✗ |
macOS 13 aarch64 3 | ✗ | ✗ |
1 Only local syncs, e.g. cdc_rsync C:\src\* C:\dst
. Support for
remote syncs is being added, see
#61.
2 See #56.
3 See #62.
Getting Started
Download the precompiled binaries from the
latest release to a
Windows device and unzip them. The Linux binaries are automatically deployed
to ~/.cache/cdc-file-transfer
by the Windows tools. There is no need to manually
deploy them. We currently provide Linux binaries compiled on
Github's latest Ubuntu version.
If the binaries work for you, you can skip the following two sections.
Alternatively, the project can be built from source. Some binaries have to be built on Windows, some on Linux.
Prerequisites for Building
To build the tools from source, the following steps have to be executed on both Windows and Linux.
- Download and install Bazel from here. See workflow logs for the currently used version.
- Clone the repository.
git clone https://github.com/google/cdc-file-transfer
- Initialize submodules.
cd cdc-file-transfer git submodule update --init --recursive
Finally, install an SSH client on the Windows machine if not present.
The file transfer tools require ssh.exe
and sftp.exe
.
Building
The two tools CDC RSync and CDC Stream can be built and used independently.
CDC RSync
- On a Linux device, build the Linux components
bazel build --config linux --compilation_mode=opt --linkopt=-Wl,--strip-all --copt=-fdata-sections --copt=-ffunction-sections --linkopt=-Wl,--gc-sections //cdc_rsync_server
- On a Windows device, build the Windows components
bazel build --config windows --compilation_mode=opt --copt=/GL //cdc_rsync
- Copy the Linux build output file
cdc_rsync_server
frombazel-bin/cdc_rsync_server
tobazel-bin\cdc_rsync
on the Windows machine.
CDC Stream
- On a Linux device, build the Linux components
bazel build --config linux --compilation_mode=opt --linkopt=-Wl,--strip-all --copt=-fdata-sections --copt=-ffunction-sections --linkopt=-Wl,--gc-sections //cdc_fuse_fs
- On a Windows device, build the Windows components
bazel build --config windows --compilation_mode=opt --copt=/GL //cdc_stream
- Copy the Linux build output files
cdc_fuse_fs
andlibfuse.so
frombazel-bin/cdc_fuse_fs
tobazel-bin\cdc_stream
on the Windows machine.
Usage
The tools require a setup where you can use SSH and SFTP from the Windows machine to the Linux device without entering a password, e.g. by using key-based authentication.
Configuring SSH and SFTP
By default, the tools search ssh.exe
and sftp.exe
from the path environment
variable. If you can run the following commands in a Windows cmd without
entering your password, you are all set:
ssh user@linux.device.com
sftp user@linux.device.com
Here, user
is the Linux user and linux.device.com
is the Linux host to
SSH into or copy the file to.
If additional arguments are required, it is recommended to provide an SSH config
file. By default, both ssh.exe
and sftp.exe
use the file at
%USERPROFILE%\.ssh\config
on Windows, if it exists. A possible config file
that sets a username, a port, an identity file and a known host file could look
as follows:
Host linux_device
HostName linux.device.com
User user
Port 12345
IdentityFile C:\path\to\id_rsa
UserKnownHostsFile C:\path\to\known_hosts
If ssh.exe
or sftp.exe
cannot be found, you can specify the full paths via
the command line arguments --ssh-command
and --sftp-command
for cdc_rsync
and cdc_stream start
(see below), or set the environment variables
CDC_SSH_COMMAND
and CDC_SFTP_COMMAND
, e.g.
set CDC_SSH_COMMAND="C:\path with space\to\ssh.exe"
set CDC_SFTP_COMMAND="C:\path with space\to\sftp.exe"
Note that you can also specify SSH configuration via the environment variables instead of using a config file:
set CDC_SSH_COMMAND=C:\path\to\ssh.exe -p 12345 -i C:\path\to\id_rsa -oUserKnownHostsFile=C:\path\to\known_hosts
set CDC_SFTP_COMMAND=C:\path\to\sftp.exe -P 12345 -i C:\path\to\id_rsa -oUserKnownHostsFile=C:\path\to\known_hosts
Note the small -p
for ssh.exe
and the capital -P
for sftp.exe
.
Google Specific
For Google internal usage, set the following environment variables to enable SSH authentication using a Google security key:
set CDC_SSH_COMMAND=C:\gnubby\bin\ssh.exe
set CDC_SFTP_COMMAND=C:\gnubby\bin\sftp.exe
Note that you will have to touch the security key multiple times during the first run. Subsequent runs only require a single touch.
CDC RSync
cdc_rsync
is used similar to scp
or the Linux rsync
command. To sync a
single Windows file C:\path\to\file.txt
to the home directory ~
on the Linux
device linux.device.com
, run
cdc_rsync C:\path\to\file.txt user@linux.device.com:~
cdc_rsync
understands the usual Windows wildcards *
and ?
.
cdc_rsync C:\path\to\*.txt user@linux.device.com:~
To sync the contents of the Windows directory C:\path\to\assets
recursively to
~/assets
on the Linux device, run
cdc_rsync C:\path\to\assets\* user@linux.device.com:~/assets -r
To get per file progress, add -v
:
cdc_rsync C:\path\to\assets\* user@linux.device.com:~/assets -vr
The tool also supports local syncs:
cdc_rsync C:\path\to\assets\* C:\path\to\destination -vr
CDC Stream
To stream the Windows directory C:\path\to\assets
to ~/assets
on the Linux
device, run
cdc_stream start C:\path\to\assets user@linux.device.com:~/assets
This makes all files and directories in C:\path\to\assets
available on
~/assets
immediately, as if it were a local copy. However, data is streamed
from Windows to Linux as files are accessed.
To stop the streaming session, enter
cdc_stream stop user@linux.device.com:~/assets
The command also accepts wildcards. For instance,
cdc_stream stop user@*:*
stops all existing streaming sessions for the given user.
Troubleshooting
On first run, cdc_stream
starts a background service, which does all the work.
The cdc_stream start
and cdc_stream stop
commands are just RPC clients that
talk to the service.
The service logs to %APPDATA%\cdc-file-transfer\logs
by default. The logs are
useful to investigate issues with asset streaming. To pass custom arguments, or
to debug the service, create a JSON config file at
%APPDATA%\cdc-file-transfer\cdc_stream.json
with command line flags.
For instance,
{ "verbosity":3 }
instructs the service to log debug messages. Try cdc_stream start-service -h
for a list of available flags. Alternatively, run the service manually with
cdc_stream start-service
and pass the flags as command line arguments. When you run the service manually,
the flag --log-to-stdout
is particularly useful as it logs to the console
instead of to the file.
cdc_rsync
always logs to the console. To increase log verbosity, pass -vvv
for debug logs or -vvvv
for verbose logs.
For both sync and stream, the debug logs contain all SSH and SFTP commands that
are attempted to run, which is very useful for troubleshooting. If a command
fails unexpectedly, copy it and run it in isolation. Pass -vv
or -vvv
for
additional debug output.
最近版本更新:(数据更新于 2024-10-04 23:44:10)
2023-06-23 18:31:49 latest
2023-05-28 04:14:42 latest
2023-04-09 03:22:59 latest
2023-01-10 20:36:40 latest
2022-12-21 19:37:55 v1.0.0
google/cdc-file-transfer同语言 C++最近更新仓库
2024-11-05 00:53:40 PCSX2/pcsx2
2024-11-05 00:42:13 ClickHouse/ClickHouse
2024-11-04 21:30:40 LizardByte/Sunshine
2024-11-03 22:31:09 MaaAssistantArknights/MaaAssistantArknights
2024-11-02 20:28:28 AaronFeng753/Waifu2x-Extension-GUI
2024-10-30 00:17:48 zerotier/ZeroTierOne