FPSYNC(1) BSD General Commands Manual FPSYNC(1)NAMEfpsync — Synchronize directories in parallel using fpart and rsync.
SYNOPSISfpsync [-h] [-v] [-n jobs] [-f files] [-s size] [-w wrks] [-d shdir]
[-t tmpdir] [-r jobname] [-o rsyncopts] [-O fpartopts] [-S]
src_dir/ dst_dir/
DESCRIPTION
The fpsync tool synchronizes directories in parallel using fpart(1) and
rsync(1). It computes subsets of src_dir/ and spawns rsync(1) jobs to
synchronize them to dst_dir/.
Synchronization jobs can be executed either locally or remotely (using
SSH workers, see option -w) and are executed on-the-fly while filesystem
crawling goes on. This makes fpsync a good tool for migrating large
filesystems.
OPTIONS-h Print help
-v Verbose mode. Can be be specified several times to increase ver‐
bosity level.
-n jobs
Start jobs concurrent sync jobs (either locally or remotely, see
below). Default: 2
-f files
Transfer at most files files per sync job. Default: 2000
-s size
Transfer at most size bytes per sync job.
Default: 4294967296 (4 GB)
-w wrks
Use remote SSH wrks to synchronize files. Synchronization jobs
are executed locally when this option is not set. wrks is a
space-separated list of login@machine connection strings and can
be specified several times. You must be allowed to connect to
those machines using a SSH key to avoid user interaction.
-d shdir
Set fpsync shared directory to shdir. This option is mandatory
when using SSH workers and set by default to tmpdir when running
locally. The specified directory must be an absolute path ; it
will be used to handle communications with SSH hosts (sharing
partitions and log files) and, as a consequence, must be made
available to all participating hosts (e.g. through a r/w NFS
mount), including the master one running fpsync.
-t tmpdir
Set fpsync temporary directory to tmpdir. This directory remains
local and does not need to be shared amongst SSH workers when
using the -w option. Default: /tmp/fpsync
-r jobname
Resume job jobname and restart synchronizing remaining partitions
from a previous run. jobname can be obtained using verbose mode
(see option -v). Note that filesystem crawling is skipped when
resuming a previous run. As a consequence, options -f, -s, -o,
-O, -S, src_dir/, and dst_dir/ are ignored.
-o rsyncopts
Override default rsync(1) options with rsyncopts. Use this
option with care as certain options are incompatible with a par‐
allel usage (e.g. --delete). Default: -av --numeric-ids
-O fpartopts
Override default fpart(1) options with fpartopts.
Default: -x .zfs -x .snapshot* -x .ckpt
-S Sudo mode. Use sudo(8) for filesystem crawling and synchroniza‐
tions.
src_dir/
Source directory. It must be absolute and available on all par‐
ticipating hosts (including the master one, running fpsync).
dst_dir/
Destination directory. It must be absolute and available on all
participating workers.
RUNNING FPSYNC
Each fpsync run generates a unique jobname, which is displayed in verbose
mode (see option -v) and within log files. You can use that jobname to
resume a previous run (see option -r). fpsync will then restart synchro‐
nizing data from the parts that were being synchonized at the time it
stopped.
This unique feature gives the administrator the ability to stop fpsync
and restart it later, without having to restart the whole filesystem
crawling and synchronization process. Note that resuming is only possible
when filesystem crawling step has finished.
During synchronization, you can press CTRL-C to interrupt the process.
The first CTRL-C prevents new synchronizations from being submitted and
the process will wait for current synchronizations to be finished before
exiting. If you press CTRL-C again, current synchronizations will be
killed and fpsync will exit immediately.
On certain systems, CTRL-T can be pressed to get the status of current
and remaining parts to be synchronized. This can also be achieved by
sending a SIGINFO to the fpsync process.
Whether you use verbose mode or not, everything is logged within
shdir/log/.
EXAMPLES
Here are some examples:
fpsync-n 4 /usr/src/ /var/src/
Synchronizes /usr/src/ to /var/src/ using 4 local jobs.
fpsync-n 2 -w login@machine1 -w login@machine2 -d /mnt/fpsync /mnt/src/
/mnt/dst/
Synchronizes /mnt/src/ to /mnt/dst/ using 2 concurrent jobs exe‐
cuted remotely on 2 SSH workers (machine1 and machine2). The
shared directory is set to /mnt/fpsync and mounted on the machine
running fpsync, as well as on machine1 and machine2. The source
directory (/mnt/src/) is also available on those 3 machines,
while the destination directory (/mnt/dst/) is mounted on SSH
workers only (machine1 and machine2).
LIMITS
Parallelizing rsync(1) makes several options not usable, such as
--delete. If your source directory is live while fpsync is running, you
will have to delete extra files from destination directory. This is usu‐
ally done by using a final -offline-rsync(1) pass that will use this
option.
fpsync enqueues synchronization jobs on disk, within the tmpdir/queue
directory. Be careful to host this queue on a filesystem that can handle
fine-grained mtime timestamps (i.e. with a sub-second precision) if you
want the queue to be processed in order when fpart(1) generates several
jobs per second. On FreeBSD, VFS(9) timestamps' precision can be tuned
using the 'vfs.timestamp_precision' sysctl. See vfs_timestamp(9).
SEE ALSOfpart(1), rsync(1), sudo(8)AUTHOR, AVAILABILITY
Fpsync has been written by Ganaël LAPLANCHE and is available under the
BSD license on
http://contribs.martymac.org
BUGS
No bug known (yet).
BSD January 27, 2015 BSD
