diff options
Diffstat (limited to 'bugs/rsync_as_transport__63__.mdwn')
| -rw-r--r-- | bugs/rsync_as_transport__63__.mdwn | 20 |
1 files changed, 0 insertions, 20 deletions
diff --git a/bugs/rsync_as_transport__63__.mdwn b/bugs/rsync_as_transport__63__.mdwn deleted file mode 100644 index abfea83..0000000 --- a/bugs/rsync_as_transport__63__.mdwn +++ /dev/null @@ -1,20 +0,0 @@ -SFTP is very slow because of the per-command, per-file latency. rsync, on the other hand, works much more quickly for transferring large numbers of small files. I don't know how the rsync protocol works, but I assume it basically batches files together into a stream, ala tar, and lets the rsync process on the receiving end split them up. - -I am considering using a local repo and rsyncing it to a remote server rather than directly backing up with SFTP. After running a few simple tests, it seems like it should work fine. Is this a good idea, assuming I have enough disk space for a local repo? - -Even better, could Obnam somehow use rsync as a transport instead of SFTP? Maybe it could write small files locally and then rsync them to the remote repo; and for reading it could rsync small files to a local directory. It could do it in batches based on total filesize, or, perhaps more usefully, based on total number of files (backend files, not source files). Doing it this way wouldn't require much disk space--a few megs, at most, I'd think--as opposed to keeping a local copy of the entire repo. - - ---- - -Backing up to a local repository and rsyncing that to the server will work just fine. After you've rsynced you can then either continue using the local repository and rsyncing it to the server for updates, or switch completely to using the remote one only. You can't mix the two approaches, however, since that will just make Obnam confused. - -As for using rsync as a transport I don't see that as a realistic thing at this time, -but it's an innovative idea. Thanks. - -Obnam does not know beforehand which files it will need from the repository, and it needs to be able to create files in the repository concurrently with other instances of Obnam, and these defeat any benefit from rsync at this time. However, a repository-side Obnam server process might be the way to go, but there's other approaches to be used as well, such as caching files locally in a way that is safe against concurrent use of repository by multiple clients. I have those ideas written down elsewhere, so I'll mark this bug report as done. - ---liw - -[[done]] - |