summaryrefslogtreecommitdiff
path: root/README
blob: 44ea171456a7ed2ad879b85bd6096be2ef9d4ba4 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
A proof of concept for transferring sparse files
================================================

In Baserock, we often need to transfer sparse files across the
network. At the moment we do that by ignoring that they're sparse:
just read the holes as long sequences of zero bytes. That works, but
is inefficient.

This directory contains a couple of scripts: xfer-hole.py and
recv-hole that transfer holes more efficiently. They use the
`SEEK_DATA` and `SEEK_HOLE` options to the `lseek` system call.
xfer-hole.py produces an encoded form of the input file, where holes
are encoded in an efficient way. recv-hole decodes that and reproduces
the original file. The nifty trick is that recv-hole is a pure shell
script, and so will hopefully work on most target systems, instead of
requiring a custom program there.

This is a proof of concept. It is not ready for production use.
I wrote this to prove it's possible.