Fetch operations

We fetch data between repositories as part of push/pull/branch operations. Fetching data is currently an very interactive process with lots of requests. For performance having the data be supplied in a stream will improve push and pull to remote servers. For purely local operations the streaming logic should help reduce memory pressure. In fetch operations we always know the formats of both the source and target.

Bundles

Bundles also create a stream of data for revisions from a repository. Unlike fetch operations we do not know the format of the target at the time the stream is created. It would be good to be able to treat bundles as frozen branches and repositories, so a serialised stream should be suitable for this.

Data conversion

At this point we are not trying to integrate data conversion into this interface, though it is likely possible.

Single round trip

All users of this should be able to create an appropriate stream from a single round trip.

Forward-only reads

There should be no need to seek in a stream when inserting data from it into a repository. This places an ordering constraint on streams which some repositories do not need.

Weaves

While there shouldn’t be too many users of weave repositories anymore, avoiding pathological behaviour when a weave is being read is a good idea. Having the weave itself embedded in the stream is very straight forward and does not need expensive on the fly extraction and re-diffing to take place.

Bundles

Being able to perform random reads from a repository stream which is a bundle would allow stacking a bundle and a real repository together. This will need the pack container format to be used in such a way that we can avoid reading more data than needed within the pack container’s readv interface.

Requesting a stream

To request a stream, three parameters are needed:

• A revision search to select the revisions to include.
• A data ordering flag. There are two values for this - ‘unordered’ and ‘topological’. ‘unordered’ streams are useful when inserting into repositories that have the ability to perform atomic insertions. ‘topological’ streams are useful when converting data, or when inserting into repositories that cannot perform atomic insertions (such as knit or weave based repositories).
• A complete_inventory flag. When provided this flag signals the stream generator to include all the data needed to construct the inventory of each revision included in the stream, rather than just deltas. This is useful when converting data from a repository with a different inventory serialisation, as pure deltas would not be able to be reconstructed.

Structure of a stream

A stream is an object. It can be consistency checked via the check method (which consumes the stream). The iter_contents method can be used to iterate the contents of the stream. The contents of the stream are a series of top level records, each of which contains one or more bytestrings (potentially as a delta against another item in the repository) and some optional metadata.

Consuming a stream

To consume a stream, obtain an iterator from the streams iter_contents method. This iterator will yield the top level records. Each record has two attributes. One is key_prefix which is a tuple key prefix for the names of each of the bytestrings in the record. The other attribute is entries, an iterator of the individual items in the record. Each item that the iterator yields is a factory which has metadata about the entry and the ability to return the compressed bytes. This factory can be decorated to allow obtaining different representations (for example from a compressed knit fulltext to a plain fulltext).

In pseudocode:

stream = repository.get_repository_stream(search, UNORDERED, False)
for record in stream.iter_contents():
    for factory in record.entries:
        compression = factory.storage_kind
        print "Object %s, compression type %s, %d bytes long." % (
            record.key_prefix + factory.key,
            compression, len(factory.get_bytes_as(compression)))

This structure should allow stream adapters to be written which can coerce all records to the type of compression that a particular client needs. For instance, inserting into weaves requires fulltexts, so a stream would be adapted for weaves by an adapter that takes a stream, and the target weave, and then uses the target weave to reconstruct full texts (which is all that the weave inserter would ask for). In a similar approach, a stream could internally delta compress many fulltexts and be able to answer both fulltext and compressed record requests without extra IO.