Streaming data between a smart server and client

It would be nice if we could combine multiple containers into a single stream by something no more expensive than concatenation (e.g. by omitting end/start marker pairs).

This doesn’t imply that such a combination necessarily produces a valid container (e.g. care must be taken to ensure that names are still unique in the combined container), or even a useful container. It is simply that the cost of assembling a new combined container is practically as cheap as simple concatenation.

Persistent storage on disk

We want a storage format that allows lock-free writes, which suggests a format that uses rename into place, and do not modify after writing.

Usable before deep model changes to Bazaar

We want a format we can use and refine sooner rather than later. So it should be usable before the anticipated model changes for Bazaar “1.0” land, while not conflicting with those changes either.

Specifically, we’d like to have this format in Bazaar 0.18.

Examples of possible record content

• full texts of file versions
• deltas of full texts
• revisions
• inventories
• inventory as tree items e.g. the inventory data for 20 files
• revision signatures
• per-file graph data
• annotation cache

No length-prefixing of entire container

The overall container is not length-prefixed. Instead there is an end marker so that readers can determine when they have read the entire container. This also does not conflict with the goal of allowing single-pass writing.

Structured as a self-contained series of records

The container contains a series of records. Each record is self-delimiting. Record markers are lightweight. The overhead in terms of bytes and processing for records in this container vs. the raw contents of those records is minimal.

Addressing records

There is a requirement that each object can be given an arbitrary name. Some version control systems address all content by the SHA-1 digest of that content, but this scheme is unsatisfactory for Bazaar’s revision objects. We can still allow addressing by SHA-1 digest for those content types where it makes sense.

Some proposed object names:

• to name a revision: “revision:revision-id“. e.g., revision:[email protected].
• to name an inventory delta: “inventory.delta:revision-id“. e.g., inventory.delta:[email protected].

It seems likely that we may want to have multiple names for an object. This format allows that (by allowing multiple name headers in a Bytes record).

Although records are in principle addressable by name, this specification alone doesn’t provide for efficient access to a particular record given its name. It is intended that separate indexes will be maintained to provide this.

It is acceptable to have records with no explicit name, if the expected use of them does not require them. For example:

• a record’s content could be self-describing in the context of a particular container, or
• a record could be accessed via an index based on SHA-1, or
• when streaming, the first record could be treated specially.

Reasonably cheap for small records

The overhead for storing fairly short records (tens of bytes, rather than thousands or millions) is minimal. The minimum overhead is 3 bytes plus the length of the decimal representation of the length value (for a record with no name).

Record types

B26
example-name1
example-name2

abcdefghijklmnopqrstuvwxyz

Names

Names should be UTF-8 encoded strings, with no whitespace. Names should be unique within a single container, but no guarantee of uniqueness outside of the container is made by this layer. Names need to be at least one character long.