MEDIUM  (factory for protocol, reads bytes & pushes to protocol,
         uses protocol to detect end-of-request, sends written
         bytes to client) e.g. socket, pipe, HTTP request handler.
 ^
 | bytes.
 v

PROTOCOL(serialization, deserialization)  accepts bytes for one
         request, decodes according to internal state, pushes
         structured data to handler.  accepts structured data from
         handler and encodes and writes to the medium.  factory for
         handler.
 ^
 | structured data
 v

HANDLER  (domain logic) accepts structured data, operates state
         machine until the request can be satisfied,
         sends structured data to the protocol.

CLIENT   domain logic, accepts domain requests, generated structured
         data, reads structured data from responses and turns into
         domain data.  Sends structured data to the protocol.
         Operates state machines until the request can be delivered
         (e.g. reading from a bundle generated in bzrlib to deliver a
         complete request).

         This is RemoteBzrDir, RemoteRepository, etc.
 ^
 | structured data
 v

PROTOCOL  (serialization, deserialization)  accepts structured data for one
         request, encodes and writes to the medium.  Reads bytes from the
         medium, decodes and allows the client to read structured data.
 ^
 | bytes.
 v

MEDIUM   accepts bytes from the protocol & delivers to the remote server.
         Allows the protocol to read bytes e.g. socket, pipe, HTTP request.

REQUEST := MESSAGE_V1
RESPONSE := MESSAGE_V1
MESSAGE_V1 := ARGS [BODY]

ARGS := ARG [MORE_ARGS] NEWLINE
MORE_ARGS := SEP ARG [MORE_ARGS]
SEP := 0x01

BODY := LENGTH NEWLINE BODY_BYTES TRAILER
LENGTH := decimal integer
TRAILER := "done" NEWLINE

REQUEST_V2 := "bzr request 2" NEWLINE MESSAGE_V2

RESPONSE_V2 := "bzr response 2" NEWLINE RESPONSE_STATUS NEWLINE MESSAGE_V2
RESPONSE_STATUS := "success" | "failed"

FUTURE_MESSAGE := VERSION_STRING NEWLINE REST_OF_MESSAGE

MESSAGE_V2 := ARGS [BODY_V2]
BODY_V2 := BODY | STREAMED_BODY

STREAMED_BODY := "chunked" NEWLINE CHUNKS TERMINATOR
CHUNKS := CHUNK [CHUNKS]
CHUNK := HEX_LENGTH CHUNK_CONTENT
HEX_LENGTH := HEX_DIGITS NEWLINE
CHUNK_CONTENT := bytes

TERMINATOR := SUCCESS_TERMINATOR | ERROR_TERMINATOR
SUCCESS_TERMINATOR := 'END' NEWLINE
ERROR_TERMINATOR := 'ERR' NEWLINE CHUNKS SUCCESS_TERMINATOR

LENGTH_PREFIX := 32-bit unsigned integer in network byte order

MESSAGE := MAGIC NEWLINE HEADERS CONTENTS END_MESSAGE
MAGIC := "bzr message 3 (bzr 1.6)"
HEADERS := LENGTH_PREFIX bencoded_dict
END_MESSAGE := "e"

BODY := MESSAGE_PART+
MESSAGE_PART := ONE_BYTE | STRUCTURE | BYTES
ONE_BYTE := "o" byte
STRUCTURE := "s" LENGTH_PREFIX bencoded_structure
BYTES := "b" LENGTH_PREFIX bytes

Headers

Each request and response will have “headers”, a dictionary of key-value pairs. The keys must be strings, not any other type of value.

Currently, the only defined header is “Software version”. Both the client and the server should include a “Software version” header, with a value of a free-form string such as “bzrlib 1.5”, to aid debugging and logging. Clients and servers should not vary behaviour based on this string.

Conventional requests and responses

By convention, most requests and responses have a simple “arguments plus optional body” structure, as in earlier protocol versions. This section describes how such messages are encoded. All requests and responses defined by earlier protocol versions must be encoded in this way.

Conventional requests will send a CONTENTS of

CONV_REQ := ARGS SINGLE_OR_STREAMED_BODY?
SINGLE_OR_STREAMED_BODY := BYTES
      | BYTES+ TRAILER

ARGS := STRUCTURE(argument_tuple)
TRAILER := SUCCESS_STATUS | ERROR
SUCCESS_STATUS := ONE_BYTE("S")
ERROR := ONE_BYTE("E") STRUCTURE(argument_tuple)

Conventional responses will send CONTENTS of

CONV_RESP := RESP_STATUS ARGS SINGLE_OR_STREAMED_BODY?
RESP_STATUS := ONE_BYTE("S") | ONE_BYTE("E")

If the RESP_STATUS is success (“S”), the arguments are the method-dependent result.

For errors (where the Status byte of a response or a streamed body is “E”), the situation is analagous to requests. The first item in the encoded sequence must be a string of the error name. The other arguments supply details about the error, and their number and types will depend on the type of error (as identified by the error name).

Note that the streamed body from version two is now just multiple BYTES parts.

The end of the request or response is indicated by the lower-level END_MESSAGE. If there’s only one BYTES element in the body, the TRAILER may or may not be present, depending on whether it was sent as a single chunk or as a stream that happens to have one element.

(Discussion) The success marker at the end of a streamed body seems redundant; it doesn’t have space for any arguments, and the end of the body is marked anyhow by the end of the message. Recipients shouldn’t take any action on it, though they should map an error into raising an error locally.

1.10 clients don’t assert that they get a status byte at the end of the message. They will complain (in ConventionalResponseHandler.byte_part_received) if they get an initial success and then another byte part with no intervening bytes. If we stop sending the final success message and only flag errors they’ll only get one if the error is detected after streaming starts but before any bytes are actually sent. Possibly we should wait until at least the first chunk is ready before declaring success.

For new methods, these sequences are just a convention and may be varied if appropriate for a particular request or response. However, each request should at least start with a STRUCTURE encoding the arguments tuple. The first element of that tuple must be a string that names the request method. (Note that arguments in this protocol version are bencoded. As a result, unlike previous protocol versions, arguments in this version are 8-bit clean.)

(Discussion) We’re discussing having the byte segments be not just a method for sending a stream across the network, but actually having them be preserved in the rpc from end to end. This may be useful when there’s an iterator on one side feeding in to an iterator on the other, if it avoids doing chunking and byte-counting at two levels, and if those iterators are a natural place to get good granularity. Also, for cases like insert_record_stream the server can’t do much with the data until it gets a whole chunk, and so it’ll be natural and efficient for it to be called with one chunk at a time.

On the other hand, there may be times when we’ve got some bytes from the network but not a full chunk, and it might be worthwhile to pass it up. If we promise to preserve chunks, then to do this we’d need two separate streaming interfaces: “we got a chunk” and “we got some bytes but not yet a full chunk”. For insert_record_stream the second might not be useful, but it might be good when writing to a file where any number of bytes can be processed.

If we promise to preserve chunks, it’ll tend to make some RPCs work only in chunks, and others just on whole blocks, and we can’t so easily migrate RPCs from one to the other transparently to older implementations.

The data inside those chunks will be serialized anyhow, and possibly the data inside them will already be able to be serialized apart without understanding the chunks. Also, we might want to use these formats e.g. for pack files or in bundles, and so they don’t particularly need lower-level chunking. So the current (unmerged, unstable) record stream serialization turns each record into a bencoded tuple and it’d be feasible to parse one tuple at a time from a byte stream that contains a sequence of them.

So we’ve decided that the chunks won’t be semantic, and code should not count on them being preserved from client to server.

Early error returns

(Discussion) It would be nice if the server could notify the client of errors even before a streaming request has finished. This could cover situtaions such as the server not understanding the request, it being unable to open the requested location, or it finding that some of the revisions being sent are not actually needed.

Especially in the last case, we’d like to be able to gracefully notice the condition while the client is writing, and then have it adapt its behaviour. In any case, we don’t want to have drop and restart the network stream.

It should be possible for the client to finish its current chunk and then its message, possibly with an error to cancel what’s already been sent.

This relies on the client being able to read back from the server while it’s writing. This is technically difficult for http but feasible over a socket or ssh.

We’d need a clean way to pass this back to the request method, even though it’s presumably in the middle of doing its body iterator. Possibly the body iterator could be manually given a reference to the request object, and it can poll it to see if there’s a response.

Perhaps we need to distinguish error conditions, which should turn into a client-side error regardless of the request code, from early success, which should be handled only if the request code specifically wants to do it.

Full-duplex operation

Code not geared to do pipelined requests, and this might require doing asynchrony within bzrlib. We might want to either go fully pipelined and asynchronous, but there might be a profitable middle ground.

The particular case where duplex communication would be good is in working towards the common points in the graphs between the client and server: we want to send speculatively, but detect as soon as they’ve matched up.

So we could for instance have a synchronous core, but rely on the OS network buffering to allow us to work on batches of say 64kB. We can also pipeline requests and responses, without allowing for them happening out of order, or mixed requests happening at the same time.

Wonder how our network performance would have turned out now if we’d done full-duplex from the start, and ignored hpss over http. We have pretty good (readonly) http support just over dumb http, and that may be better for many users.