Patterns used

Note that these are able to be changed by changing what we store. For instance if the repository satisfies mpdiff requests, then bundle can be defined in terms of mpdiff lookups rather than file text lookups appropriate to create mpdiffs. If the repository satisfies full text requests only, then you need the topological access to build up the desired mpdiffs.

Indices

We want < linear access to all data in the repository. This suggests everything is indexed to some degree.

Often we know the kind of data we are accessing; which allows us to partition our indices if that will help (e.g. by reducing the total index size for queries that only care about the revision graph).

Indices that support our data access patterns will usually display increased locality of reference, reducing the impact of a large indices without needing careful page size management or other tricks.

We need repository wide indices. For the current repositories this is achieved by dividing the keyspace (revisions, signatures, inventories, per-fileid) and then having an append only index within each keyspace. For pack based repositories we will want some means to query the index of each component pack, presumably as a single logical index.

It would be nice if indexing was made cleanly separate from storage. So that suggests indices don’t know the meaning of the lookup; indices which offer particular ordering, or graph walking facilities will clearly need that information, but perhaps they don’t need to know the semantics ?

Data

Moving to pack based repositories

We have a number of challenges to solve.

Locks

Locks come in two flavours - read locks and write locks. Read locks allow data to be read from the repository. Write locks allow data to be read and signal that you intend to write data at some point. The actual writing of data must take place within a Write Group.

Write locks provide a cache of repository data during the period of the write lock, and allow write_groups to be acquired. For some repositories the presence of a write lock is exclusive to a single client, for others which are lock free or use server side locks (e.g. svn), the write lock simply provides the cache context.

Write Groups

Write groups are the only allowed means for inserting data into a repository. These are created by start_write_group, and concluded by either commit_write_group or abort_write_group. A write lock must be held on the repository for the entire duration. At most one write group can be active on a repository at a time.

Write groups signal to the repository the window during which data is actively being inserted. Several write groups could be committed during a single lock.

There is no guarantee that data inserted during a write group will be invisible in the repository if the write group is not committed. Specifically repositories without atomic insertion facilities will be writing data as it is inserted within the write group, and may not be able to revert that data - e.g. in the event of a dropped SFTP connection in a knit repository, inserted file data will be visible in the repository. Some repositories have an atomic insertion facility, and for those all-or-nothing will apply.

The precise meaning of a write group is format specific. For instance a knit based repository treats the write group methods as dummy calls, simply meeting the api that clients will use. A pack based repository will open a new pack container at the start of a write group, and rename it into place at commit time.