The multipack index is a list of (sha1, packid, offset) ordered by sha1. We can build it from the individual pack indexes (sha1, offset) ordered by sha1, with a simple merge ignoring duplicates.
This class encapsulates the merging logic and precalculates the stats that the index needs (like total count of objects). To limit memory consumption, it does the merge as it goes during the iteration and iterators use mutable entries. The stats of the combined index are calculated in an iteration at construction time.
| Modifier and Type | Class and Description |
|---|---|
| private static class | |
| pack-priv static class | PackIndexMerger.
Object returned by the iterator. |
| private static class | PackIndexMerger.
Iterator over n-indexes in ObjectId order. |
| pack-priv static class | PackIndexMerger.
Convenience around the PackIndex iterator to read the current value multiple times without consuming it. |
| Modifier and Type | Field and Description |
|---|---|
| private final List | |
| private static final int | |
| private static final long | |
| private final boolean | |
| private final int | |
| private final List | |
| private final int |
| Access | Constructor and Description |
|---|---|
| pack-priv |
| Modifier and Type | Method and Description |
|---|---|
| pack-priv Iterator | Returns: an iterator in sha1 order without duplicates.Iterator over the merged indexes in sha1 order without duplicates The returned entry in the iterator is mutable, callers should NOT keep a reference to it. |
| pack-priv int | Returns: number of objects with offsets over 2^31-1How many object have offsets over 2^31-1 Per multipack index spec, IF there is large offset chunk, all this offsets should be there. |
| pack-priv int | |
| pack-priv List | Returns: List of pack names, in the order used by the merge.List of pack names in alphabetical order. |
| pack-priv int | Returns: object count of the merged indexObject count of the merged index (i.e. without duplicates) |
| pack-priv boolean | Returns: true if there is any object with offset > 2^32 -1If any object in any of the indexes has an offset over 2^32-1 |
| pack-priv Iterator |
| indexes | back to summary |
|---|---|
| private final List<PackIndex> indexes | |
| LIMIT_31_BITS | back to summary |
|---|---|
| private static final int LIMIT_31_BITS | |
| LIMIT_32_BITS | back to summary |
|---|---|
| private static final long LIMIT_32_BITS | |
| needsLargeOffsetsChunk | back to summary |
|---|---|
| private final boolean needsLargeOffsetsChunk | |
| offsetsOver31BitsCount | back to summary |
|---|---|
| private final int offsetsOver31BitsCount | |
| packNames | back to summary |
|---|---|
| private final List<String> packNames | |
| uniqueObjectCount | back to summary |
|---|---|
| private final int uniqueObjectCount | |
| PackIndexMerger | back to summary |
|---|---|
| pack-priv PackIndexMerger(Map<String, PackIndex> packs) | |
| bySha1Iterator | back to summary |
|---|---|
| pack-priv Iterator Iterator over the merged indexes in sha1 order without duplicates The returned entry in the iterator is mutable, callers should NOT keep a reference to it.
| |
| getOffsetsOver31BitsCount | back to summary |
|---|---|
| pack-priv int getOffsetsOver31BitsCount() How many object have offsets over 2^31-1 Per multipack index spec, IF there is large offset chunk, all this offsets should be there.
| |
| getPackCount | back to summary |
|---|---|
| pack-priv int getPackCount() How many packs are being merged
| |
| getPackNames | back to summary |
|---|---|
| pack-priv List List of pack names in alphabetical order. Order matters: In case of duplicates, the multipack index prefers the first package with it. This is in the same order we are using to prioritize duplicates. | |
| getUniqueObjectCount | back to summary |
|---|---|
| pack-priv int getUniqueObjectCount() Object count of the merged index (i.e. without duplicates)
| |
| needsLargeOffsetsChunk | back to summary |
|---|---|
| pack-priv boolean needsLargeOffsetsChunk() If any object in any of the indexes has an offset over 2^32-1
| |
| rawIterator | back to summary |
|---|---|
| pack-priv Iterator For testing. Iterate all entries, not skipping duplicates (stable order)
| |
| Modifier and Type | Field and Description |
|---|---|
| private final MutableObjectId | |
| private int | |
| private final PackIndexMerger. |
| Access | Constructor and Description |
|---|---|
| pack-priv |
| Modifier and Type | Method and Description |
|---|---|
| public boolean | |
| public PackIndexMerger. |
| lastOid | back to summary |
|---|---|
| private final MutableObjectId lastOid | |
| remaining | back to summary |
|---|---|
| private int remaining | |
| src | back to summary |
|---|---|
| private final PackIndexMerger. | |
| DedupMultiIndexIterator | back to summary |
|---|---|
| pack-priv DedupMultiIndexIterator(PackIndexMerger. | |
| hasNext | back to summary |
|---|---|
| public boolean hasNext() Implements java. Doc from java. Returns
| |
| next | back to summary |
|---|---|
| public PackIndexMerger. Implements java. Doc from java. Returns the next element in the iteration.
| |
The iterator returns (on each next()) the same instance with different values, to avoid allocating many short-lived objects. Callers should not keep a reference to that returned value.
| Modifier and Type | Field and Description |
|---|---|
| private long | |
| private final MutableObjectId | |
| private int |
| Access | Constructor and Description |
|---|---|
| pack-priv |
| Modifier and Type | Method and Description |
|---|---|
| private void | fill(int
packId packId, PackIndex.another mutable entry otherCopy values from another mutable entry |
| public AnyObjectId | |
| public long | |
| public int |
| offset | back to summary |
|---|---|
| private long offset | |
| oid | back to summary |
|---|---|
| private final MutableObjectId oid | |
| packId | back to summary |
|---|---|
| private int packId | |
| MidxMutableEntry | back to summary |
|---|---|
| pack-priv MidxMutableEntry() | |
| fill | back to summary |
|---|---|
| private void fill(int packId, PackIndex. Copy values from another mutable entry
| |
| getObjectId | back to summary |
|---|---|
| public AnyObjectId getObjectId() | |
| getOffset | back to summary |
|---|---|
| public long getOffset() | |
| getPackId | back to summary |
|---|---|
| public int getPackId() | |
It returns duplicates if the same object id is in different indexes. Wrap
it with (Iterator, int) to avoid
duplicates.
| Modifier and Type | Field and Description |
|---|---|
| private final List | |
| private final PackIndexMerger. |
| Access | Constructor and Description |
|---|---|
| pack-priv |
| Modifier and Type | Method and Description |
|---|---|
| public boolean | |
| public PackIndexMerger. |
| indexIterators | back to summary |
|---|---|
| private final List<PackIndexMerger. | |
| mutableEntry | back to summary |
|---|---|
| private final PackIndexMerger. | |
| MultiIndexIterator | back to summary |
|---|---|
| pack-priv MultiIndexIterator(List<PackIndex> indexes) | |
| hasNext | back to summary |
|---|---|
| public boolean hasNext() Implements java. Doc from java. Returns
| |
| next | back to summary |
|---|---|
| public PackIndexMerger. Implements java. Doc from java. Returns the next element in the iteration.
| |
This is used to merge indexes in the multipack index, where we need to compare the current value between indexes multiple times to find the next.
We could also implement this keeping the position (int) and MutableEntry#getObjectId, but that would create an ObjectId per entry. This implementation reuses the MutableEntry and avoid instantiations.
| Modifier and Type | Field and Description |
|---|---|
| pack-priv PackIndex. | |
| private final Iterator | |
| private final int |
| Access | Constructor and Description |
|---|---|
| pack-priv |
| Modifier and Type | Method and Description |
|---|---|
| pack-priv int | |
| pack-priv PackIndex. | |
| pack-priv PackIndex. |
| current | back to summary |
|---|---|
| pack-priv PackIndex. | |
| it | back to summary |
|---|---|
| private final Iterator<PackIndex. | |
| packId | back to summary |
|---|---|
| private final int packId | |
| PackIndexPeekIterator | back to summary |
|---|---|
| pack-priv PackIndexPeekIterator(int packId, PackIndex index) | |
| getPackId | back to summary |
|---|---|
| pack-priv int getPackId() | |
| next | back to summary |
|---|---|
| pack-priv PackIndex. | |
| peek | back to summary |
|---|---|
| pack-priv PackIndex. | |