long.
Implementation of the PGM-Index described at https://pgm.di.unipi.it/, based on the paper
Paolo Ferragina and Giorgio Vinciguerra. The PGM-index: a fully-dynamic compressed learned index with provable worst-case bounds. PVLDB, 13(8): 1162-1175, 2020.It provides
rank and range search operations. indexOf() is faster than
B+Tree, and the index is much more compact. contains() is between 4x to 7x slower than
IntHashSet#contains(), but between 2.5x to 3x faster than Arrays#binarySearch.
Its compactness (40KB for 200MB of keys) makes it efficient for very large collections, the
index fitting easily in the L2 cache. The epsilon parameter should be set according to
the desired space-time trade-off. A smaller value makes the estimation more precise and the range
smaller but at the cost of increased space usage. In practice, epsilon 64 is a good sweet
spot.
Internally the index uses an optimal piecewise linear mapping from keys to their position in the sorted order. This mapping is represented as a sequence of linear models (segments) which are themselves recursively indexed by other piecewise linear mappings.
| Modifier and Type | Class and Description |
|---|---|
| public static class | LongPgmIndex.
Builds a |
| private static class | LongPgmIndex.
Empty immutable PGM Index. |
| protected static class | LongPgmIndex.
Iterator over a range of elements in a sorted array. |
| Modifier and Type | Field and Description |
|---|---|
| public static final int | BEYOND_EPSILON_JUMP
Initial value of the exponential jump when scanning out of the epsilon range. |
| public static final int | |
| public static final LongPgmIndex | EMPTY
Empty immutable LongPgmIndex. |
| public static final int | EPSILON
Epsilon approximation range when searching the list of keys. |
| public final int | epsilon
The epsilon range used to build this index. |
| public static final int | EPSILON_RECURSIVE
Epsilon approximation range for the segments layers. |
| public final int | epsilonRecursive
The recursive epsilon range used to build this index. |
| public final long | |
| public static final int | KEY_SIZE
Size of a key, measured in |
| public final LongArrayList | keys
The list of keys for which this index is built. |
| public final long | |
| public final int[] | levelOffsets
The offsets in |
| public static final int | SEGMENT_DATA_SIZE
Data size of a segment, measured in |
| public final int[] | segmentData
The index data. |
| public final int | size
The size of the key set. |
| Access | Constructor and Description |
|---|---|
| private | LongPgmIndex(LongArrayList keys, int size, int epsilon, int epsilonRecursive, int[] levelOffsets, int[] segmentData)
|
| private |
| Modifier and Type | Method and Description |
|---|---|
| private int | |
| public boolean | |
| private int | Returns: the segment data index; or -1 if none.Finds the segment responsible for a given key, that is, the rightmost segment having its first key <= the searched key. |
| public <T extends LongProcedure> T | forEachInRange(T procedure, long minKey, long maxKey)
Applies |
| private static long | |
| private long | |
| private static double | |
| public int | Returns: The index of the searched key if it is present; otherwise,(-(<i>insertion
point</i>) - 1). The insertion point is defined as the point at which the key would
be inserted into the list: the index of the first element greater than the key, or keys#size() if all the elements are less than the specified key. Note that this
guarantees that the return value will be >= 0 if and only if the key is found.Searches the specified key, and returns its index in the element list. |
| public boolean | |
| public long | ramBytesAllocated()
Implements com. Estimates the allocated memory. |
| public long | ramBytesUsed()
Implements com. Estimates the bytes that are actually used. |
| public int | rangeCardinality(long minKey, long maxKey)
Returns the number of keys in the list that are greater than or equal to |
| public Iterator | rangeIterator(long minKey, long maxKey)
Returns an iterator over the keys in the list that are greater than or equal to |
| public int | Returns: The index of the searched key if it is present; otherwise, theinsertion point.
The insertion point is defined as the point at which the key would be inserted into
the list: the index of the first element greater than the key, or keys#size() if all the elements are less than the specified key. Note that this method always
returns a value >= 0.Returns, for any value |
| public int |
| BEYOND_EPSILON_JUMP | back to summary |
|---|---|
| public static final int BEYOND_EPSILON_JUMP Initial value of the exponential jump when scanning out of the epsilon range. | |
| DOUBLE_KEY_SIZE | back to summary |
|---|---|
| public static final int DOUBLE_KEY_SIZE 2x | |
| EMPTY | back to summary |
|---|---|
| public static final LongPgmIndex EMPTY Empty immutable LongPgmIndex. | |
| EPSILON | back to summary |
|---|---|
| public static final int EPSILON Epsilon approximation range when searching the list of keys. Controls the size of the returned search range, strictly greater than 0. It should be set according to the desired space-time trade-off. A smaller value makes the estimation more precise and the range smaller but at the cost of increased space usage. With EPSILON=64 the benchmark with 200MB of keys shows that this PGM index requires only 2% additional memory on average (40KB). It depends on the distribution of the keys. This epsilon value is good even for 2MB of keys. With EPSILON=32: +5% speed, but 4x space (160KB). | |
| epsilon | back to summary |
|---|---|
| public final int epsilon The epsilon range used to build this index. | |
| EPSILON_RECURSIVE | back to summary |
|---|---|
| public static final int EPSILON_RECURSIVE Epsilon approximation range for the segments layers. Controls the size of the search range in the hierarchical segment lists, strictly greater than 0. | |
| epsilonRecursive | back to summary |
|---|---|
| public final int epsilonRecursive The recursive epsilon range used to build this index. | |
| firstKey | back to summary |
|---|---|
| public final long firstKey The lowest key in | |
| KEY_SIZE | back to summary |
|---|---|
| public static final int KEY_SIZE Size of a key, measured in | |
| keys | back to summary |
|---|---|
| public final LongArrayList keys The list of keys for which this index is built. It is sorted and may contain duplicate elements. | |
| lastKey | back to summary |
|---|---|
| public final long lastKey The highest key in | |
| levelOffsets | back to summary |
|---|---|
| public final int[] levelOffsets The offsets in | |
| SEGMENT_DATA_SIZE | back to summary |
|---|---|
| public static final int SEGMENT_DATA_SIZE Data size of a segment, measured in | |
| segmentData | back to summary |
|---|---|
| public final int[] segmentData The index data. It contains all the segments for all the levels. | |
| size | back to summary |
|---|---|
| public final int size The size of the key set. That is, the number of distinct elements in | |
| LongPgmIndex | back to summary |
|---|---|
| private LongPgmIndex(LongArrayList keys, int size, int epsilon, int epsilonRecursive, int[] levelOffsets, int[] segmentData) | |
| LongPgmIndex | back to summary |
|---|---|
| private LongPgmIndex() Empty set constructor. | |
| approximateIndex | back to summary |
|---|---|
| private int approximateIndex(long key, int segmentDataIndex, int[] segmentData) | |
| contains | back to summary |
|---|---|
| public boolean contains(long key) Returns whether this key set contains the given key. | |
| findSegment | back to summary |
|---|---|
| private int findSegment(long key) Finds the segment responsible for a given key, that is, the rightmost segment having its first key <= the searched key.
| |
| forEachInRange | back to summary |
|---|---|
| public <T extends LongProcedure> T forEachInRange(T procedure, long minKey, long maxKey) Applies | |
| getIntercept | back to summary |
|---|---|
| private static long getIntercept(int segmentDataIndex, int[] segmentData) | |
| getKey | back to summary |
|---|---|
| private long getKey(int segmentDataIndex, int[] segmentData) | |
| getSlope | back to summary |
|---|---|
| private static double getSlope(int segmentDataIndex, int[] segmentData) | |
| indexOf | back to summary |
|---|---|
| public int indexOf(long key) Searches the specified key, and returns its index in the element list. If multiple elements are equal to the specified key, there is no guarantee which one will be found.
| |
| isEmpty | back to summary |
|---|---|
| public boolean isEmpty() Returns whether this key set is empty. | |
| ramBytesAllocated | back to summary |
|---|---|
| public long ramBytesAllocated() Implements com. Estimates the allocated memory. It does not count the memory for the list of keys, only for the index itself.
| |
| ramBytesUsed | back to summary |
|---|---|
| public long ramBytesUsed() Implements com. Estimates the bytes that are actually used. It does not count the memory for the list of keys, only for the index itself.
| |
| rangeCardinality | back to summary |
|---|---|
| public int rangeCardinality(long minKey, long maxKey) Returns the number of keys in the list that are greater than or equal to | |
| rangeIterator | back to summary |
|---|---|
| public Iterator Returns an iterator over the keys in the list that are greater than or equal to | |
| rank | back to summary |
|---|---|
| public int rank(long x) Returns, for any value If multiple elements are equal to the specified key, there is no guarantee which one will be found.
| |
| size | back to summary |
|---|---|
| public int size() Returns the size of the key set. That is, the number of distinct elements in | |
LongPgmIndex on a provided sorted list of keys.
| Modifier and Type | Field and Description |
|---|---|
| protected int | |
| protected int | |
| protected LongArrayList | |
| protected int | |
| protected PlaModel | |
| protected IntArrayList | |
| protected int |
| Access | Constructor and Description |
|---|---|
| public |
| Modifier and Type | Method and Description |
|---|---|
| public void | accept(double
The first key of the segment. firstKey, double The segment slope. slope, long The segment intercept. intercept)Implements com. Consumes a new segment. |
| private void | addSentinelSegment(int endIndex)
Adds a sentinel segment that is used to give a limit for the position approximation, but does not count in the number of segments per level. |
| public LongPgmIndex | |
| private int | |
| private int | |
| private long | |
| public long | ramBytesAllocated()
Implements com. Estimates the allocated memory. |
| public long | ramBytesUsed()
Implements com. Estimates the bytes that are actually used. |
| public LongPgmIndex. | setEpsilon(int epsilon)
Sets the epsilon range to use when learning the segments for the list of keys. |
| public LongPgmIndex. | setEpsilonRecursive(int epsilonRecursive)
Sets the recursive epsilon range to use when learning the segments for the segment levels. |
| public LongPgmIndex. | setSortedKeys(LongArrayList keys)
Sets the sorted list of keys to build the index for; duplicate elements are allowed. |
| public LongPgmIndex. | setSortedKeys(long[] keys, int length)
Sets the sorted array of keys to build the index for; duplicate elements are allowed. |
| epsilon | back to summary |
|---|---|
| protected int epsilon | |
| epsilonRecursive | back to summary |
|---|---|
| protected int epsilonRecursive | |
| keys | back to summary |
|---|---|
| protected LongArrayList keys | |
| numSegments | back to summary |
|---|---|
| protected int numSegments | |
| plam | back to summary |
|---|---|
| protected PlaModel plam | |
| segmentData | back to summary |
|---|---|
| protected IntArrayList segmentData | |
| size | back to summary |
|---|---|
| protected int size | |
| LongBuilder | back to summary |
|---|---|
| public LongBuilder() | |
| accept | back to summary |
|---|---|
| public void accept(double firstKey, double slope, long intercept) Implements com. Doc from com. Consumes a new segment. The segment is defined by the epsilon-approximation function fs(k) = k × slope + intercept.
| |
| addSentinelSegment | back to summary |
|---|---|
| private void addSentinelSegment(int endIndex) Adds a sentinel segment that is used to give a limit for the position approximation, but does not count in the number of segments per level. | |
| build | back to summary |
|---|---|
| public LongPgmIndex build() Builds the | |
| buildFirstLevel | back to summary |
|---|---|
| private int buildFirstLevel() | |
| buildUpperLevel | back to summary |
|---|---|
| private int buildUpperLevel(int levelOffset, int levelNumSegments) | |
| getKey | back to summary |
|---|---|
| private long getKey(int segmentDataIndex, int[] segmentData) | |
| ramBytesAllocated | back to summary |
|---|---|
| public long ramBytesAllocated() Implements com. Estimates the allocated memory. It does not count the memory for the list of keys, only for the builder itself.
| |
| ramBytesUsed | back to summary |
|---|---|
| public long ramBytesUsed() Implements com. Estimates the bytes that are actually used. It does not count the memory for the list of keys, only for the builder itself.
| |
| setEpsilon | back to summary |
|---|---|
| public LongPgmIndex. Sets the epsilon range to use when learning the segments for the list of keys. | |
| setEpsilonRecursive | back to summary |
|---|---|
| public LongPgmIndex. Sets the recursive epsilon range to use when learning the segments for the segment levels. | |
| setSortedKeys | back to summary |
|---|---|
| public LongPgmIndex. Sets the sorted list of keys to build the index for; duplicate elements are allowed. | |
| setSortedKeys | back to summary |
|---|---|
| public LongPgmIndex. Sets the sorted array of keys to build the index for; duplicate elements are allowed. | |
| Modifier and Type | Class and Description |
|---|---|
| private static class |
| Modifier and Type | Field and Description |
|---|---|
| private final Iterator |
| Access | Constructor and Description |
|---|---|
| private |
| Modifier and Type | Method and Description |
|---|---|
| public <T extends LongProcedure> T | forEachInRange(T procedure, long minKey, long maxKey)
Overrides com. Applies |
| public int | indexOf(long key)
Overrides com. Searches the specified key, and returns its index in the element list. |
| public Iterator | rangeIterator(long minKey, long maxKey)
Overrides com. Returns an iterator over the keys in the list that are greater than or equal to |
| emptyIterator | back to summary |
|---|---|
| private final Iterator<LongCursor> emptyIterator | |
| LongEmptyPgmIndex | back to summary |
|---|---|
| private LongEmptyPgmIndex() | |
| forEachInRange | back to summary |
|---|---|
| public <T extends LongProcedure> T forEachInRange(T procedure, long minKey, long maxKey) Overrides com. Doc from com. Applies
| |
| indexOf | back to summary |
|---|---|
| public int indexOf(long key) Overrides com. Doc from com. Searches the specified key, and returns its index in the element list. If multiple elements are equal to the specified key, there is no guarantee which one will be found.
| |
| rangeIterator | back to summary |
|---|---|
| public Iterator Overrides com. Doc from com. Returns an iterator over the keys in the list that are greater than or equal to
| |
| Access | Constructor and Description |
|---|---|
| private |
| Modifier and Type | Method and Description |
|---|---|
| protected LongCursor |
| LongEmptyIterator | back to summary |
|---|---|
| private LongEmptyIterator() | |
| fetch | back to summary |
|---|---|
| protected LongCursor fetch() Implements abstract com. Doc from com. Fetch next element. The implementation must return
| |
| Modifier and Type | Field and Description |
|---|---|
| private final long[] | |
| private final LongCursor | |
| private final long | |
| private final int |
| Access | Constructor and Description |
|---|---|
| protected | RangeIterator(LongArrayList keys, int fromIndex, long maxKey)
Range iterator from |
| Modifier and Type | Method and Description |
|---|---|
| protected LongCursor |
| buffer | back to summary |
|---|---|
| private final long[] buffer | |
| cursor | back to summary |
|---|---|
| private final LongCursor cursor | |
| maxKey | back to summary |
|---|---|
| private final long maxKey | |
| size | back to summary |
|---|---|
| private final int size | |
| RangeIterator | back to summary |
|---|---|
| protected RangeIterator(LongArrayList keys, int fromIndex, long maxKey) Range iterator from | |
| fetch | back to summary |
|---|---|
| protected LongCursor fetch() Implements abstract com. Doc from com. Fetch next element. The implementation must return
| |