double.
Implementation of the PGM-Index described at https://pgm.di.unipi.it/, based on the paper
Paolo Ferragina and Giorgio Vinciguerra. The PGM-index: a fully-dynamic compressed learned index with provable worst-case bounds. PVLDB, 13(8): 1162-1175, 2020.It provides
rank and range search operations. indexOf() is faster than
B+Tree, and the index is much more compact. contains() is between 4x to 7x slower than
IntHashSet#contains(), but between 2.5x to 3x faster than Arrays#binarySearch.
Its compactness (40KB for 200MB of keys) makes it efficient for very large collections, the
index fitting easily in the L2 cache. The epsilon parameter should be set according to
the desired space-time trade-off. A smaller value makes the estimation more precise and the range
smaller but at the cost of increased space usage. In practice, epsilon 64 is a good sweet
spot.
Internally the index uses an optimal piecewise linear mapping from keys to their position in the sorted order. This mapping is represented as a sequence of linear models (segments) which are themselves recursively indexed by other piecewise linear mappings.
| Modifier and Type | Class and Description |
|---|---|
| public static class | DoublePgmIndex.
Builds a |
| private static class | DoublePgmIndex.
Empty immutable PGM Index. |
| protected static class | DoublePgmIndex.
Iterator over a range of elements in a sorted array. |
| Modifier and Type | Field and Description |
|---|---|
| public static final int | BEYOND_EPSILON_JUMP
Initial value of the exponential jump when scanning out of the epsilon range. |
| public static final int | |
| public static final DoublePgmIndex | EMPTY
Empty immutable DoublePgmIndex. |
| public static final int | EPSILON
Epsilon approximation range when searching the list of keys. |
| public final int | epsilon
The epsilon range used to build this index. |
| public static final int | EPSILON_RECURSIVE
Epsilon approximation range for the segments layers. |
| public final int | epsilonRecursive
The recursive epsilon range used to build this index. |
| public final double | |
| public static final int | KEY_SIZE
Size of a key, measured in |
| public final DoubleArrayList | keys
The list of keys for which this index is built. |
| public final double | |
| public final int[] | levelOffsets
The offsets in |
| public static final int | SEGMENT_DATA_SIZE
Data size of a segment, measured in |
| public final int[] | segmentData
The index data. |
| public final int | size
The size of the key set. |
| Access | Constructor and Description |
|---|---|
| private | DoublePgmIndex(DoubleArrayList keys, int size, int epsilon, int epsilonRecursive, int[] levelOffsets, int[] segmentData)
|
| private |
| Modifier and Type | Method and Description |
|---|---|
| private int | |
| public boolean | |
| private int | Returns: the segment data index; or -1 if none.Finds the segment responsible for a given key, that is, the rightmost segment having its first key <= the searched key. |
| public <T extends DoubleProcedure> T | forEachInRange(T procedure, double minKey, double maxKey)
Applies |
| private static long | |
| private double | |
| private static double | |
| public int | Returns: The index of the searched key if it is present; otherwise,(-(<i>insertion
point</i>) - 1). The insertion point is defined as the point at which the key would
be inserted into the list: the index of the first element greater than the key, or keys#size() if all the elements are less than the specified key. Note that this
guarantees that the return value will be >= 0 if and only if the key is found.Searches the specified key, and returns its index in the element list. |
| public boolean | |
| public long | ramBytesAllocated()
Implements com. Estimates the allocated memory. |
| public long | ramBytesUsed()
Implements com. Estimates the bytes that are actually used. |
| public int | rangeCardinality(double minKey, double maxKey)
Returns the number of keys in the list that are greater than or equal to |
| public Iterator | rangeIterator(double minKey, double maxKey)
Returns an iterator over the keys in the list that are greater than or equal to |
| public int | Returns: The index of the searched key if it is present; otherwise, theinsertion point.
The insertion point is defined as the point at which the key would be inserted into
the list: the index of the first element greater than the key, or keys#size() if all the elements are less than the specified key. Note that this method always
returns a value >= 0.Returns, for any value |
| public int |
| BEYOND_EPSILON_JUMP | back to summary |
|---|---|
| public static final int BEYOND_EPSILON_JUMP Initial value of the exponential jump when scanning out of the epsilon range. | |
| DOUBLE_KEY_SIZE | back to summary |
|---|---|
| public static final int DOUBLE_KEY_SIZE 2x | |
| EMPTY | back to summary |
|---|---|
| public static final DoublePgmIndex EMPTY Empty immutable DoublePgmIndex. | |
| EPSILON | back to summary |
|---|---|
| public static final int EPSILON Epsilon approximation range when searching the list of keys. Controls the size of the returned search range, strictly greater than 0. It should be set according to the desired space-time trade-off. A smaller value makes the estimation more precise and the range smaller but at the cost of increased space usage. With EPSILON=64 the benchmark with 200MB of keys shows that this PGM index requires only 2% additional memory on average (40KB). It depends on the distribution of the keys. This epsilon value is good even for 2MB of keys. With EPSILON=32: +5% speed, but 4x space (160KB). | |
| epsilon | back to summary |
|---|---|
| public final int epsilon The epsilon range used to build this index. | |
| EPSILON_RECURSIVE | back to summary |
|---|---|
| public static final int EPSILON_RECURSIVE Epsilon approximation range for the segments layers. Controls the size of the search range in the hierarchical segment lists, strictly greater than 0. | |
| epsilonRecursive | back to summary |
|---|---|
| public final int epsilonRecursive The recursive epsilon range used to build this index. | |
| firstKey | back to summary |
|---|---|
| public final double firstKey The lowest key in | |
| KEY_SIZE | back to summary |
|---|---|
| public static final int KEY_SIZE Size of a key, measured in | |
| keys | back to summary |
|---|---|
| public final DoubleArrayList keys The list of keys for which this index is built. It is sorted and may contain duplicate elements. | |
| lastKey | back to summary |
|---|---|
| public final double lastKey The highest key in | |
| levelOffsets | back to summary |
|---|---|
| public final int[] levelOffsets The offsets in | |
| SEGMENT_DATA_SIZE | back to summary |
|---|---|
| public static final int SEGMENT_DATA_SIZE Data size of a segment, measured in | |
| segmentData | back to summary |
|---|---|
| public final int[] segmentData The index data. It contains all the segments for all the levels. | |
| size | back to summary |
|---|---|
| public final int size The size of the key set. That is, the number of distinct elements in | |
| DoublePgmIndex | back to summary |
|---|---|
| private DoublePgmIndex(DoubleArrayList keys, int size, int epsilon, int epsilonRecursive, int[] levelOffsets, int[] segmentData) | |
| DoublePgmIndex | back to summary |
|---|---|
| private DoublePgmIndex() Empty set constructor. | |
| approximateIndex | back to summary |
|---|---|
| private int approximateIndex(double key, int segmentDataIndex, int[] segmentData) | |
| contains | back to summary |
|---|---|
| public boolean contains(double key) Returns whether this key set contains the given key. | |
| findSegment | back to summary |
|---|---|
| private int findSegment(double key) Finds the segment responsible for a given key, that is, the rightmost segment having its first key <= the searched key.
| |
| forEachInRange | back to summary |
|---|---|
| public <T extends DoubleProcedure> T forEachInRange(T procedure, double minKey, double maxKey) Applies | |
| getIntercept | back to summary |
|---|---|
| private static long getIntercept(int segmentDataIndex, int[] segmentData) | |
| getKey | back to summary |
|---|---|
| private double getKey(int segmentDataIndex, int[] segmentData) | |
| getSlope | back to summary |
|---|---|
| private static double getSlope(int segmentDataIndex, int[] segmentData) | |
| indexOf | back to summary |
|---|---|
| public int indexOf(double key) Searches the specified key, and returns its index in the element list. If multiple elements are equal to the specified key, there is no guarantee which one will be found.
| |
| isEmpty | back to summary |
|---|---|
| public boolean isEmpty() Returns whether this key set is empty. | |
| ramBytesAllocated | back to summary |
|---|---|
| public long ramBytesAllocated() Implements com. Estimates the allocated memory. It does not count the memory for the list of keys, only for the index itself.
| |
| ramBytesUsed | back to summary |
|---|---|
| public long ramBytesUsed() Implements com. Estimates the bytes that are actually used. It does not count the memory for the list of keys, only for the index itself.
| |
| rangeCardinality | back to summary |
|---|---|
| public int rangeCardinality(double minKey, double maxKey) Returns the number of keys in the list that are greater than or equal to | |
| rangeIterator | back to summary |
|---|---|
| public Iterator Returns an iterator over the keys in the list that are greater than or equal to | |
| rank | back to summary |
|---|---|
| public int rank(double x) Returns, for any value If multiple elements are equal to the specified key, there is no guarantee which one will be found.
| |
| size | back to summary |
|---|---|
| public int size() Returns the size of the key set. That is, the number of distinct elements in | |
DoublePgmIndex on a provided sorted list of keys.
| Modifier and Type | Field and Description |
|---|---|
| protected int | |
| protected int | |
| protected DoubleArrayList | |
| protected int | |
| protected PlaModel | |
| protected IntArrayList | |
| protected int |
| Access | Constructor and Description |
|---|---|
| public |
| Modifier and Type | Method and Description |
|---|---|
| public void | accept(double
The first key of the segment. firstKey, double The segment slope. slope, long The segment intercept. intercept)Implements com. Consumes a new segment. |
| private void | addSentinelSegment(int endIndex)
Adds a sentinel segment that is used to give a limit for the position approximation, but does not count in the number of segments per level. |
| public DoublePgmIndex | |
| private int | |
| private int | |
| private double | |
| public long | ramBytesAllocated()
Implements com. Estimates the allocated memory. |
| public long | ramBytesUsed()
Implements com. Estimates the bytes that are actually used. |
| public DoublePgmIndex. | setEpsilon(int epsilon)
Sets the epsilon range to use when learning the segments for the list of keys. |
| public DoublePgmIndex. | setEpsilonRecursive(int epsilonRecursive)
Sets the recursive epsilon range to use when learning the segments for the segment levels. |
| public DoublePgmIndex. | setSortedKeys(DoubleArrayList keys)
Sets the sorted list of keys to build the index for; duplicate elements are allowed. |
| public DoublePgmIndex. | setSortedKeys(double[] keys, int length)
Sets the sorted array of keys to build the index for; duplicate elements are allowed. |
| epsilon | back to summary |
|---|---|
| protected int epsilon | |
| epsilonRecursive | back to summary |
|---|---|
| protected int epsilonRecursive | |
| keys | back to summary |
|---|---|
| protected DoubleArrayList keys | |
| numSegments | back to summary |
|---|---|
| protected int numSegments | |
| plam | back to summary |
|---|---|
| protected PlaModel plam | |
| segmentData | back to summary |
|---|---|
| protected IntArrayList segmentData | |
| size | back to summary |
|---|---|
| protected int size | |
| DoubleBuilder | back to summary |
|---|---|
| public DoubleBuilder() | |
| accept | back to summary |
|---|---|
| public void accept(double firstKey, double slope, long intercept) Implements com. Doc from com. Consumes a new segment. The segment is defined by the epsilon-approximation function fs(k) = k × slope + intercept.
| |
| addSentinelSegment | back to summary |
|---|---|
| private void addSentinelSegment(int endIndex) Adds a sentinel segment that is used to give a limit for the position approximation, but does not count in the number of segments per level. | |
| build | back to summary |
|---|---|
| public DoublePgmIndex build() Builds the | |
| buildFirstLevel | back to summary |
|---|---|
| private int buildFirstLevel() | |
| buildUpperLevel | back to summary |
|---|---|
| private int buildUpperLevel(int levelOffset, int levelNumSegments) | |
| getKey | back to summary |
|---|---|
| private double getKey(int segmentDataIndex, int[] segmentData) | |
| ramBytesAllocated | back to summary |
|---|---|
| public long ramBytesAllocated() Implements com. Estimates the allocated memory. It does not count the memory for the list of keys, only for the builder itself.
| |
| ramBytesUsed | back to summary |
|---|---|
| public long ramBytesUsed() Implements com. Estimates the bytes that are actually used. It does not count the memory for the list of keys, only for the builder itself.
| |
| setEpsilon | back to summary |
|---|---|
| public DoublePgmIndex. Sets the epsilon range to use when learning the segments for the list of keys. | |
| setEpsilonRecursive | back to summary |
|---|---|
| public DoublePgmIndex. Sets the recursive epsilon range to use when learning the segments for the segment levels. | |
| setSortedKeys | back to summary |
|---|---|
| public DoublePgmIndex. Sets the sorted list of keys to build the index for; duplicate elements are allowed. | |
| setSortedKeys | back to summary |
|---|---|
| public DoublePgmIndex. Sets the sorted array of keys to build the index for; duplicate elements are allowed. | |
| Modifier and Type | Class and Description |
|---|---|
| private static class |
| Modifier and Type | Field and Description |
|---|---|
| private final Iterator |
| Access | Constructor and Description |
|---|---|
| private |
| Modifier and Type | Method and Description |
|---|---|
| public <T extends DoubleProcedure> T | forEachInRange(T procedure, double minKey, double maxKey)
Overrides com. Applies |
| public int | indexOf(double key)
Overrides com. Searches the specified key, and returns its index in the element list. |
| public Iterator | rangeIterator(double minKey, double maxKey)
Overrides com. Returns an iterator over the keys in the list that are greater than or equal to |
| emptyIterator | back to summary |
|---|---|
| private final Iterator<DoubleCursor> emptyIterator | |
| DoubleEmptyPgmIndex | back to summary |
|---|---|
| private DoubleEmptyPgmIndex() | |
| forEachInRange | back to summary |
|---|---|
| public <T extends DoubleProcedure> T forEachInRange(T procedure, double minKey, double maxKey) Overrides com. Doc from com. Applies
| |
| indexOf | back to summary |
|---|---|
| public int indexOf(double key) Overrides com. Doc from com. Searches the specified key, and returns its index in the element list. If multiple elements are equal to the specified key, there is no guarantee which one will be found.
| |
| rangeIterator | back to summary |
|---|---|
| public Iterator Overrides com. Doc from com. Returns an iterator over the keys in the list that are greater than or equal to
| |
| Access | Constructor and Description |
|---|---|
| private |
| Modifier and Type | Method and Description |
|---|---|
| protected DoubleCursor |
| DoubleEmptyIterator | back to summary |
|---|---|
| private DoubleEmptyIterator() | |
| fetch | back to summary |
|---|---|
| protected DoubleCursor fetch() Implements abstract com. Doc from com. Fetch next element. The implementation must return
| |
| Modifier and Type | Field and Description |
|---|---|
| private final double[] | |
| private final DoubleCursor | |
| private final double | |
| private final int |
| Access | Constructor and Description |
|---|---|
| protected | RangeIterator(DoubleArrayList keys, int fromIndex, double maxKey)
Range iterator from |
| Modifier and Type | Method and Description |
|---|---|
| protected DoubleCursor |
| buffer | back to summary |
|---|---|
| private final double[] buffer | |
| cursor | back to summary |
|---|---|
| private final DoubleCursor cursor | |
| maxKey | back to summary |
|---|---|
| private final double maxKey | |
| size | back to summary |
|---|---|
| private final int size | |
| RangeIterator | back to summary |
|---|---|
| protected RangeIterator(DoubleArrayList keys, int fromIndex, double maxKey) Range iterator from | |
| fetch | back to summary |
|---|---|
| protected DoubleCursor fetch() Implements abstract com. Doc from com. Fetch next element. The implementation must return
| |