float.
Implementation of the PGM-Index described at https://pgm.di.unipi.it/, based on the paper
Paolo Ferragina and Giorgio Vinciguerra. The PGM-index: a fully-dynamic compressed learned index with provable worst-case bounds. PVLDB, 13(8): 1162-1175, 2020.It provides
rank and range search operations. indexOf() is faster than
B+Tree, and the index is much more compact. contains() is between 4x to 7x slower than
IntHashSet#contains(), but between 2.5x to 3x faster than Arrays#binarySearch.
Its compactness (40KB for 200MB of keys) makes it efficient for very large collections, the
index fitting easily in the L2 cache. The epsilon parameter should be set according to
the desired space-time trade-off. A smaller value makes the estimation more precise and the range
smaller but at the cost of increased space usage. In practice, epsilon 64 is a good sweet
spot.
Internally the index uses an optimal piecewise linear mapping from keys to their position in the sorted order. This mapping is represented as a sequence of linear models (segments) which are themselves recursively indexed by other piecewise linear mappings.
| Modifier and Type | Class and Description |
|---|---|
| public static class | FloatPgmIndex.
Builds a |
| private static class | FloatPgmIndex.
Empty immutable PGM Index. |
| protected static class | FloatPgmIndex.
Iterator over a range of elements in a sorted array. |
| Modifier and Type | Field and Description |
|---|---|
| public static final int | BEYOND_EPSILON_JUMP
Initial value of the exponential jump when scanning out of the epsilon range. |
| public static final int | |
| public static final FloatPgmIndex | EMPTY
Empty immutable FloatPgmIndex. |
| public static final int | EPSILON
Epsilon approximation range when searching the list of keys. |
| public final int | epsilon
The epsilon range used to build this index. |
| public static final int | EPSILON_RECURSIVE
Epsilon approximation range for the segments layers. |
| public final int | epsilonRecursive
The recursive epsilon range used to build this index. |
| public final float | |
| public static final int | KEY_SIZE
Size of a key, measured in |
| public final FloatArrayList | keys
The list of keys for which this index is built. |
| public final float | |
| public final int[] | levelOffsets
The offsets in |
| public static final int | SEGMENT_DATA_SIZE
Data size of a segment, measured in |
| public final int[] | segmentData
The index data. |
| public final int | size
The size of the key set. |
| Access | Constructor and Description |
|---|---|
| private | FloatPgmIndex(FloatArrayList keys, int size, int epsilon, int epsilonRecursive, int[] levelOffsets, int[] segmentData)
|
| private |
| Modifier and Type | Method and Description |
|---|---|
| private int | |
| public boolean | |
| private int | Returns: the segment data index; or -1 if none.Finds the segment responsible for a given key, that is, the rightmost segment having its first key <= the searched key. |
| public <T extends FloatProcedure> T | forEachInRange(T procedure, float minKey, float maxKey)
Applies |
| private static long | |
| private float | |
| private static double | |
| public int | Returns: The index of the searched key if it is present; otherwise,(-(<i>insertion
point</i>) - 1). The insertion point is defined as the point at which the key would
be inserted into the list: the index of the first element greater than the key, or keys#size() if all the elements are less than the specified key. Note that this
guarantees that the return value will be >= 0 if and only if the key is found.Searches the specified key, and returns its index in the element list. |
| public boolean | |
| public long | ramBytesAllocated()
Implements com. Estimates the allocated memory. |
| public long | ramBytesUsed()
Implements com. Estimates the bytes that are actually used. |
| public int | rangeCardinality(float minKey, float maxKey)
Returns the number of keys in the list that are greater than or equal to |
| public Iterator | rangeIterator(float minKey, float maxKey)
Returns an iterator over the keys in the list that are greater than or equal to |
| public int | Returns: The index of the searched key if it is present; otherwise, theinsertion point.
The insertion point is defined as the point at which the key would be inserted into
the list: the index of the first element greater than the key, or keys#size() if all the elements are less than the specified key. Note that this method always
returns a value >= 0.Returns, for any value |
| public int |
| BEYOND_EPSILON_JUMP | back to summary |
|---|---|
| public static final int BEYOND_EPSILON_JUMP Initial value of the exponential jump when scanning out of the epsilon range. | |
| DOUBLE_KEY_SIZE | back to summary |
|---|---|
| public static final int DOUBLE_KEY_SIZE 2x | |
| EMPTY | back to summary |
|---|---|
| public static final FloatPgmIndex EMPTY Empty immutable FloatPgmIndex. | |
| EPSILON | back to summary |
|---|---|
| public static final int EPSILON Epsilon approximation range when searching the list of keys. Controls the size of the returned search range, strictly greater than 0. It should be set according to the desired space-time trade-off. A smaller value makes the estimation more precise and the range smaller but at the cost of increased space usage. With EPSILON=64 the benchmark with 200MB of keys shows that this PGM index requires only 2% additional memory on average (40KB). It depends on the distribution of the keys. This epsilon value is good even for 2MB of keys. With EPSILON=32: +5% speed, but 4x space (160KB). | |
| epsilon | back to summary |
|---|---|
| public final int epsilon The epsilon range used to build this index. | |
| EPSILON_RECURSIVE | back to summary |
|---|---|
| public static final int EPSILON_RECURSIVE Epsilon approximation range for the segments layers. Controls the size of the search range in the hierarchical segment lists, strictly greater than 0. | |
| epsilonRecursive | back to summary |
|---|---|
| public final int epsilonRecursive The recursive epsilon range used to build this index. | |
| firstKey | back to summary |
|---|---|
| public final float firstKey The lowest key in | |
| KEY_SIZE | back to summary |
|---|---|
| public static final int KEY_SIZE Size of a key, measured in | |
| keys | back to summary |
|---|---|
| public final FloatArrayList keys The list of keys for which this index is built. It is sorted and may contain duplicate elements. | |
| lastKey | back to summary |
|---|---|
| public final float lastKey The highest key in | |
| levelOffsets | back to summary |
|---|---|
| public final int[] levelOffsets The offsets in | |
| SEGMENT_DATA_SIZE | back to summary |
|---|---|
| public static final int SEGMENT_DATA_SIZE Data size of a segment, measured in | |
| segmentData | back to summary |
|---|---|
| public final int[] segmentData The index data. It contains all the segments for all the levels. | |
| size | back to summary |
|---|---|
| public final int size The size of the key set. That is, the number of distinct elements in | |
| FloatPgmIndex | back to summary |
|---|---|
| private FloatPgmIndex(FloatArrayList keys, int size, int epsilon, int epsilonRecursive, int[] levelOffsets, int[] segmentData) | |
| FloatPgmIndex | back to summary |
|---|---|
| private FloatPgmIndex() Empty set constructor. | |
| approximateIndex | back to summary |
|---|---|
| private int approximateIndex(float key, int segmentDataIndex, int[] segmentData) | |
| contains | back to summary |
|---|---|
| public boolean contains(float key) Returns whether this key set contains the given key. | |
| findSegment | back to summary |
|---|---|
| private int findSegment(float key) Finds the segment responsible for a given key, that is, the rightmost segment having its first key <= the searched key.
| |
| forEachInRange | back to summary |
|---|---|
| public <T extends FloatProcedure> T forEachInRange(T procedure, float minKey, float maxKey) Applies | |
| getIntercept | back to summary |
|---|---|
| private static long getIntercept(int segmentDataIndex, int[] segmentData) | |
| getKey | back to summary |
|---|---|
| private float getKey(int segmentDataIndex, int[] segmentData) | |
| getSlope | back to summary |
|---|---|
| private static double getSlope(int segmentDataIndex, int[] segmentData) | |
| indexOf | back to summary |
|---|---|
| public int indexOf(float key) Searches the specified key, and returns its index in the element list. If multiple elements are equal to the specified key, there is no guarantee which one will be found.
| |
| isEmpty | back to summary |
|---|---|
| public boolean isEmpty() Returns whether this key set is empty. | |
| ramBytesAllocated | back to summary |
|---|---|
| public long ramBytesAllocated() Implements com. Estimates the allocated memory. It does not count the memory for the list of keys, only for the index itself.
| |
| ramBytesUsed | back to summary |
|---|---|
| public long ramBytesUsed() Implements com. Estimates the bytes that are actually used. It does not count the memory for the list of keys, only for the index itself.
| |
| rangeCardinality | back to summary |
|---|---|
| public int rangeCardinality(float minKey, float maxKey) Returns the number of keys in the list that are greater than or equal to | |
| rangeIterator | back to summary |
|---|---|
| public Iterator Returns an iterator over the keys in the list that are greater than or equal to | |
| rank | back to summary |
|---|---|
| public int rank(float x) Returns, for any value If multiple elements are equal to the specified key, there is no guarantee which one will be found.
| |
| size | back to summary |
|---|---|
| public int size() Returns the size of the key set. That is, the number of distinct elements in | |
FloatPgmIndex on a provided sorted list of keys.
| Modifier and Type | Field and Description |
|---|---|
| protected int | |
| protected int | |
| protected FloatArrayList | |
| protected int | |
| protected PlaModel | |
| protected IntArrayList | |
| protected int |
| Access | Constructor and Description |
|---|---|
| public |
| Modifier and Type | Method and Description |
|---|---|
| public void | accept(double
The first key of the segment. firstKey, double The segment slope. slope, long The segment intercept. intercept)Implements com. Consumes a new segment. |
| private void | addSentinelSegment(int endIndex)
Adds a sentinel segment that is used to give a limit for the position approximation, but does not count in the number of segments per level. |
| public FloatPgmIndex | |
| private int | |
| private int | |
| private float | |
| public long | ramBytesAllocated()
Implements com. Estimates the allocated memory. |
| public long | ramBytesUsed()
Implements com. Estimates the bytes that are actually used. |
| public FloatPgmIndex. | setEpsilon(int epsilon)
Sets the epsilon range to use when learning the segments for the list of keys. |
| public FloatPgmIndex. | setEpsilonRecursive(int epsilonRecursive)
Sets the recursive epsilon range to use when learning the segments for the segment levels. |
| public FloatPgmIndex. | setSortedKeys(FloatArrayList keys)
Sets the sorted list of keys to build the index for; duplicate elements are allowed. |
| public FloatPgmIndex. | setSortedKeys(float[] keys, int length)
Sets the sorted array of keys to build the index for; duplicate elements are allowed. |
| epsilon | back to summary |
|---|---|
| protected int epsilon | |
| epsilonRecursive | back to summary |
|---|---|
| protected int epsilonRecursive | |
| keys | back to summary |
|---|---|
| protected FloatArrayList keys | |
| numSegments | back to summary |
|---|---|
| protected int numSegments | |
| plam | back to summary |
|---|---|
| protected PlaModel plam | |
| segmentData | back to summary |
|---|---|
| protected IntArrayList segmentData | |
| size | back to summary |
|---|---|
| protected int size | |
| FloatBuilder | back to summary |
|---|---|
| public FloatBuilder() | |
| accept | back to summary |
|---|---|
| public void accept(double firstKey, double slope, long intercept) Implements com. Doc from com. Consumes a new segment. The segment is defined by the epsilon-approximation function fs(k) = k × slope + intercept.
| |
| addSentinelSegment | back to summary |
|---|---|
| private void addSentinelSegment(int endIndex) Adds a sentinel segment that is used to give a limit for the position approximation, but does not count in the number of segments per level. | |
| build | back to summary |
|---|---|
| public FloatPgmIndex build() Builds the | |
| buildFirstLevel | back to summary |
|---|---|
| private int buildFirstLevel() | |
| buildUpperLevel | back to summary |
|---|---|
| private int buildUpperLevel(int levelOffset, int levelNumSegments) | |
| getKey | back to summary |
|---|---|
| private float getKey(int segmentDataIndex, int[] segmentData) | |
| ramBytesAllocated | back to summary |
|---|---|
| public long ramBytesAllocated() Implements com. Estimates the allocated memory. It does not count the memory for the list of keys, only for the builder itself.
| |
| ramBytesUsed | back to summary |
|---|---|
| public long ramBytesUsed() Implements com. Estimates the bytes that are actually used. It does not count the memory for the list of keys, only for the builder itself.
| |
| setEpsilon | back to summary |
|---|---|
| public FloatPgmIndex. Sets the epsilon range to use when learning the segments for the list of keys. | |
| setEpsilonRecursive | back to summary |
|---|---|
| public FloatPgmIndex. Sets the recursive epsilon range to use when learning the segments for the segment levels. | |
| setSortedKeys | back to summary |
|---|---|
| public FloatPgmIndex. Sets the sorted list of keys to build the index for; duplicate elements are allowed. | |
| setSortedKeys | back to summary |
|---|---|
| public FloatPgmIndex. Sets the sorted array of keys to build the index for; duplicate elements are allowed. | |
| Modifier and Type | Class and Description |
|---|---|
| private static class |
| Modifier and Type | Field and Description |
|---|---|
| private final Iterator |
| Access | Constructor and Description |
|---|---|
| private |
| Modifier and Type | Method and Description |
|---|---|
| public <T extends FloatProcedure> T | forEachInRange(T procedure, float minKey, float maxKey)
Overrides com. Applies |
| public int | indexOf(float key)
Overrides com. Searches the specified key, and returns its index in the element list. |
| public Iterator | rangeIterator(float minKey, float maxKey)
Overrides com. Returns an iterator over the keys in the list that are greater than or equal to |
| emptyIterator | back to summary |
|---|---|
| private final Iterator<FloatCursor> emptyIterator | |
| FloatEmptyPgmIndex | back to summary |
|---|---|
| private FloatEmptyPgmIndex() | |
| forEachInRange | back to summary |
|---|---|
| public <T extends FloatProcedure> T forEachInRange(T procedure, float minKey, float maxKey) Overrides com. Doc from com. Applies
| |
| indexOf | back to summary |
|---|---|
| public int indexOf(float key) Overrides com. Doc from com. Searches the specified key, and returns its index in the element list. If multiple elements are equal to the specified key, there is no guarantee which one will be found.
| |
| rangeIterator | back to summary |
|---|---|
| public Iterator Overrides com. Doc from com. Returns an iterator over the keys in the list that are greater than or equal to
| |
| Access | Constructor and Description |
|---|---|
| private |
| Modifier and Type | Method and Description |
|---|---|
| protected FloatCursor |
| FloatEmptyIterator | back to summary |
|---|---|
| private FloatEmptyIterator() | |
| fetch | back to summary |
|---|---|
| protected FloatCursor fetch() Implements abstract com. Doc from com. Fetch next element. The implementation must return
| |
| Modifier and Type | Field and Description |
|---|---|
| private final float[] | |
| private final FloatCursor | |
| private final float | |
| private final int |
| Access | Constructor and Description |
|---|---|
| protected | RangeIterator(FloatArrayList keys, int fromIndex, float maxKey)
Range iterator from |
| Modifier and Type | Method and Description |
|---|---|
| protected FloatCursor |
| buffer | back to summary |
|---|---|
| private final float[] buffer | |
| cursor | back to summary |
|---|---|
| private final FloatCursor cursor | |
| maxKey | back to summary |
|---|---|
| private final float maxKey | |
| size | back to summary |
|---|---|
| private final int size | |
| RangeIterator | back to summary |
|---|---|
| protected RangeIterator(FloatArrayList keys, int fromIndex, float maxKey) Range iterator from | |
| fetch | back to summary |
|---|---|
| protected FloatCursor fetch() Implements abstract com. Doc from com. Fetch next element. The implementation must return
| |