Top Description Inners Fields Constructors Methods
com.carrotsearch.hppc

public Class FloatPgmIndex

extends Object
implements Accountable
Class Inheritance
All Implemented Interfaces
com.carrotsearch.hppc.Accountable
Known Direct Subclasses
com.carrotsearch.hppc.FloatPgmIndex.FloatEmptyPgmIndex
Annotations
@Generated:KTypePgmIndex.java
date:2024-06-04T15:20:17+0200
Imports
com.carrotsearch.hppc.cursors.FloatCursor, com.carrotsearch.hppc.procedures.FloatProcedure, java.util.Arrays, .Iterator

Space-efficient index that enables fast rank/range search operations on a sorted sequence of float.

Implementation of the PGM-Index described at https://pgm.di.unipi.it/, based on the paper

  Paolo Ferragina and Giorgio Vinciguerra.
  The PGM-index: a fully-dynamic compressed learned index with provable worst-case bounds.
  PVLDB, 13(8): 1162-1175, 2020.
It provides rank and range search operations. indexOf() is faster than B+Tree, and the index is much more compact. contains() is between 4x to 7x slower than IntHashSet#contains(), but between 2.5x to 3x faster than Arrays#binarySearch.

Its compactness (40KB for 200MB of keys) makes it efficient for very large collections, the index fitting easily in the L2 cache. The epsilon parameter should be set according to the desired space-time trade-off. A smaller value makes the estimation more precise and the range smaller but at the cost of increased space usage. In practice, epsilon 64 is a good sweet spot.

Internally the index uses an optimal piecewise linear mapping from keys to their position in the sorted order. This mapping is represented as a sequence of linear models (segments) which are themselves recursively indexed by other piecewise linear mappings.

Nested and Inner Type Summary

Modifier and TypeClass and Description
public static class
FloatPgmIndex.FloatBuilder

Builds a FloatPgmIndex on a provided sorted list of keys.

private static class
FloatPgmIndex.FloatEmptyPgmIndex

Empty immutable PGM Index.

protected static class
FloatPgmIndex.RangeIterator

Iterator over a range of elements in a sorted array.

Field Summary

Modifier and TypeField and Description
public static final int
BEYOND_EPSILON_JUMP

Initial value of the exponential jump when scanning out of the epsilon range.

public static final int
public static final FloatPgmIndex
EMPTY

Empty immutable FloatPgmIndex.

public static final int
EPSILON

Epsilon approximation range when searching the list of keys.

public final int
epsilon

The epsilon range used to build this index.

public static final int
EPSILON_RECURSIVE

Epsilon approximation range for the segments layers.

public final int
epsilonRecursive

The recursive epsilon range used to build this index.

public final float
firstKey

The lowest key in keys.

public static final int
KEY_SIZE

Size of a key, measured in Integer#BYTES because the key is stored in an int[].

public final FloatArrayList
keys

The list of keys for which this index is built.

public final float
lastKey

The highest key in keys.

public final int[]
levelOffsets

The offsets in segmentData of the first segment of each segment level.

public static final int
SEGMENT_DATA_SIZE

Data size of a segment, measured in Integer#BYTES, because segments are stored in an int[].

public final int[]
segmentData

The index data.

public final int
size

The size of the key set.

Constructor Summary

AccessConstructor and Description
private
FloatPgmIndex(FloatArrayList keys, int size, int epsilon, int epsilonRecursive, int[] levelOffsets, int[] segmentData)

private
FloatPgmIndex()

Empty set constructor.

Method Summary

Modifier and TypeMethod and Description
private int
approximateIndex(float key, int segmentDataIndex, int[] segmentData)

public boolean
contains(float key)

Returns whether this key set contains the given key.

private int

Returns:

the segment data index; or -1 if none.
findSegment
(float key)

Finds the segment responsible for a given key, that is, the rightmost segment having its first key <= the searched key.

public <T extends FloatProcedure> T
forEachInRange(T procedure, float minKey, float maxKey)

Applies procedure to the keys in the list that are greater than or equal to minKey (inclusive), and less than or equal to maxKey (inclusive).

private static long
getIntercept(int segmentDataIndex, int[] segmentData)

private float
getKey(int segmentDataIndex, int[] segmentData)

private static double
getSlope(int segmentDataIndex, int[] segmentData)

public int

Returns:

The index of the searched key if it is present; otherwise, (-(<i>insertion point</i>) - 1). The insertion point is defined as the point at which the key would be inserted into the list: the index of the first element greater than the key, or keys#size() if all the elements are less than the specified key. Note that this guarantees that the return value will be >= 0 if and only if the key is found.
indexOf
(float key)

Searches the specified key, and returns its index in the element list.

public boolean
isEmpty()

Returns whether this key set is empty.

public long
ramBytesAllocated()

Implements com.carrotsearch.hppc.Accountable.ramBytesAllocated.

Estimates the allocated memory.

public long
ramBytesUsed()

Implements com.carrotsearch.hppc.Accountable.ramBytesUsed.

Estimates the bytes that are actually used.

public int
rangeCardinality(float minKey, float maxKey)

Returns the number of keys in the list that are greater than or equal to minKey (inclusive), and less than or equal to maxKey (inclusive).

public Iterator<FloatCursor>
rangeIterator(float minKey, float maxKey)

Returns an iterator over the keys in the list that are greater than or equal to minKey (inclusive), and less than or equal to maxKey (inclusive).

public int

Returns:

The index of the searched key if it is present; otherwise, the insertion point. The insertion point is defined as the point at which the key would be inserted into the list: the index of the first element greater than the key, or keys#size() if all the elements are less than the specified key. Note that this method always returns a value >= 0.
rank
(float x)

Returns, for any value x, the number of keys in the sorted list which are smaller than x.

public int
size()

Returns the size of the key set.

Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Field Detail

BEYOND_EPSILON_JUMPback to summary
public static final int BEYOND_EPSILON_JUMP

Initial value of the exponential jump when scanning out of the epsilon range.

DOUBLE_KEY_SIZEback to summary
public static final int DOUBLE_KEY_SIZE

2x KEY_SIZE.

EMPTYback to summary
public static final FloatPgmIndex EMPTY

Empty immutable FloatPgmIndex.

EPSILONback to summary
public static final int EPSILON

Epsilon approximation range when searching the list of keys. Controls the size of the returned search range, strictly greater than 0. It should be set according to the desired space-time trade-off. A smaller value makes the estimation more precise and the range smaller but at the cost of increased space usage.

With EPSILON=64 the benchmark with 200MB of keys shows that this PGM index requires only 2% additional memory on average (40KB). It depends on the distribution of the keys. This epsilon value is good even for 2MB of keys. With EPSILON=32: +5% speed, but 4x space (160KB).

epsilonback to summary
public final int epsilon

The epsilon range used to build this index.

EPSILON_RECURSIVEback to summary
public static final int EPSILON_RECURSIVE

Epsilon approximation range for the segments layers. Controls the size of the search range in the hierarchical segment lists, strictly greater than 0.

epsilonRecursiveback to summary
public final int epsilonRecursive

The recursive epsilon range used to build this index.

firstKeyback to summary
public final float firstKey

The lowest key in keys.

KEY_SIZEback to summary
public static final int KEY_SIZE

Size of a key, measured in Integer#BYTES because the key is stored in an int[].

keysback to summary
public final FloatArrayList keys

The list of keys for which this index is built. It is sorted and may contain duplicate elements.

lastKeyback to summary
public final float lastKey

The highest key in keys.

levelOffsetsback to summary
public final int[] levelOffsets

The offsets in segmentData of the first segment of each segment level.

SEGMENT_DATA_SIZEback to summary
public static final int SEGMENT_DATA_SIZE

Data size of a segment, measured in Integer#BYTES, because segments are stored in an int[].

segmentDataback to summary
public final int[] segmentData

The index data. It contains all the segments for all the levels.

sizeback to summary
public final int size

The size of the key set. That is, the number of distinct elements in keys.

Constructor Detail

FloatPgmIndexback to summary
private FloatPgmIndex(FloatArrayList keys, int size, int epsilon, int epsilonRecursive, int[] levelOffsets, int[] segmentData)
FloatPgmIndexback to summary
private FloatPgmIndex()

Empty set constructor.

Method Detail

approximateIndexback to summary
private int approximateIndex(float key, int segmentDataIndex, int[] segmentData)
containsback to summary
public boolean contains(float key)

Returns whether this key set contains the given key.

findSegmentback to summary
private int findSegment(float key)

Finds the segment responsible for a given key, that is, the rightmost segment having its first key <= the searched key.

Returns:int

the segment data index; or -1 if none.

forEachInRangeback to summary
public <T extends FloatProcedure> T forEachInRange(T procedure, float minKey, float maxKey)

Applies procedure to the keys in the list that are greater than or equal to minKey (inclusive), and less than or equal to maxKey (inclusive).

getInterceptback to summary
private static long getIntercept(int segmentDataIndex, int[] segmentData)
getKeyback to summary
private float getKey(int segmentDataIndex, int[] segmentData)
getSlopeback to summary
private static double getSlope(int segmentDataIndex, int[] segmentData)
indexOfback to summary
public int indexOf(float key)

Searches the specified key, and returns its index in the element list. If multiple elements are equal to the specified key, there is no guarantee which one will be found.

Returns:int

The index of the searched key if it is present; otherwise, (-(<i>insertion point</i>) - 1). The insertion point is defined as the point at which the key would be inserted into the list: the index of the first element greater than the key, or keys#size() if all the elements are less than the specified key. Note that this guarantees that the return value will be >= 0 if and only if the key is found.

isEmptyback to summary
public boolean isEmpty()

Returns whether this key set is empty.

ramBytesAllocatedback to summary
public long ramBytesAllocated()

Implements com.carrotsearch.hppc.Accountable.ramBytesAllocated.

Estimates the allocated memory. It does not count the memory for the list of keys, only for the index itself.

Returns:long

Doc from com.carrotsearch.hppc.Accountable.ramBytesAllocated.

Ram allocated in bytes

Annotations
@Override
ramBytesUsedback to summary
public long ramBytesUsed()

Implements com.carrotsearch.hppc.Accountable.ramBytesUsed.

Estimates the bytes that are actually used. It does not count the memory for the list of keys, only for the index itself.

Returns:long

Doc from com.carrotsearch.hppc.Accountable.ramBytesUsed.

Ram used in bytes

Annotations
@Override
rangeCardinalityback to summary
public int rangeCardinality(float minKey, float maxKey)

Returns the number of keys in the list that are greater than or equal to minKey (inclusive), and less than or equal to maxKey (inclusive).

rangeIteratorback to summary
public Iterator<FloatCursor> rangeIterator(float minKey, float maxKey)

Returns an iterator over the keys in the list that are greater than or equal to minKey (inclusive), and less than or equal to maxKey (inclusive).

rankback to summary
public int rank(float x)

Returns, for any value x, the number of keys in the sorted list which are smaller than x. It is equal to indexOf if x belongs to the list, or -indexOf-1 otherwise.

If multiple elements are equal to the specified key, there is no guarantee which one will be found.

Returns:int

The index of the searched key if it is present; otherwise, the insertion point. The insertion point is defined as the point at which the key would be inserted into the list: the index of the first element greater than the key, or keys#size() if all the elements are less than the specified key. Note that this method always returns a value >= 0.

sizeback to summary
public int size()

Returns the size of the key set. That is, the number of distinct elements in keys.

com.carrotsearch.hppc back to summary

public Class FloatPgmIndex.FloatBuilder

extends Object
implements SegmentConsumer, Accountable
Class Inheritance
All Implemented Interfaces
com.carrotsearch.hppc.Accountable, com.carrotsearch.hppc.PlaModel.SegmentConsumer

Builds a FloatPgmIndex on a provided sorted list of keys.

Field Summary

Modifier and TypeField and Description
protected int
protected int
protected FloatArrayList
protected int
protected PlaModel
protected IntArrayList
protected int

Constructor Summary

AccessConstructor and Description
public

Method Summary

Modifier and TypeMethod and Description
public void
accept(double
The first key of the segment.
firstKey
,
double
The segment slope.
slope
,
long
The segment intercept.
intercept
)

Implements com.carrotsearch.hppc.PlaModel.SegmentConsumer.accept.

Consumes a new segment.

private void
addSentinelSegment(int endIndex)

Adds a sentinel segment that is used to give a limit for the position approximation, but does not count in the number of segments per level.

public FloatPgmIndex
build()

Builds the FloatPgmIndex; or EMPTY if there are no keys in the list.

private int
private int
buildUpperLevel(int levelOffset, int levelNumSegments)

private float
getKey(int segmentDataIndex, int[] segmentData)

public long
ramBytesAllocated()

Implements com.carrotsearch.hppc.Accountable.ramBytesAllocated.

Estimates the allocated memory.

public long
ramBytesUsed()

Implements com.carrotsearch.hppc.Accountable.ramBytesUsed.

Estimates the bytes that are actually used.

public FloatPgmIndex.FloatBuilder
setEpsilon(int epsilon)

Sets the epsilon range to use when learning the segments for the list of keys.

public FloatPgmIndex.FloatBuilder
setEpsilonRecursive(int epsilonRecursive)

Sets the recursive epsilon range to use when learning the segments for the segment levels.

public FloatPgmIndex.FloatBuilder
setSortedKeys(FloatArrayList keys)

Sets the sorted list of keys to build the index for; duplicate elements are allowed.

public FloatPgmIndex.FloatBuilder
setSortedKeys(float[] keys, int length)

Sets the sorted array of keys to build the index for; duplicate elements are allowed.

Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Field Detail

epsilonback to summary
protected int epsilon
epsilonRecursiveback to summary
protected int epsilonRecursive
keysback to summary
protected FloatArrayList keys
numSegmentsback to summary
protected int numSegments
plamback to summary
protected PlaModel plam
segmentDataback to summary
protected IntArrayList segmentData
sizeback to summary
protected int size

Constructor Detail

FloatBuilderback to summary
public FloatBuilder()

Method Detail

acceptback to summary
public void accept(double firstKey, double slope, long intercept)

Implements com.carrotsearch.hppc.PlaModel.SegmentConsumer.accept.

Doc from com.carrotsearch.hppc.PlaModel.SegmentConsumer.accept.

Consumes a new segment. The segment is defined by the epsilon-approximation function fs(k) = k × slope + intercept.

Parameters
firstKey:double

The first key of the segment.

slope:double

The segment slope.

intercept:long

The segment intercept.

Annotations
@Override
addSentinelSegmentback to summary
private void addSentinelSegment(int endIndex)

Adds a sentinel segment that is used to give a limit for the position approximation, but does not count in the number of segments per level.

buildback to summary
public FloatPgmIndex build()

Builds the FloatPgmIndex; or EMPTY if there are no keys in the list.

buildFirstLevelback to summary
private int buildFirstLevel()
buildUpperLevelback to summary
private int buildUpperLevel(int levelOffset, int levelNumSegments)
getKeyback to summary
private float getKey(int segmentDataIndex, int[] segmentData)
ramBytesAllocatedback to summary
public long ramBytesAllocated()

Implements com.carrotsearch.hppc.Accountable.ramBytesAllocated.

Estimates the allocated memory. It does not count the memory for the list of keys, only for the builder itself.

Returns:long

Doc from com.carrotsearch.hppc.Accountable.ramBytesAllocated.

Ram allocated in bytes

Annotations
@Override
ramBytesUsedback to summary
public long ramBytesUsed()

Implements com.carrotsearch.hppc.Accountable.ramBytesUsed.

Estimates the bytes that are actually used. It does not count the memory for the list of keys, only for the builder itself.

Returns:long

Doc from com.carrotsearch.hppc.Accountable.ramBytesUsed.

Ram used in bytes

Annotations
@Override
setEpsilonback to summary
public FloatPgmIndex.FloatBuilder setEpsilon(int epsilon)

Sets the epsilon range to use when learning the segments for the list of keys.

setEpsilonRecursiveback to summary
public FloatPgmIndex.FloatBuilder setEpsilonRecursive(int epsilonRecursive)

Sets the recursive epsilon range to use when learning the segments for the segment levels.

setSortedKeysback to summary
public FloatPgmIndex.FloatBuilder setSortedKeys(FloatArrayList keys)

Sets the sorted list of keys to build the index for; duplicate elements are allowed.

setSortedKeysback to summary
public FloatPgmIndex.FloatBuilder setSortedKeys(float[] keys, int length)

Sets the sorted array of keys to build the index for; duplicate elements are allowed.

com.carrotsearch.hppc back to summary

private Class FloatPgmIndex.FloatEmptyPgmIndex

extends FloatPgmIndex
Class Inheritance

Empty immutable PGM Index.

Nested and Inner Type Summary

Modifier and TypeClass and Description
private static class

Field Summary

Modifier and TypeField and Description
private final Iterator<FloatCursor>
Inherited from com.carrotsearch.hppc.FloatPgmIndex:
BEYOND_EPSILON_JUMPDOUBLE_KEY_SIZEEMPTYEPSILONepsilonEPSILON_RECURSIVEepsilonRecursivefirstKeyKEY_SIZEkeyslastKeylevelOffsetsSEGMENT_DATA_SIZEsegmentDatasize

Constructor Summary

AccessConstructor and Description
private

Method Summary

Modifier and TypeMethod and Description
public <T extends FloatProcedure> T
forEachInRange(T procedure, float minKey, float maxKey)

Overrides com.carrotsearch.hppc.FloatPgmIndex.forEachInRange.

Applies procedure to the keys in the list that are greater than or equal to minKey (inclusive), and less than or equal to maxKey (inclusive).

public int
indexOf(float key)

Overrides com.carrotsearch.hppc.FloatPgmIndex.indexOf.

Searches the specified key, and returns its index in the element list.

public Iterator<FloatCursor>
rangeIterator(float minKey, float maxKey)

Overrides com.carrotsearch.hppc.FloatPgmIndex.rangeIterator.

Returns an iterator over the keys in the list that are greater than or equal to minKey (inclusive), and less than or equal to maxKey (inclusive).

Inherited from com.carrotsearch.hppc.FloatPgmIndex:
containsisEmptyramBytesAllocatedramBytesUsedrangeCardinalityranksize

Field Detail

emptyIteratorback to summary
private final Iterator<FloatCursor> emptyIterator

Constructor Detail

FloatEmptyPgmIndexback to summary
private FloatEmptyPgmIndex()

Method Detail

forEachInRangeback to summary
public <T extends FloatProcedure> T forEachInRange(T procedure, float minKey, float maxKey)

Overrides com.carrotsearch.hppc.FloatPgmIndex.forEachInRange.

Doc from com.carrotsearch.hppc.FloatPgmIndex.forEachInRange.

Applies procedure to the keys in the list that are greater than or equal to minKey (inclusive), and less than or equal to maxKey (inclusive).

Annotations
@Override
indexOfback to summary
public int indexOf(float key)

Overrides com.carrotsearch.hppc.FloatPgmIndex.indexOf.

Doc from com.carrotsearch.hppc.FloatPgmIndex.indexOf.

Searches the specified key, and returns its index in the element list. If multiple elements are equal to the specified key, there is no guarantee which one will be found.

Returns:int

The index of the searched key if it is present; otherwise, (-(<i>insertion point</i>) - 1). The insertion point is defined as the point at which the key would be inserted into the list: the index of the first element greater than the key, or keys#size() if all the elements are less than the specified key. Note that this guarantees that the return value will be >= 0 if and only if the key is found.

Annotations
@Override
rangeIteratorback to summary
public Iterator<FloatCursor> rangeIterator(float minKey, float maxKey)

Overrides com.carrotsearch.hppc.FloatPgmIndex.rangeIterator.

Doc from com.carrotsearch.hppc.FloatPgmIndex.rangeIterator.

Returns an iterator over the keys in the list that are greater than or equal to minKey (inclusive), and less than or equal to maxKey (inclusive).

Annotations
@Override
com.carrotsearch.hppc back to summary

private Class FloatPgmIndex.FloatEmptyPgmIndex.FloatEmptyIterator

extends AbstractIterator<FloatCursor>
Class Inheritance

Constructor Summary

AccessConstructor and Description
private

Method Summary

Modifier and TypeMethod and Description
protected FloatCursor
fetch()

Implements abstract com.carrotsearch.hppc.AbstractIterator.fetch.

Fetch next element.

Inherited from com.carrotsearch.hppc.AbstractIterator:
donehasNextnextremove

Constructor Detail

FloatEmptyIteratorback to summary
private FloatEmptyIterator()

Method Detail

fetchback to summary
protected FloatCursor fetch()

Implements abstract com.carrotsearch.hppc.AbstractIterator.fetch.

Doc from com.carrotsearch.hppc.AbstractIterator.fetch.

Fetch next element. The implementation must return done() when all elements have been fetched.

Returns:FloatCursor

Returns the next value for the iterator or chain-calls done().

Annotations
@Override
com.carrotsearch.hppc back to summary

protected Class FloatPgmIndex.RangeIterator

extends AbstractIterator<FloatCursor>
Class Inheritance

Iterator over a range of elements in a sorted array.

Field Summary

Modifier and TypeField and Description
private final float[]
private final FloatCursor
private final float
private final int

Constructor Summary

AccessConstructor and Description
protected
RangeIterator(FloatArrayList keys, int fromIndex, float maxKey)

Range iterator from fromIndex (inclusive) to maxKey (inclusive).

Method Summary

Modifier and TypeMethod and Description
protected FloatCursor
fetch()

Implements abstract com.carrotsearch.hppc.AbstractIterator.fetch.

Fetch next element.

Inherited from com.carrotsearch.hppc.AbstractIterator:
donehasNextnextremove

Field Detail

bufferback to summary
private final float[] buffer
cursorback to summary
private final FloatCursor cursor
maxKeyback to summary
private final float maxKey
sizeback to summary
private final int size

Constructor Detail

RangeIteratorback to summary
protected RangeIterator(FloatArrayList keys, int fromIndex, float maxKey)

Range iterator from fromIndex (inclusive) to maxKey (inclusive).

Method Detail

fetchback to summary
protected FloatCursor fetch()

Implements abstract com.carrotsearch.hppc.AbstractIterator.fetch.

Doc from com.carrotsearch.hppc.AbstractIterator.fetch.

Fetch next element. The implementation must return done() when all elements have been fetched.

Returns:FloatCursor

Returns the next value for the iterator or chain-calls done().

Annotations
@Override