Top Description Inners Fields Constructors Methods
com.fasterxml.jackson.core.sym

public final Class CharsToNameCanonicalizer

extends Object
Class Inheritance
Imports
java.io.IOException, java.util.Arrays, .BitSet, java.util.concurrent.atomic.AtomicReference, com.fasterxml.jackson.core.JsonFactory, .StreamReadConstraints, .TokenStreamFactory, com.fasterxml.jackson.core.exc.StreamConstraintsException, com.fasterxml.jackson.core.util.InternCache

This class is a kind of specialized type-safe Map, from char array to String value. Specialization means that in addition to type-safety and specific access patterns (key char array, Value optionally interned String; values added on access if necessary), and that instances are meant to be used concurrently, but by using well-defined mechanisms to obtain such concurrently usable instances. Main use for the class is to store symbol table information for things like compilers and parsers; especially when number of symbols (keywords) is limited.

For optimal performance, usage pattern should be one where matches should be very common (especially after "warm-up"), and as with most hash-based maps/sets, that hash codes are uniformly distributed. Also, collisions are slightly more expensive than with HashMap or HashSet, since hash codes are not used in resolving collisions; that is, equals() comparison is done with all symbols in same bucket index.
Finally, rehashing is also more expensive, as hash codes are not stored; rehashing requires all entries' hash codes to be recalculated. Reason for not storing hash codes is reduced memory usage, hoping for better memory locality.

Usual usage pattern is to create a single "master" instance, and either use that instance in sequential fashion, or to create derived "child" instances, which after use, are asked to return possible symbol additions to master instance. In either case benefit is that symbol table gets initialized so that further uses are more efficient, as eventually all symbols needed will already be in symbol table. At that point no more Symbol String allocations are needed, nor changes to symbol table itself.

Note that while individual SymbolTable instances are NOT thread-safe (much like generic collection classes), concurrently used "child" instances can be freely used without synchronization. However, using master table concurrently with child instances can only be done if access to master instance is read-only (i.e. no modifications done).

Nested and Inner Type Summary

Modifier and TypeClass and Description
pack-priv static class
CharsToNameCanonicalizer.Bucket

This class is a symbol table entry.

private static class
CharsToNameCanonicalizer.TableInfo

Immutable value class used for sharing information as efficiently as possible, by only require synchronization of reference manipulation but not access to contents.

Field Summary

Modifier and TypeField and Description
protected CharsToNameCanonicalizer.Bucket[]
_buckets

Overflow buckets; if primary doesn't match, lookup is done from here.

protected boolean
_canonicalize

Whether any canonicalization should be attempted (whether using intern or not.

protected final int
_factoryFeatures

Feature flags of TokenStreamFactory that uses this canonicalizer.

protected boolean
_hashShared

Flag that indicates whether underlying data structures for the main hash area are shared or not.

protected int
_indexMask

Mask used to get index from hash values; equal to _buckets.length - 1, when _buckets.length is a power of two.

protected int
_longestCollisionList

We need to keep track of the longest collision list; this is needed both to indicate problems with attacks and to allow flushing for other cases.

protected BitSet
_overflows

Lazily constructed structure that is used to keep track of collision buckets that have overflowed once: this is used to detect likely attempts at denial-of-service attacks that uses hash collisions.

protected final CharsToNameCanonicalizer
_parent

Sharing of learnt symbols is done by optional linking of symbol table instances with their parents.

protected final int
_seed

Seed value we use as the base to make hash codes non-static between different runs, but still stable for lifetime of a single symbol table instance.

protected int
_size

Current size (number of entries); needed to know if and when rehash.

protected int
_sizeThreshold

Limit that indicates maximum size this instance can hold before it needs to be expanded and rehashed.

protected final StreamReadConstraints
_streamReadConstraints

Constraints used by TokenStreamFactory that uses this canonicalizer.

protected String[]
_symbols

Primary matching symbols; it's expected most match occur from here.

protected final AtomicReference<CharsToNameCanonicalizer.TableInfo>
_tableInfo

Member that is only used by the root table instance: root passes immutable state info child instances, and children may return new state if they add entries to the table.

private static final int
DEFAULT_T_SIZE

Default initial table size.

public static final int
pack-priv static final int
MAX_COLL_CHAIN_LENGTH

Also: to thwart attacks based on hash collisions (which may or may not be cheap to calculate), we will need to detect "too long" collision chains.

pack-priv static final int
MAX_ENTRIES_FOR_REUSE

Let's only share reasonably sized symbol tables.

private static final int
MAX_T_SIZE

Let's not expand symbol tables past some maximum size; this should protected against OOMEs caused by large documents with unique (~= random) names.

Constructor Summary

AccessConstructor and Description
private
CharsToNameCanonicalizer(StreamReadConstraints src, int factoryFeatures, int seed)

Main method for constructing a root symbol table instance.

private
CharsToNameCanonicalizer(CharsToNameCanonicalizer parent, StreamReadConstraints src, int factoryFeatures, int seed, CharsToNameCanonicalizer.TableInfo parentState)

Internal constructor used when creating child instances.

Method Summary

Modifier and TypeMethod and Description
private String
_addSymbol(char[] buffer, int start, int len, int h, int index)

private String
_findSymbol2(char[] buffer, int start, int len, CharsToNameCanonicalizer.Bucket b)

private void
_handleSpillOverflow(int bucketIndex, CharsToNameCanonicalizer.Bucket newBucket, int mainIndex)

Method called when an overflow bucket has hit the maximum expected length: this may be a case of DoS attack.

public int

Returns:

Index value calculated
_hashToIndex
(int
Raw hash value to use for calculating index
rawHash
)

Helper method that takes in a "raw" hash value, shuffles it as necessary, and truncates to be used as the index.

protected void
_reportTooManyCollisions(int
Maximum allowed length of collision chain
maxLen
)

private static int
_thresholdSize(int hashAreaSize)

public int

Returns:

number of primary slots table has currently
bucketCount
()

Method for checking number of primary hash buckets this symbol table uses.

public int

Returns:

Hash code calculated
calcHash
(char[]
Input buffer that contains name to decode
buffer
,
int
Pointer to the first character of the name
start
,
int
Length of String; has to be at least 1 (caller guarantees)
len
)

Implementation of a hashing method for variable length Strings.

public int
public int

Returns:

Number of collisions in the primary hash area
collisionCount
()

Method mostly needed by unit tests; calculates number of entries that are in collision list.

private void
copyArrays()

Method called when copy-on-write is needed; generally when first change is made to a derived symbol table.

public static CharsToNameCanonicalizer

Returns:

Root instance to use for constructing new child instances
createRoot
()

Deprecated Since 2.16 use createRoot(TokenStreamFactory) instead
public static CharsToNameCanonicalizer

Returns:

Root instance to use for constructing new child instances
createRoot
(int
Seed for hash value calculation
seed
)

Deprecated Since 2.16 use createRoot(TokenStreamFactory) instead
public static CharsToNameCanonicalizer

Returns:

Root instance to use for constructing new child instances
createRoot
(TokenStreamFactory
Factory that will use the root instance; used for accessing configuration
owner
)

Method called to create root canonicalizer for a com.fasterxml.jackson.core.JsonFactory instance.

public static CharsToNameCanonicalizer
createRoot(TokenStreamFactory owner, int seed)

public String
findSymbol(char[] buffer, int start, int len, int h)

public int
public CharsToNameCanonicalizer

Returns:

Actual canonicalizer instance that can be used by a parser
makeChild
()

"Factory" method; will create a new child instance of this symbol table.

public CharsToNameCanonicalizer

Returns:

Actual canonicalizer instance that can be used by a parser
makeChild
(int
Configuration flags (ignored)
flags
)

Deprecated Since 2.16 use makeChild() instead.
public int

Returns:

Length of the collision chain
maxCollisionLength
()

Method mostly needed by unit tests; calculates length of the longest collision chain.

public boolean
private void
mergeChild(CharsToNameCanonicalizer.TableInfo childState)

Method that allows contents of child table to potentially be "merged in" with contents of this symbol table.

private void
rehash()

Method called when size (number of entries) of symbol table grows so big that load factor is exceeded.

public void
release()

Method called by the using code to indicate it is done with this instance.

public int

Returns:

Number of symbol entries contained by this canonicalizer instance
size
()

protected void
verifyInternalConsistency()

Diagnostics method that will verify that internal data structures are consistent; not meant as user-facing method but only for test suites and possible troubleshooting.

Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Field Detail

_bucketsback to summary
protected CharsToNameCanonicalizer.Bucket[] _buckets

Overflow buckets; if primary doesn't match, lookup is done from here.

Note

Number of buckets is half of number of symbol entries, on assumption there's less need for buckets.

_canonicalizeback to summary
protected boolean _canonicalize

Whether any canonicalization should be attempted (whether using intern or not.

Note

non-final since we may need to disable this with overflow.

_factoryFeaturesback to summary
protected final int _factoryFeatures

Feature flags of TokenStreamFactory that uses this canonicalizer.

_hashSharedback to summary
protected boolean _hashShared

Flag that indicates whether underlying data structures for the main hash area are shared or not. If they are, then they need to be handled in copy-on-write way, i.e. if they need to be modified, a copy needs to be made first; at this point it will not be shared any more, and can be modified.

This flag needs to be checked both when adding new main entries, and when adding new collision list queues (i.e. creating a new collision list head entry)

_indexMaskback to summary
protected int _indexMask

Mask used to get index from hash values; equal to _buckets.length - 1, when _buckets.length is a power of two.

_longestCollisionListback to summary
protected int _longestCollisionList

We need to keep track of the longest collision list; this is needed both to indicate problems with attacks and to allow flushing for other cases.

Since
2.1
_overflowsback to summary
protected BitSet _overflows

Lazily constructed structure that is used to keep track of collision buckets that have overflowed once: this is used to detect likely attempts at denial-of-service attacks that uses hash collisions.

Since
2.4
_parentback to summary
protected final CharsToNameCanonicalizer _parent

Sharing of learnt symbols is done by optional linking of symbol table instances with their parents. When parent linkage is defined, and child instance is released (call to release), parent's shared tables may be updated from the child instance.

_seedback to summary
protected final int _seed

Seed value we use as the base to make hash codes non-static between different runs, but still stable for lifetime of a single symbol table instance. This is done for security reasons, to avoid potential DoS attack via hash collisions.

Since
2.1
_sizeback to summary
protected int _size

Current size (number of entries); needed to know if and when rehash.

_sizeThresholdback to summary
protected int _sizeThreshold

Limit that indicates maximum size this instance can hold before it needs to be expanded and rehashed. Calculated using fill factor passed in to constructor.

_streamReadConstraintsback to summary
protected final StreamReadConstraints _streamReadConstraints

Constraints used by TokenStreamFactory that uses this canonicalizer.

Since
2.16
_symbolsback to summary
protected String[] _symbols

Primary matching symbols; it's expected most match occur from here.

_tableInfoback to summary
protected final AtomicReference<CharsToNameCanonicalizer.TableInfo> _tableInfo

Member that is only used by the root table instance: root passes immutable state info child instances, and children may return new state if they add entries to the table. Child tables do NOT use the reference.

DEFAULT_T_SIZEback to summary
private static final int DEFAULT_T_SIZE

Default initial table size. Shouldn't be miniscule (as there's cost to both array realloc and rehashing), but let's keep it reasonably small. For systems that properly reuse factories it doesn't matter either way; but when recreating factories often, initial overhead may dominate.

HASH_MULTback to summary
public static final int HASH_MULT
MAX_COLL_CHAIN_LENGTHback to summary
pack-priv static final int MAX_COLL_CHAIN_LENGTH

Also: to thwart attacks based on hash collisions (which may or may not be cheap to calculate), we will need to detect "too long" collision chains. Started with static value of 100 entries for the longest legal chain, but increased in Jackson 2.13 to 150 to work around specific test case.

Note

longest chain we have been able to produce without malicious intent has been 38 (with "com.fasterxml.jackson.core.main.TestWithTonsaSymbols"); our setting should be reasonable here.

Since
2.1 (changed in 2.13)
MAX_ENTRIES_FOR_REUSEback to summary
pack-priv static final int MAX_ENTRIES_FOR_REUSE

Let's only share reasonably sized symbol tables. Max size set to 3/4 of 16k; this corresponds to 64k main hash index. This should allow for enough distinct names for almost any case.

MAX_T_SIZEback to summary
private static final int MAX_T_SIZE

Let's not expand symbol tables past some maximum size; this should protected against OOMEs caused by large documents with unique (~= random) names.

Constructor Detail

CharsToNameCanonicalizerback to summary
private CharsToNameCanonicalizer(StreamReadConstraints src, int factoryFeatures, int seed)

Main method for constructing a root symbol table instance.

CharsToNameCanonicalizerback to summary
private CharsToNameCanonicalizer(CharsToNameCanonicalizer parent, StreamReadConstraints src, int factoryFeatures, int seed, CharsToNameCanonicalizer.TableInfo parentState)

Internal constructor used when creating child instances.

Method Detail

_addSymbolback to summary
private String _addSymbol(char[] buffer, int start, int len, int h, int index) throws IOException
_findSymbol2back to summary
private String _findSymbol2(char[] buffer, int start, int len, CharsToNameCanonicalizer.Bucket b)
_handleSpillOverflowback to summary
private void _handleSpillOverflow(int bucketIndex, CharsToNameCanonicalizer.Bucket newBucket, int mainIndex) throws IOException

Method called when an overflow bucket has hit the maximum expected length: this may be a case of DoS attack. Deal with it based on settings by either clearing up bucket (to avoid indefinite expansion) or throwing exception. Currently, the first overflow for any single bucket DOES NOT throw an exception, only second time (per symbol table instance)

_hashToIndexback to summary
public int _hashToIndex(int rawHash)

Helper method that takes in a "raw" hash value, shuffles it as necessary, and truncates to be used as the index.

Parameters
rawHash:int

Raw hash value to use for calculating index

Returns:int

Index value calculated

_reportTooManyCollisionsback to summary
protected void _reportTooManyCollisions(int maxLen) throws StreamConstraintsException
Parameters
maxLen:int

Maximum allowed length of collision chain

Exceptions
StreamConstraintsException:
if there are too many collisions (was an IllegalStateException before v2.15)
Since
2.1
_thresholdSizeback to summary
private static int _thresholdSize(int hashAreaSize)
bucketCountback to summary
public int bucketCount()

Method for checking number of primary hash buckets this symbol table uses.

Returns:int

number of primary slots table has currently

calcHashback to summary
public int calcHash(char[] buffer, int start, int len)

Implementation of a hashing method for variable length Strings. Most of the time intention is that this calculation is done by caller during parsing, not here; however, sometimes it needs to be done for parsed "String" too.

Parameters
buffer:char[]

Input buffer that contains name to decode

start:int

Pointer to the first character of the name

len:int

Length of String; has to be at least 1 (caller guarantees)

Returns:int

Hash code calculated

calcHashback to summary
public int calcHash(String key)
collisionCountback to summary
public int collisionCount()

Method mostly needed by unit tests; calculates number of entries that are in collision list. Value can be at most (size - 1), but should usually be much lower, ideally 0.

Returns:int

Number of collisions in the primary hash area

Since
2.1
copyArraysback to summary
private void copyArrays()

Method called when copy-on-write is needed; generally when first change is made to a derived symbol table.

createRootback to summary
public static CharsToNameCanonicalizer createRoot()

Deprecated

Since 2.16 use createRoot(TokenStreamFactory) instead

Returns:CharsToNameCanonicalizer

Root instance to use for constructing new child instances

Annotations
@Deprecated
createRootback to summary
public static CharsToNameCanonicalizer createRoot(int seed)

Deprecated

Since 2.16 use createRoot(TokenStreamFactory) instead

Parameters
seed:int

Seed for hash value calculation

Returns:CharsToNameCanonicalizer

Root instance to use for constructing new child instances

Annotations
@Deprecated
createRootback to summary
public static CharsToNameCanonicalizer createRoot(TokenStreamFactory owner)

Method called to create root canonicalizer for a com.fasterxml.jackson.core.JsonFactory instance. Root instance is never used directly; its main use is for storing and sharing underlying symbol arrays as needed.

Parameters
owner:TokenStreamFactory

Factory that will use the root instance; used for accessing configuration

Returns:CharsToNameCanonicalizer

Root instance to use for constructing new child instances

createRootback to summary
public static CharsToNameCanonicalizer createRoot(TokenStreamFactory owner, int seed)
findSymbolback to summary
public String findSymbol(char[] buffer, int start, int len, int h) throws IOException
hashSeedback to summary
public int hashSeed()
makeChildback to summary
public CharsToNameCanonicalizer makeChild()

"Factory" method; will create a new child instance of this symbol table. It will be a copy-on-write instance, ie. it will only use read-only copy of parent's data, but when changes are needed, a copy will be created.

Note

while this method is synchronized, it is generally not safe to both use makeChild/mergeChild, AND to use instance actively. Instead, a separate 'root' instance should be used on which only makeChild/mergeChild are called, but instance itself is not used as a symbol table.

Returns:CharsToNameCanonicalizer

Actual canonicalizer instance that can be used by a parser

makeChildback to summary
public CharsToNameCanonicalizer makeChild(int flags)

Deprecated

Since 2.16 use makeChild() instead.

Parameters
flags:int

Configuration flags (ignored)

Returns:CharsToNameCanonicalizer

Actual canonicalizer instance that can be used by a parser

Annotations
@Deprecated
maxCollisionLengthback to summary
public int maxCollisionLength()

Method mostly needed by unit tests; calculates length of the longest collision chain. This should typically be a low number, but may be up to size - 1 in the pathological case

Returns:int

Length of the collision chain

Since
2.1
maybeDirtyback to summary
public boolean maybeDirty()
mergeChildback to summary
private void mergeChild(CharsToNameCanonicalizer.TableInfo childState)

Method that allows contents of child table to potentially be "merged in" with contents of this symbol table.

Note that caller has to make sure symbol table passed in is really a child or sibling of this symbol table.

rehashback to summary
private void rehash() throws IOException

Method called when size (number of entries) of symbol table grows so big that load factor is exceeded. Since size has to remain power of two, arrays will then always be doubled. Main work is really redistributing old entries into new String/Bucket entries.

releaseback to summary
public void release()

Method called by the using code to indicate it is done with this instance. This lets instance merge accumulated changes into parent (if need be), safely and efficiently, and without calling code having to know about parent information.

sizeback to summary
public int size()
Returns:int

Number of symbol entries contained by this canonicalizer instance

verifyInternalConsistencyback to summary
protected void verifyInternalConsistency()

Diagnostics method that will verify that internal data structures are consistent; not meant as user-facing method but only for test suites and possible troubleshooting.

Since
2.10
com.fasterxml.jackson.core.sym back to summary

pack-priv final Class CharsToNameCanonicalizer.Bucket

extends Object
Class Inheritance
  • java.lang.Object
  • com.fasterxml.jackson.core.sym.CharsToNameCanonicalizer.Bucket

This class is a symbol table entry. Each entry acts as a node in a linked list.

Field Summary

Modifier and TypeField and Description
public final int
public final CharsToNameCanonicalizer.Bucket
public final String

Constructor Summary

AccessConstructor and Description
public

Method Summary

Modifier and TypeMethod and Description
public String
has(char[] buf, int start, int len)

Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Field Detail

lengthback to summary
public final int length
nextback to summary
public final CharsToNameCanonicalizer.Bucket next
symbolback to summary
public final String symbol

Constructor Detail

Bucketback to summary
public Bucket(String s, CharsToNameCanonicalizer.Bucket n)

Method Detail

hasback to summary
public String has(char[] buf, int start, int len)
com.fasterxml.jackson.core.sym back to summary

private final Class CharsToNameCanonicalizer.TableInfo

extends Object
Class Inheritance
  • java.lang.Object
  • com.fasterxml.jackson.core.sym.CharsToNameCanonicalizer.TableInfo

Immutable value class used for sharing information as efficiently as possible, by only require synchronization of reference manipulation but not access to contents.
Since
2.8.7

Field Summary

Modifier and TypeField and Description
pack-priv final CharsToNameCanonicalizer.Bucket[]
pack-priv final int
pack-priv final int
pack-priv final String[]

Constructor Summary

AccessConstructor and Description
public
TableInfo(int size, int longestCollisionList, String[] symbols, CharsToNameCanonicalizer.Bucket[] buckets)

public

Method Summary

Modifier and TypeMethod and Description
public static CharsToNameCanonicalizer.TableInfo
createInitial(int sz)

Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Field Detail

bucketsback to summary
pack-priv final CharsToNameCanonicalizer.Bucket[] buckets
longestCollisionListback to summary
pack-priv final int longestCollisionList
sizeback to summary
pack-priv final int size
symbolsback to summary
pack-priv final String[] symbols

Constructor Detail

TableInfoback to summary
public TableInfo(int size, int longestCollisionList, String[] symbols, CharsToNameCanonicalizer.Bucket[] buckets)
TableInfoback to summary
public TableInfo(CharsToNameCanonicalizer src)

Method Detail

createInitialback to summary
public static CharsToNameCanonicalizer.TableInfo createInitial(int sz)