Top Inners Fields Constructors Methods
com.ibm.icu.impl.coll

public final Class CollationBuilder

extends Sink
Class Inheritance
Imports
java.text.ParseException, com.ibm.icu.impl.Norm2AllModes, .Normalizer2Impl, .Normalizer2Impl.Hangul, com.ibm.icu.lang.UScript, com.ibm.icu.text.CanonicalIterator, .Collator, .Normalizer2, .UnicodeSet, .UnicodeSetIterator, com.ibm.icu.util.ICUInputTooLongException, .ULocale

Nested and Inner Type Summary

Modifier and TypeClass and Description
private static class
private static class

Field Summary

Modifier and TypeField and Description
private CollationTailoring
private CollationData
private long[]
private int
private static final UnicodeSet
private CollationDataBuilder
private static final boolean
private boolean
private Normalizer2
private static final int
HAS_BEFORE2

Node bit 6 is set on a primary node if there are nodes with secondary values below the common secondary weight (05).

private static final int
HAS_BEFORE3

Node bit 5 is set on a primary or secondary node if there are nodes with tertiary values below the common tertiary weight (05).

private static final int
IS_TAILORED

Node bit 3 distinguishes a tailored node, which has no weight value, from a node with an explicit (root or default) weight.

private static int
private static final int
MAX_INDEX

At most 1M nodes, limited by the 20 bits in node bit fields.

private Normalizer2Impl
private Normalizer2
private UVector64
nodes

Data structure for assigning tailored weights and CEs.

private UnicodeSet
private CollationRootElements
private UVector32
rootPrimaryIndexes

Indexes of nodes with root primary weights, sorted by primary.

private long

Constructor Summary

AccessConstructor and Description
public

Method Summary

Modifier and TypeMethod and Description
private int
addIfDifferent(CharSequence prefix, CharSequence str, long[] newCEs, int newCEsLength, int ce32)

private int
addOnlyClosure(CharSequence nfdPrefix, CharSequence nfdString, long[] newCEs, int newCEsLength, int ce32)

pack-priv void
addRelation(int strength, CharSequence prefix, CharSequence str, CharSequence extension)

Implements abstract com.ibm.icu.impl.coll.CollationRuleParser.Sink.addRelation.

Implements CollationRuleParser.Sink.
pack-priv void
addReset(int strength, CharSequence str)

Implements abstract com.ibm.icu.impl.coll.CollationRuleParser.Sink.addReset.

Implements CollationRuleParser.Sink.
private void
private int
addWithClosure(CharSequence nfdPrefix, CharSequence nfdString, long[] newCEs, int newCEsLength, int ce32)

Adds the mapping and its canonical closure.

private static final int
private static final int

Returns:

the index>=0 where the item was found, or the index<0 for inserting the string at ~index in sorted order (index into rootPrimaryIndexes)
binarySearchForRootPrimaryNode
(int[] rootPrimaryIndexes, int length, long[] nodes, long p)

Like Java Collections.binarySearch(List, key, Comparator).

private static int
ceStrength(long ce)

private static long
changeNodeNextIndex(long node, int next)

private static long
changeNodePreviousIndex(long node, int previous)

private void
private static int
countTailoredNodes(long[] nodesArray, int i, int strength)

Counts the tailored nodes of the given strength up to the next node which is either stronger or has an explicit weight of this strength.

private boolean
equalSubSequences(CharSequence left, int leftStart, CharSequence right, int rightStart)

private void
finalizeCEs()

Replaces temporary CEs with the final CEs they point to.

private int
findCommonNode(int index, int strength)

Finds the node which implies or contains a common=05 weight of the given strength (secondary or tertiary), if the current node is stronger.

private int
findOrInsertNodeForCEs(int strength)

Picks one of the current CEs and finds or inserts a node in the graph for the CE + strength.

private int
findOrInsertNodeForPrimary(long p)

Finds or inserts the node for a root CE's primary weight.

private int
findOrInsertNodeForRootCE(long ce, int strength)

private int
findOrInsertWeakNode(int index, int weight16, int level)

Finds or inserts the node for a secondary or tertiary weight.

private long
private int
getWeight16Before(int index, long node, int level)

Returns the secondary or tertiary weight preceding the current node's weight.

private boolean
private boolean
private static int
indexFromTempCE(long tempCE)

private static int
indexFromTempCE32(int tempCE32)

private int

Returns:

the new node's index
insertNodeBetween
(int index, int nextIndex, long node)

Inserts a new node into the list, between list-adjacent items.

private int

Returns:

the new node's index
insertTailoredNodeAfter
(int index, int strength)

Makes and inserts a new tailored node into the list, after the one at index.

private boolean
private static boolean
isTailoredNode(long node)

private static boolean
isTempCE(long ce)

private static boolean
isTempCE32(int ce32)

private void
makeTailoredCEs()

Walks the tailoring graph and overwrites tailored nodes with new CEs.

private boolean
mergeCompositeIntoString(CharSequence nfdString, int indexAfterLastStarter, int composite, CharSequence decomp, StringBuilder newNFDString, StringBuilder newString)

private static int
nextIndexFromNode(long node)

private static long
private static long
nodeFromPreviousIndex(int previous)

private static long
nodeFromStrength(int strength)

private static long
nodeFromWeight16(int weight16)

private static long
nodeFromWeight32(long weight32)

private static boolean
nodeHasAnyBefore(long node)

private static boolean
nodeHasBefore2(long node)

private static boolean
nodeHasBefore3(long node)

pack-priv void
optimize(UnicodeSet set)

Overrides com.ibm.icu.impl.coll.CollationRuleParser.Sink.optimize.

Implements CollationRuleParser.Sink.
public CollationTailoring
parseAndBuild(String ruleString)

private static int
private static boolean
sameCEs(long[] ces1, int ces1Length, long[] ces2, int ces2Length)

private void
private static int
strengthFromNode(long node)

private static int
strengthFromTempCE(long tempCE)

pack-priv void
private static long
tempCEFromIndexAndStrength(int index, int strength)

Encodes "temporary CE" data into a CE that fits into the CE32 data structure, with 2-byte primary, 1-byte secondary and 6-bit tertiary, with valid CE byte values.

private static int
weight16FromNode(long node)

private static long
weight32FromNode(long node)

Field Detail

baseback to summary
private CollationTailoring base
baseDataback to summary
private CollationData baseData
cesback to summary
private long[] ces
cesLengthback to summary
private int cesLength
COMPOSITESback to summary
private static final UnicodeSet COMPOSITES
dataBuilderback to summary
private CollationDataBuilder dataBuilder
DEBUGback to summary
private static final boolean DEBUG
fastLatinEnabledback to summary
private boolean fastLatinEnabled
fcdback to summary
private Normalizer2 fcd
HAS_BEFORE2back to summary
private static final int HAS_BEFORE2

Node bit 6 is set on a primary node if there are nodes with secondary values below the common secondary weight (05).

HAS_BEFORE3back to summary
private static final int HAS_BEFORE3

Node bit 5 is set on a primary or secondary node if there are nodes with tertiary values below the common tertiary weight (05).

IS_TAILOREDback to summary
private static final int IS_TAILORED

Node bit 3 distinguishes a tailored node, which has no weight value, from a node with an explicit (root or default) weight.

kClosureLoopLimitback to summary
private static int kClosureLoopLimit
MAX_INDEXback to summary
private static final int MAX_INDEX

At most 1M nodes, limited by the 20 bits in node bit fields.

nfcImplback to summary
private Normalizer2Impl nfcImpl
nfdback to summary
private Normalizer2 nfd
nodesback to summary
private UVector64 nodes

Data structure for assigning tailored weights and CEs. Doubly-linked lists of nodes in mostly collation order. Each list starts with a root primary node and ends with a nextIndex of 0. When there are any nodes in the list, then there is always a root primary node at index 0. This allows some code not to have to check explicitly for nextIndex==0. Root primary nodes have 32-bit weights but do not have previous indexes. All other nodes have at most 16-bit weights and do have previous indexes. Nodes with explicit weights store root collator weights, or default weak weights (e.g., secondary 05) for stronger nodes. "Tailored" nodes, with the IS_TAILORED bit set, do not store explicit weights but rather create a difference of a certain strength from the preceding node. A root node is followed by either - a root/default node of the same strength, or - a root/default node of the next-weaker strength, or - a tailored node of the same strength. A node of a given strength normally implies "common" weights on weaker levels. A node with HAS_BEFORE2 must be immediately followed by a secondary node with an explicit below-common weight, then a secondary tailored node, and later an explicit common-secondary node. The below-common weight can be a root weight, or it can be BEFORE_WEIGHT16 for tailoring before an implied common weight or before the lowest root weight. (&[before 2] resets to an explicit secondary node so that the following addRelation(secondary) tailors right after that. If we did not have this node and instead were to reset on the primary node, then addRelation(secondary) would skip forward to the the COMMON_WEIGHT16 node.) If the flag is not set, then there are no explicit secondary nodes with the common or lower weights. Same for HAS_BEFORE3 for tertiary nodes and weights. A node must not have both flags set. Tailored CEs are initially represented in a CollationDataBuilder as temporary CEs which point to stable indexes in this list, and temporary CEs stored in a CollationDataBuilder only point to tailored nodes. A temporary CE in the ces[] array may point to a non-tailored reset-before-position node, until the next relation is added. At the end, the tailored weights are allocated as necessary, then the tailored nodes are replaced with final CEs, and the CollationData is rewritten by replacing temporary CEs with final ones. We cannot simply insert new nodes in the middle of the array because that would invalidate the indexes stored in existing temporary CEs. We need to use a linked graph with stable indexes to existing nodes. A doubly-linked list seems easiest to maintain. Each node is stored as an long, with its fields stored as bit fields. Root primary node: - primary weight: 32 bits 63..32 - reserved/unused/zero: 4 bits 31..28 Weaker root nodes & tailored nodes: - a weight: 16 bits 63..48 + a root or default weight for a non-tailored node + unused/zero for a tailored node - index to the previous node: 20 bits 47..28 All types of nodes: - index to the next node: 20 bits 27..8 + nextIndex=0 in last node per root-primary list - reserved/unused/zero bits: bits 7, 4, 2 - HAS_BEFORE2: bit 6 - HAS_BEFORE3: bit 5 - IS_TAILORED: bit 3 - the difference strength (primary/secondary/tertiary/quaternary): 2 bits 1..0 We could allocate structs with pointers, but we would have to store them in a pointer list so that they can be indexed from temporary CEs, and they would require more memory allocations.

optimizeSetback to summary
private UnicodeSet optimizeSet
rootElementsback to summary
private CollationRootElements rootElements
rootPrimaryIndexesback to summary
private UVector32 rootPrimaryIndexes

Indexes of nodes with root primary weights, sorted by primary. Compact form of a TreeMap from root primary to node index. This is a performance optimization for finding reset positions. Without this, we would have to search through the entire nodes list. It also allows storing root primary weights in list head nodes, without previous index, leaving room in root primary nodes for 32-bit primary weights.

variableTopback to summary
private long variableTop

Constructor Detail

CollationBuilderback to summary
public CollationBuilder(CollationTailoring b)

Method Detail

addIfDifferentback to summary
private int addIfDifferent(CharSequence prefix, CharSequence str, long[] newCEs, int newCEsLength, int ce32)
addOnlyClosureback to summary
private int addOnlyClosure(CharSequence nfdPrefix, CharSequence nfdString, long[] newCEs, int newCEsLength, int ce32)
addRelationback to summary
pack-priv void addRelation(int strength, CharSequence prefix, CharSequence str, CharSequence extension)

Implements abstract com.ibm.icu.impl.coll.CollationRuleParser.Sink.addRelation.

Implements CollationRuleParser.Sink.

Annotations
@Override
addResetback to summary
pack-priv void addReset(int strength, CharSequence str)

Implements abstract com.ibm.icu.impl.coll.CollationRuleParser.Sink.addReset.

Implements CollationRuleParser.Sink.

Annotations
@Override
addTailCompositesback to summary
private void addTailComposites(CharSequence nfdPrefix, CharSequence nfdString)
addWithClosureback to summary
private int addWithClosure(CharSequence nfdPrefix, CharSequence nfdString, long[] newCEs, int newCEsLength, int ce32)

Adds the mapping and its canonical closure. Takes ce32=dataBuilder.encodeCEs(...) so that the data builder need not re-encode the CEs multiple times.

alignWeightRightback to summary
private static final int alignWeightRight(int w)
binarySearchForRootPrimaryNodeback to summary
private static final int binarySearchForRootPrimaryNode(int[] rootPrimaryIndexes, int length, long[] nodes, long p)

Like Java Collections.binarySearch(List, key, Comparator).

Returns:int

the index>=0 where the item was found, or the index<0 for inserting the string at ~index in sorted order (index into rootPrimaryIndexes)

ceStrengthback to summary
private static int ceStrength(long ce)
changeNodeNextIndexback to summary
private static long changeNodeNextIndex(long node, int next)
changeNodePreviousIndexback to summary
private static long changeNodePreviousIndex(long node, int previous)
closeOverCompositesback to summary
private void closeOverComposites()
countTailoredNodesback to summary
private static int countTailoredNodes(long[] nodesArray, int i, int strength)

Counts the tailored nodes of the given strength up to the next node which is either stronger or has an explicit weight of this strength.

equalSubSequencesback to summary
private boolean equalSubSequences(CharSequence left, int leftStart, CharSequence right, int rightStart)
finalizeCEsback to summary
private void finalizeCEs()

Replaces temporary CEs with the final CEs they point to.

findCommonNodeback to summary
private int findCommonNode(int index, int strength)

Finds the node which implies or contains a common=05 weight of the given strength (secondary or tertiary), if the current node is stronger. Skips weaker nodes and tailored nodes if the current node is stronger and is followed by an explicit-common-weight node. Always returns the input index if that node is no stronger than the given strength.

findOrInsertNodeForCEsback to summary
private int findOrInsertNodeForCEs(int strength)

Picks one of the current CEs and finds or inserts a node in the graph for the CE + strength.

findOrInsertNodeForPrimaryback to summary
private int findOrInsertNodeForPrimary(long p)

Finds or inserts the node for a root CE's primary weight.

findOrInsertNodeForRootCEback to summary
private int findOrInsertNodeForRootCE(long ce, int strength)
findOrInsertWeakNodeback to summary
private int findOrInsertWeakNode(int index, int weight16, int level)

Finds or inserts the node for a secondary or tertiary weight.

getSpecialResetPositionback to summary
private long getSpecialResetPosition(CharSequence str)
getWeight16Beforeback to summary
private int getWeight16Before(int index, long node, int level)

Returns the secondary or tertiary weight preceding the current node's weight. node=nodes[index].

ignorePrefixback to summary
private boolean ignorePrefix(CharSequence s)
ignoreStringback to summary
private boolean ignoreString(CharSequence s)
indexFromTempCEback to summary
private static int indexFromTempCE(long tempCE)
indexFromTempCE32back to summary
private static int indexFromTempCE32(int tempCE32)
insertNodeBetweenback to summary
private int insertNodeBetween(int index, int nextIndex, long node)

Inserts a new node into the list, between list-adjacent items. The node's previous and next indexes must not be set yet.

Returns:int

the new node's index

insertTailoredNodeAfterback to summary
private int insertTailoredNodeAfter(int index, int strength)

Makes and inserts a new tailored node into the list, after the one at index. Skips over nodes of weaker strength to maintain collation order ("postpone insertion").

Returns:int

the new node's index

isFCDback to summary
private boolean isFCD(CharSequence s)
isTailoredNodeback to summary
private static boolean isTailoredNode(long node)
isTempCEback to summary
private static boolean isTempCE(long ce)
isTempCE32back to summary
private static boolean isTempCE32(int ce32)
makeTailoredCEsback to summary
private void makeTailoredCEs()

Walks the tailoring graph and overwrites tailored nodes with new CEs. After this, the graph is destroyed. The nodes array can then be used only as a source of tailored CEs.

mergeCompositeIntoStringback to summary
private boolean mergeCompositeIntoString(CharSequence nfdString, int indexAfterLastStarter, int composite, CharSequence decomp, StringBuilder newNFDString, StringBuilder newString)
nextIndexFromNodeback to summary
private static int nextIndexFromNode(long node)
nodeFromNextIndexback to summary
private static long nodeFromNextIndex(int next)
nodeFromPreviousIndexback to summary
private static long nodeFromPreviousIndex(int previous)
nodeFromStrengthback to summary
private static long nodeFromStrength(int strength)
nodeFromWeight16back to summary
private static long nodeFromWeight16(int weight16)
nodeFromWeight32back to summary
private static long nodeFromWeight32(long weight32)
nodeHasAnyBeforeback to summary
private static boolean nodeHasAnyBefore(long node)
nodeHasBefore2back to summary
private static boolean nodeHasBefore2(long node)
nodeHasBefore3back to summary
private static boolean nodeHasBefore3(long node)
optimizeback to summary
pack-priv void optimize(UnicodeSet set)

Overrides com.ibm.icu.impl.coll.CollationRuleParser.Sink.optimize.

Implements CollationRuleParser.Sink.

Annotations
@Override
parseAndBuildback to summary
public CollationTailoring parseAndBuild(String ruleString) throws ParseException
previousIndexFromNodeback to summary
private static int previousIndexFromNode(long node)
sameCEsback to summary
private static boolean sameCEs(long[] ces1, int ces1Length, long[] ces2, int ces2Length)
setCaseBitsback to summary
private void setCaseBits(CharSequence nfdString)
strengthFromNodeback to summary
private static int strengthFromNode(long node)
strengthFromTempCEback to summary
private static int strengthFromTempCE(long tempCE)
suppressContractionsback to summary
pack-priv void suppressContractions(UnicodeSet set)

Overrides com.ibm.icu.impl.coll.CollationRuleParser.Sink.suppressContractions.

Implements CollationRuleParser.Sink.

Annotations
@Override
tempCEFromIndexAndStrengthback to summary
private static long tempCEFromIndexAndStrength(int index, int strength)

Encodes "temporary CE" data into a CE that fits into the CE32 data structure, with 2-byte primary, 1-byte secondary and 6-bit tertiary, with valid CE byte values. The index must not exceed 20 bits (0xfffff). The strength must fit into 2 bits (Collator.PRIMARY..Collator.QUATERNARY). Temporary CEs are distinguished from real CEs by their use of secondary weights 06..45 which are otherwise reserved for compressed sort keys. The case bits are unused and available.

weight16FromNodeback to summary
private static int weight16FromNode(long node)
weight32FromNodeback to summary
private static long weight32FromNode(long node)
com.ibm.icu.impl.coll back to summary

private final Class CollationBuilder.BundleImporter

extends Object
implements Importer
Class Inheritance
All Implemented Interfaces
com.ibm.icu.impl.coll.CollationRuleParser.Importer

Constructor Summary

AccessConstructor and Description
pack-priv

Method Summary

Modifier and TypeMethod and Description
public String
Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Constructor Detail

BundleImporterback to summary
pack-priv BundleImporter()

Method Detail

getRulesback to summary
public String getRules(String localeID, String collationType)

Implements com.ibm.icu.impl.coll.CollationRuleParser.Importer.getRules.

Annotations
@Override
com.ibm.icu.impl.coll back to summary

private final Class CollationBuilder.CEFinalizer

extends Object
implements CEModifier
Class Inheritance
All Implemented Interfaces
com.ibm.icu.impl.coll.CollationDataBuilder.CEModifier

Field Summary

Modifier and TypeField and Description
private long[]

Constructor Summary

AccessConstructor and Description
pack-priv
CEFinalizer(long[] ces)

Method Summary

Modifier and TypeMethod and Description
public long
modifyCE(long ce)

Implements com.ibm.icu.impl.coll.CollationDataBuilder.CEModifier.modifyCE.

Returns a new CE to replace the input CE, or else Collation.NO_CE.
public long
modifyCE32(int ce32)

Implements com.ibm.icu.impl.coll.CollationDataBuilder.CEModifier.modifyCE32.

Returns a new CE to replace the non-special input CE32, or else Collation.NO_CE.
Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Field Detail

finalCEsback to summary
private long[] finalCEs

Constructor Detail

CEFinalizerback to summary
pack-priv CEFinalizer(long[] ces)

Method Detail

modifyCEback to summary
public long modifyCE(long ce)

Implements com.ibm.icu.impl.coll.CollationDataBuilder.CEModifier.modifyCE.

Doc from com.ibm.icu.impl.coll.CollationDataBuilder.CEModifier.modifyCE.

Returns a new CE to replace the input CE, or else Collation.NO_CE.

Annotations
@Override
modifyCE32back to summary
public long modifyCE32(int ce32)

Implements com.ibm.icu.impl.coll.CollationDataBuilder.CEModifier.modifyCE32.

Doc from com.ibm.icu.impl.coll.CollationDataBuilder.CEModifier.modifyCE32.

Returns a new CE to replace the non-special input CE32, or else Collation.NO_CE.

Annotations
@Override