OpenJDK 27 · sun.text.DictionaryBasedBreakIterator · APIdia, the JavaDoc alternative

A subclass of RuleBasedBreakIterator that adds the ability to use a dictionary to further subdivide ranges of text beyond what is possible using just the state-table-based algorithm. This is necessary, for example, to handle word and line breaking in Thai, which doesn't use spaces between words. The state-table-based algorithm used by RuleBasedBreakIterator is used to divide up text as far as possible, and then contiguous ranges of letters are repeatedly compared against a list of known words (i.e., the dictionary) to divide them up into words. DictionaryBasedBreakIterator uses the same rule language as RuleBasedBreakIterator, but adds one more special substitution name: <dictionary>. This substitution name is used to identify characters in words in the dictionary. The idea is that if the iterator passes over a chunk of text that includes two or more characters in a row that are included in <dictionary>, it goes back through that range and derives additional break positions (if possible) using the dictionary. DictionaryBasedBreakIterator is also constructed with the filename of a dictionary file. It follows a prescribed search path to locate the dictionary (right now, it looks for it in /com/ibm/text/resources in each directory in the classpath, and won't find it in JAR files, but this location is likely to change). The dictionary file is in a serialized binary format. We have a very primitive (and slow) BuildDictionaryFile utility for creating dictionary files, but aren't currently making it public. Contact us for help.

Field Summary

Modifier and Type	Field and Description
private int[]	cachedBreakPositions when a range of characters is divided up using the dictionary, the break positions that are discovered are stored here, preventing us from having to use either the dictionary or the state table again until the iterator leaves this range of text
private boolean[]	categoryFlags a list of flags indicating which character categories are contained in the dictionary file (this is used to determine which ranges of characters to apply the dictionary to)
private BreakDictionary	dictionary a list of known words that is used to divide up contiguous ranges of letters, stored in a compressed, indexed, format that offers fast access
private int	dictionaryCharCount a temporary hiding place for the number of dictionary characters in the last range passed over by next()
private int	positionInCache if cachedBreakPositions is not null, this indicates which item in the cache the current iteration position refers to

Inherited from sun.text.RuleBasedBreakIterator:: IGNORE LABEL LABEL_LENGTH supportedVersion

Constructor Summary

Access	Constructor and Description
public	DictionaryBasedBreakIterator(String the name of the rule data file ruleFile, byte[] the rule data loaded from the rule data file ruleData, String the name of the dictionary file dictionaryFile, byte[] the dictionary data loaded from the dictionary file dictionaryData) Constructs a DictionaryBasedBreakIterator.

Method Summary

Modifier and Type	Method and Description
private void	divideUpDictionaryRange(int startPos, int endPos) This is the function that actually implements the dictionary-based algorithm.
public int	Returns: The offset of the beginning of the text. first() Overrides sun.text.RuleBasedBreakIterator.first. Sets the current iteration position to the beginning of the text.
public int	Returns: The position of the first boundary after "offset" following(int The position to begin searching forward from offset) Overrides sun.text.RuleBasedBreakIterator.following. Sets the current iteration position to the first boundary position after the specified position.
protected int	handleNext() Overrides sun.text.RuleBasedBreakIterator.handleNext. This is the implementation function for next().
public int	Returns: The text's past-the-end offset. last() Overrides sun.text.RuleBasedBreakIterator.last. Sets the current iteration position to the end of the text.
protected int	lookupCategory(int c) Overrides sun.text.RuleBasedBreakIterator.lookupCategory. Looks up a character category for a character.
public int	Returns: The position of the last boundary before "offset" preceding(int The position to begin searching from offset) Overrides sun.text.RuleBasedBreakIterator.preceding. Sets the current iteration position to the last boundary position before the specified position.
private void	prepareCategoryFlags(byte[] data)
public int	Returns: The position of the last boundary position before the current iteration position previous() Overrides sun.text.RuleBasedBreakIterator.previous. Advances the iterator one step backwards.
public void	setText(CharacterIterator newText) Overrides sun.text.RuleBasedBreakIterator.setText. Set the iterator to analyze a new piece of text.

Inherited from sun.text.RuleBasedBreakIterator:: checkOffset clone current equals getAdditionalData getCurrent getNext getText handlePrevious hashCode isBoundary lookupBackwardState lookupState next next setAdditionalData toString validateRuleData

Field Detail

cachedBreakPositions	back to summary
private int[] cachedBreakPositions when a range of characters is divided up using the dictionary, the break positions that are discovered are stored here, preventing us from having to use either the dictionary or the state table again until the iterator leaves this range of text

categoryFlags	back to summary
private boolean[] categoryFlags a list of flags indicating which character categories are contained in the dictionary file (this is used to determine which ranges of characters to apply the dictionary to)

dictionary	back to summary
private BreakDictionary dictionary a list of known words that is used to divide up contiguous ranges of letters, stored in a compressed, indexed, format that offers fast access

dictionaryCharCount	back to summary
private int dictionaryCharCount a temporary hiding place for the number of dictionary characters in the last range passed over by next()

positionInCache	back to summary
private int positionInCache if cachedBreakPositions is not null, this indicates which item in the cache the current iteration position refers to

Constructor Detail

DictionaryBasedBreakIterator	back to summary
public DictionaryBasedBreakIterator(String ruleFile, byte[] ruleData, String dictionaryFile, byte[] dictionaryData) Constructs a DictionaryBasedBreakIterator. Parameters ruleFile:String the name of the rule data file ruleData:byte[] the rule data loaded from the rule data file dictionaryFile:String the name of the dictionary file dictionaryData:byte[] the dictionary data loaded from the dictionary file Exceptions MissingResourceException: if rule data or dictionary initialization failed

DictionaryBasedBreakIterator

back to summary

public DictionaryBasedBreakIterator(String ruleFile, byte[] ruleData, String dictionaryFile, byte[] dictionaryData)

Constructs a DictionaryBasedBreakIterator.

Parameters

ruleFile:String: the name of the rule data file
ruleData:byte[]: the rule data loaded from the rule data file
dictionaryFile:String: the name of the dictionary file
dictionaryData:byte[]: the dictionary data loaded from the dictionary file

Exceptions

MissingResourceException:: if rule data or dictionary initialization failed

Method Detail

divideUpDictionaryRange	back to summary
private void divideUpDictionaryRange(int startPos, int endPos) This is the function that actually implements the dictionary-based algorithm. Given the endpoints of a range of text, it uses the dictionary to determine the positions of any boundaries in this range. It stores all the boundary positions it discovers in cachedBreakPositions so that we only have to do this work once for each time we enter the range.

divideUpDictionaryRange

back to summary

private void divideUpDictionaryRange(int startPos, int endPos)

This is the function that actually implements the dictionary-based algorithm. Given the endpoints of a range of text, it uses the dictionary to determine the positions of any boundaries in this range. It stores all the boundary positions it discovers in cachedBreakPositions so that we only have to do this work once for each time we enter the range.

first	back to summary
public int first() Overrides sun.text.RuleBasedBreakIterator.first. Sets the current iteration position to the beginning of the text. (i.e., the CharacterIterator's starting offset). Returns:int The offset of the beginning of the text. Annotations @Override

first

back to summary

public int first()

Overrides sun.text.RuleBasedBreakIterator.first.

Sets the current iteration position to the beginning of the text. (i.e., the CharacterIterator's starting offset).

Returns:int

The offset of the beginning of the text.

Annotations

@Override

following	back to summary
public int following(int offset) Overrides sun.text.RuleBasedBreakIterator.following. Sets the current iteration position to the first boundary position after the specified position. Parameters offset:int The position to begin searching forward from Returns:int The position of the first boundary after "offset" Annotations @Override

following

back to summary

public int following(int offset)

Overrides sun.text.RuleBasedBreakIterator.following.

Sets the current iteration position to the first boundary position after the specified position.

Parameters

offset:int: The position to begin searching forward from

Returns:int

The position of the first boundary after "offset"

Annotations

@Override

handleNext	back to summary
protected int handleNext() Overrides sun.text.RuleBasedBreakIterator.handleNext. This is the implementation function for next(). Annotations @Override

handleNext

back to summary

protected int handleNext()

Overrides sun.text.RuleBasedBreakIterator.handleNext.

This is the implementation function for next().

Annotations

@Override

last	back to summary
public int last() Overrides sun.text.RuleBasedBreakIterator.last. Sets the current iteration position to the end of the text. (i.e., the CharacterIterator's ending offset). Returns:int The text's past-the-end offset. Annotations @Override

last

back to summary

public int last()

Overrides sun.text.RuleBasedBreakIterator.last.

Sets the current iteration position to the end of the text. (i.e., the CharacterIterator's ending offset).

Returns:int

The text's past-the-end offset.

Annotations

@Override

lookupCategory	back to summary
protected int lookupCategory(int c) Overrides sun.text.RuleBasedBreakIterator.lookupCategory. Looks up a character category for a character. Annotations @Override

lookupCategory

back to summary

protected int lookupCategory(int c)

Overrides sun.text.RuleBasedBreakIterator.lookupCategory.

Looks up a character category for a character.

Annotations

@Override

preceding	back to summary
public int preceding(int offset) Overrides sun.text.RuleBasedBreakIterator.preceding. Sets the current iteration position to the last boundary position before the specified position. Parameters offset:int The position to begin searching from Returns:int The position of the last boundary before "offset" Annotations @Override

preceding

back to summary

public int preceding(int offset)

Overrides sun.text.RuleBasedBreakIterator.preceding.

Sets the current iteration position to the last boundary position before the specified position.

Parameters

offset:int: The position to begin searching from

Returns:int

The position of the last boundary before "offset"

Annotations

@Override

prepareCategoryFlags	back to summary
private void prepareCategoryFlags(byte[] data)

previous	back to summary
public int previous() Overrides sun.text.RuleBasedBreakIterator.previous. Advances the iterator one step backwards. Returns:int The position of the last boundary position before the current iteration position Annotations @Override

back to summary

public int previous()

Overrides sun.text.RuleBasedBreakIterator.previous.

Advances the iterator one step backwards.

Returns:int

The position of the last boundary position before the current iteration position

Annotations

@Override

setText	back to summary
public void setText(CharacterIterator newText) Overrides sun.text.RuleBasedBreakIterator.setText. Doc from sun.text.RuleBasedBreakIterator.setText. Set the iterator to analyze a new piece of text. This function resets the current iteration position to the beginning of the text. Annotations @Override

setText

back to summary

public void setText(CharacterIterator newText)

Overrides sun.text.RuleBasedBreakIterator.setText.

Doc from sun.text.RuleBasedBreakIterator.setText.

Set the iterator to analyze a new piece of text. This function resets the current iteration position to the beginning of the text.

Annotations

@Override

public Class DictionaryBasedBreakIterator

Field Summary

Constructor Summary

Method Summary

Field Detail

Constructor Detail

Method Detail