APIdia Launch Partner
Top Description Inners Fields Constructors Methods
sun.text

public Class RuleBasedBreakIterator

extends BreakIterator
Class Inheritance
Known Direct Subclasses
sun.text.DictionaryBasedBreakIterator
Imports
java.nio.BufferUnderflowException, .ByteBuffer, java.text.BreakIterator, .CharacterIterator, .StringCharacterIterator, java.util.MissingResourceException, .Objects, sun.text.CompactByteArray, .SupplementaryCharacterData

A subclass of BreakIterator whose behavior is specified using a list of rules.

There are two kinds of rules, which are separated by semicolons: substitutions and regular expressions.

A substitution rule defines a name that can be used in place of an expression. It consists of a name, which is a string of characters contained in angle brackets, an equals sign, and an expression. (There can be no whitespace on either side of the equals sign.) To keep its syntactic meaning intact, the expression must be enclosed in parentheses or square brackets. A substitution is visible after its definition, and is filled in using simple textual substitution. Substitution definitions can contain other substitutions, as long as those substitutions have been defined first. Substitutions are generally used to make the regular expressions (which can get quite complex) shorted and easier to read. They typically define either character categories or commonly-used subexpressions.

There is one special substitution.  If the description defines a substitution called "<ignore>", the expression must be a [] expression, and the expression defines a set of characters (the "ignore characters") that will be transparent to the BreakIterator.  A sequence of characters will break the same way it would if any ignore characters it contains are taken out.  Break positions never occur before ignore characters.

A regular expression uses a subset of the normal Unix regular-expression syntax, and defines a sequence of characters to be kept together. With one significant exception, the iterator uses a longest-possible-match algorithm when matching text to regular expressions. The iterator also treats descriptions containing multiple regular expressions as if they were ORed together (i.e., as if they were separated by |).

The special characters recognized by the regular-expression parser are as follows:

* Specifies that the expression preceding the asterisk may occur any number of times (including not at all).
{} Encloses a sequence of characters that is optional.
() Encloses a sequence of characters.  If followed by *, the sequence repeats.  Otherwise, the parentheses are just a grouping device and a way to delimit the ends of expressions containing |.
| Separates two alternative sequences of characters.  Either one sequence or the other, but not both, matches this expression.  The | character can only occur inside ().
. Matches any character.
*? Specifies a non-greedy asterisk.  *? works the same way as *, except when there is overlap between the last group of characters in the expression preceding the * and the first group of characters following the *.  When there is this kind of overlap, * will match the longest sequence of characters that match the expression before the *, and *? will match the shortest sequence of characters matching the expression before the *?.  For example, if you have "xxyxyyyxyxyxxyxyxyy" in the text, "x[xy]*x" will match through to the last x (i.e., "xxyxyyyxyxyxxyxyxyy", but "x[xy]*?x" will only match the first two xes ("xxyxyyyxyxyxxyxyxyy").
[] Specifies a group of alternative characters.  A [] expression will match any single character that is specified in the [] expression.  For more on the syntax of [] expressions, see below.
/ Specifies where the break position should go if text matches this expression.  (e.g., "[a-z]*/[:Zs:]*[1-0]" will match if the iterator sees a run of letters, followed by a run of whitespace, followed by a digit, but the break position will actually go before the whitespace).  Expressions that don't contain / put the break position at the end of the matching text.
\ Escape character.  The \ itself is ignored, but causes the next character to be treated as literal character.  This has no effect for many characters, but for the characters listed above, this deprives them of their special meaning.  (There are no special escape sequences for Unicode characters, or tabs and newlines; these are all handled by a higher-level protocol.  In a Java string, "\n" will be converted to a literal newline character by the time the regular-expression parser sees it.  Of course, this means that \ sequences that are visible to the regexp parser must be written as \\ when inside a Java string.)  All characters in the ASCII range except for letters, digits, and control characters are reserved characters to the parser and must be preceded by \ even if they currently don't mean anything.
! If ! appears at the beginning of a regular expression, it tells the regexp parser that this expression specifies the backwards-iteration behavior of the iterator, and not its normal iteration behavior.  This is generally only used in situations where the automatically-generated backwards-iteration behavior doesn't produce satisfactory results and must be supplemented with extra client-specified rules.
(all others) All other characters are treated as literal characters, which must match the corresponding character(s) in the text exactly.

Within a [] expression, a number of other special characters can be used to specify groups of characters:

- Specifies a range of matching characters.  For example "[a-p]" matches all lowercase Latin letters from a to p (inclusive).  The - sign specifies ranges of continuous Unicode numeric values, not ranges of characters in a language's alphabetical order: "[a-z]" doesn't include capital letters, nor does it include accented letters such as a-umlaut.
:: A pair of colons containing a one- or two-letter code matches all characters in the corresponding Unicode category.  The two-letter codes are the same as the two-letter codes in the Unicode database (for example, "[:Sc::Sm:]" matches all currency symbols and all math symbols).  Specifying a one-letter code is the same as specifying all two-letter codes that begin with that letter (for example, "[:L:]" matches all letters, and is equivalent to "[:Lu::Ll::Lo::Lm::Lt:]").  Anything other than a valid two-letter Unicode category code or a single letter that begins a Unicode category code is illegal within colons.
[] [] expressions can nest.  This has no effect, except when used in conjunction with the ^ token.
^ Excludes the character (or the characters in the [] expression) following it from the group of characters.  For example, "[a-z^p]" matches all Latin lowercase letters except p.  "[:L:^[\u4e00-\u9fff]]" matches all letters except the Han ideographs.
(all others) All other characters are treated as literal characters.  (For example, "[aeiou]" specifies just the letters a, e, i, o, and u.)

For a more complete explanation, see Text Boundary Analysis in Java by Richard Gillam.   For examples, see the resource data (which is annotated).

Author
Richard Gillam

Nested and Inner Type Summary

Modifier and TypeClass and Description
private static class

Field Summary

Modifier and TypeField and Description
private byte[]
additionalData

A table for additional data.

private short[]
backwardsStateTable

The table of state transitions used to sync up the iterator with the text in backwards and random-access iteration

private static final int
BMP_INDICES_LENGTH

An array length of indices for BMP characters

private int
private CompactByteArray
charCategoryTable

Tables that indexes from character values to character category numbers

private long
checksum

A CRC32 value of all data in datafile

private boolean[]
endStates

A list of flags indicating which states in the state table are accepting ("end") states

protected static final byte
IGNORE

A token used as a character-category value to identify ignore characters

pack-priv static final byte[]
LABEL

Magic number for the BreakIterator data file format.

pack-priv static final int
private boolean[]
lookaheadStates

A list of flags indicating which states in the state table are lookahead states (states which turn lookahead on and off)

private int
numCategories

The number of character categories (and, thus, the number of columns in the state tables)

private static final short
START_STATE

The state number of the starting state

private short[]
stateTable

The table of state transitions used for forward iteration

private static final short
STOP_STATE

The state-transition value indicating "stop"

private SupplementaryCharacterData
pack-priv static final byte
supportedVersion

Version number of the dictionary that was read in.

private CharacterIterator
text

The character iterator through which this BreakIterator accesses the text

Inherited from java.text.BreakIterator:
DONE

Constructor Summary

AccessConstructor and Description
public
RuleBasedBreakIterator(String ruleFile, byte[] ruleData)

Constructs a RuleBasedBreakIterator using the given rule data.

Method Summary

Modifier and TypeMethod and Description
protected static final void
checkOffset(int offset, CharacterIterator text)

Throw IllegalArgumentException unless begin <= offset < end.

public Object

Returns:

A newly-constructed RuleBasedBreakIterator with the same behavior as this one.
clone
()

Overrides java.text.BreakIterator.clone.

Clones this iterator.

public int

Returns:

The current iteration position.
current
()

Implements abstract java.text.BreakIterator.current.

Returns the current iteration position.

public boolean
equals(Object that)

Overrides java.lang.Object.equals.

Returns true if both BreakIterators are of the same class, have the same rules, and iterate over the same text.

public int

Returns:

The offset of the beginning of the text.
first
()

Implements abstract java.text.BreakIterator.first.

Sets the current iteration position to the beginning of the text.

public int

Returns:

The position of the first break after the current position.
following
(int offset)

Implements abstract java.text.BreakIterator.following.

Sets the iterator to refer to the first boundary position following the specified position.

pack-priv byte[]
pack-priv int
getCurrent()

Returns current character

private int
getCurrentCodePointCount()

Returns the count of next character.

pack-priv int
getNext()

Returns next character

private int
getNextIndex()

Returns the position of next character.

private int
getPrevious()

Returns previous character

public CharacterIterator

Returns:

An iterator over the text being analyzed.
getText
()

Implements abstract java.text.BreakIterator.getText.

Return a CharacterIterator over the text being analyzed.

protected int
handleNext()

This method is the actual implementation of the next() method.

protected int
handlePrevious()

This method backs the iterator back up to a "safe position" in the text.

public int

Returns:

hashcode for this BreakIterator
hashCode
()

Overrides java.lang.Object.hashCode.

Returns hashcode for this BreakIterator.

public boolean

Returns:

True if "offset" is a boundary position.
isBoundary
(int
the offset to check.
offset
)

Overrides java.text.BreakIterator.isBoundary.

Returns true if the specified position is a boundary position.

public int

Returns:

The text's past-the-end offset.
last
()

Implements abstract java.text.BreakIterator.last.

Sets the current iteration position to the end of the text.

protected int
lookupBackwardState(int state, int category)

Given a current state and a character category, looks up the next state to transition to in the backwards state table.

protected int
lookupCategory(int c)

Looks up a character's category (i.e., its category for breaking purposes, not its Unicode category)

protected int
lookupState(int state, int category)

Given a current state and a character category, looks up the next state to transition to in the state table.

public int

Returns:

The character offset of the boundary position n boundaries away from the current one.
next
(int
The number of steps to move. The sign indicates the direction (negative is backwards, and positive is forwards).
n
)

Implements abstract java.text.BreakIterator.next.

Advances the iterator either forward or backward the specified number of steps.

public int

Returns:

The position of the first boundary after this one.
next
()

Implements abstract java.text.BreakIterator.next.

Advances the iterator to the next boundary position.

public int

Returns:

The position of the last boundary before the starting position.
preceding
(int offset)

Overrides java.text.BreakIterator.preceding.

Sets the iterator to refer to the last boundary position before the specified position.

public int

Returns:

The position of the last boundary position preceding this one.
previous
()

Implements abstract java.text.BreakIterator.previous.

Advances the iterator backwards, to the last boundary preceding this one.

pack-priv void
public void
setText(CharacterIterator
An iterator over the text to analyze.
newText
)

Implements abstract java.text.BreakIterator.setText.

Set the iterator to analyze a new piece of text.

private void
setupTables(String ruleFile, ByteBuffer bb)

Initializes the fields with the given rule data.

public String
toString()

Overrides java.lang.Object.toString.

Returns text

pack-priv void
validateRuleData(String ruleFile, ByteBuffer bb)

Validates the magic number, version, and the length of the given data.

Inherited from java.text.BreakIterator:
getAvailableLocalesgetCharacterInstancegetCharacterInstancegetLineInstancegetLineInstancegetSentenceInstancegetSentenceInstancegetWordInstancegetWordInstancesetText

Field Detail

additionalDataback to summary
private byte[] additionalData

A table for additional data. May be used by a subclass of RuleBasedBreakIterator.

backwardsStateTableback to summary
private short[] backwardsStateTable

The table of state transitions used to sync up the iterator with the text in backwards and random-access iteration

BMP_INDICES_LENGTHback to summary
private static final int BMP_INDICES_LENGTH

An array length of indices for BMP characters

cachedLastKnownBreakback to summary
private int cachedLastKnownBreak
charCategoryTableback to summary
private CompactByteArray charCategoryTable

Tables that indexes from character values to character category numbers

checksumback to summary
private long checksum

A CRC32 value of all data in datafile

endStatesback to summary
private boolean[] endStates

A list of flags indicating which states in the state table are accepting ("end") states

IGNOREback to summary
protected static final byte IGNORE

A token used as a character-category value to identify ignore characters

LABELback to summary
pack-priv static final byte[] LABEL

Magic number for the BreakIterator data file format.

LABEL_LENGTHback to summary
pack-priv static final int LABEL_LENGTH
lookaheadStatesback to summary
private boolean[] lookaheadStates

A list of flags indicating which states in the state table are lookahead states (states which turn lookahead on and off)

numCategoriesback to summary
private int numCategories

The number of character categories (and, thus, the number of columns in the state tables)

START_STATEback to summary
private static final short START_STATE

The state number of the starting state

stateTableback to summary
private short[] stateTable

The table of state transitions used for forward iteration

STOP_STATEback to summary
private static final short STOP_STATE

The state-transition value indicating "stop"

supplementaryCharCategoryTableback to summary
private SupplementaryCharacterData supplementaryCharCategoryTable
supportedVersionback to summary
pack-priv static final byte supportedVersion

Version number of the dictionary that was read in.

textback to summary
private CharacterIterator text

The character iterator through which this BreakIterator accesses the text

Constructor Detail

RuleBasedBreakIteratorback to summary
public RuleBasedBreakIterator(String ruleFile, byte[] ruleData)

Constructs a RuleBasedBreakIterator using the given rule data.

Exceptions
MissingResourceException:
if the rule data is invalid or corrupted

Method Detail

checkOffsetback to summary
protected static final void checkOffset(int offset, CharacterIterator text)

Throw IllegalArgumentException unless begin <= offset < end.

cloneback to summary
public Object clone()

Overrides java.text.BreakIterator.clone.

Clones this iterator.

Returns:Object

A newly-constructed RuleBasedBreakIterator with the same behavior as this one.

Annotations
@Override
currentback to summary
public int current()

Implements abstract java.text.BreakIterator.current.

Returns the current iteration position.

Returns:int

The current iteration position.

Annotations
@Override
equalsback to summary
public boolean equals(Object that)

Overrides java.lang.Object.equals.

Returns true if both BreakIterators are of the same class, have the same rules, and iterate over the same text.

Annotations
@Override
firstback to summary
public int first()

Implements abstract java.text.BreakIterator.first.

Sets the current iteration position to the beginning of the text. (i.e., the CharacterIterator's starting offset).

Returns:int

The offset of the beginning of the text.

Annotations
@Override
followingback to summary
public int following(int offset)

Implements abstract java.text.BreakIterator.following.

Sets the iterator to refer to the first boundary position following the specified position.

Returns:int

The position of the first break after the current position.

Annotations
@Override
getAdditionalDataback to summary
pack-priv byte[] getAdditionalData()
getCurrentback to summary
pack-priv int getCurrent()

Returns current character

getCurrentCodePointCountback to summary
private int getCurrentCodePointCount()

Returns the count of next character.

getNextback to summary
pack-priv int getNext()

Returns next character

getNextIndexback to summary
private int getNextIndex()

Returns the position of next character.

getPreviousback to summary
private int getPrevious()

Returns previous character

getTextback to summary
public CharacterIterator getText()

Implements abstract java.text.BreakIterator.getText.

Return a CharacterIterator over the text being analyzed. This version of this method returns the actual CharacterIterator we're using internally. Changing the state of this iterator can have undefined consequences. If you need to change it, clone it first.

Returns:CharacterIterator

An iterator over the text being analyzed.

Annotations
@Override
handleNextback to summary
protected int handleNext()

This method is the actual implementation of the next() method. All iteration vectors through here. This method initializes the state machine to state 1 and advances through the text character by character until we reach the end of the text or the state machine transitions to state 0. We update our return value every time the state machine passes through a possible end state.

handlePreviousback to summary
protected int handlePrevious()

This method backs the iterator back up to a "safe position" in the text. This is a position that we know, without any context, must be a break position. The various calling methods then iterate forward from this safe position to the appropriate position to return. (For more information, see the description of buildBackwardsStateTable() in RuleBasedBreakIterator.Builder.)

hashCodeback to summary
public int hashCode()

Overrides java.lang.Object.hashCode.

Returns hashcode for this BreakIterator.

Returns:int

hashcode for this BreakIterator

Annotations
@Override
isBoundaryback to summary
public boolean isBoundary(int offset)

Overrides java.text.BreakIterator.isBoundary.

Returns true if the specified position is a boundary position. As a side effect, leaves the iterator pointing to the first boundary position at or after "offset".

Parameters
offset:int

the offset to check.

Returns:boolean

True if "offset" is a boundary position.

Annotations
@Override
lastback to summary
public int last()

Implements abstract java.text.BreakIterator.last.

Sets the current iteration position to the end of the text. (i.e., the CharacterIterator's ending offset).

Returns:int

The text's past-the-end offset.

Annotations
@Override
lookupBackwardStateback to summary
protected int lookupBackwardState(int state, int category)

Given a current state and a character category, looks up the next state to transition to in the backwards state table.

lookupCategoryback to summary
protected int lookupCategory(int c)

Looks up a character's category (i.e., its category for breaking purposes, not its Unicode category)

lookupStateback to summary
protected int lookupState(int state, int category)

Given a current state and a character category, looks up the next state to transition to in the state table.

nextback to summary
public int next(int n)

Implements abstract java.text.BreakIterator.next.

Advances the iterator either forward or backward the specified number of steps. Negative values move backward, and positive values move forward. This is equivalent to repeatedly calling next() or previous().

Parameters
n:int

The number of steps to move. The sign indicates the direction (negative is backwards, and positive is forwards).

Returns:int

The character offset of the boundary position n boundaries away from the current one.

Annotations
@Override
nextback to summary
public int next()

Implements abstract java.text.BreakIterator.next.

Advances the iterator to the next boundary position.

Returns:int

The position of the first boundary after this one.

Annotations
@Override
precedingback to summary
public int preceding(int offset)

Overrides java.text.BreakIterator.preceding.

Sets the iterator to refer to the last boundary position before the specified position.

Returns:int

The position of the last boundary before the starting position.

Annotations
@Override
previousback to summary
public int previous()

Implements abstract java.text.BreakIterator.previous.

Advances the iterator backwards, to the last boundary preceding this one.

Returns:int

The position of the last boundary position preceding this one.

Annotations
@Override
setAdditionalDataback to summary
pack-priv void setAdditionalData(byte[] b)
setTextback to summary
public void setText(CharacterIterator newText)

Implements abstract java.text.BreakIterator.setText.

Set the iterator to analyze a new piece of text. This function resets the current iteration position to the beginning of the text.

Parameters
newText:CharacterIterator

An iterator over the text to analyze.

Annotations
@Override
setupTablesback to summary
private void setupTables(String ruleFile, ByteBuffer bb)

Initializes the fields with the given rule data. The data format is as follows:

  BreakIteratorData {
      u1           magic[7];
      u1           version;
      u4           totalDataSize;
      header_info  header;
      body         value;
  }
totalDataSize is the summation of the size of header_info and body in byte count.

In header, each field except for checksum implies the length of each field. Since BMPdataLength is a fixed-length data(512 entries), its length isn't included in header. checksum is a CRC32 value of all in body.

  header_info {
      u4           stateTableLength;
      u4           backwardsStateTableLength;
      u4           endStatesLength;
      u4           lookaheadStatesLength;
      u4           BMPdataLength;
      u4           nonBMPdataLength;
      u4           additionalDataLength;
      u8           checksum;
  }

Finally, BMPindices and BMPdata are set to charCategoryTable. nonBMPdata is set to supplementaryCharCategoryTable.

  body {
      u2           stateTable[stateTableLength];
      u2           backwardsStateTable[backwardsStateTableLength];
      u1           endStates[endStatesLength];
      u1           lookaheadStates[lookaheadStatesLength];
      u2           BMPindices[512];
      u1           BMPdata[BMPdataLength];
      u4           nonBMPdata[numNonBMPdataLength];
      u1           additionalData[additionalDataLength];
  }
Exceptions
BufferUnderflowException:
if the end-of-data is reached before setting up all the tables
toStringback to summary
public String toString()

Overrides java.lang.Object.toString.

Returns text

Annotations
@Override
validateRuleDataback to summary
pack-priv void validateRuleData(String ruleFile, ByteBuffer bb)

Validates the magic number, version, and the length of the given data.

Exceptions
BufferUnderflowException:
if the end-of-data is reached while validating data
MissingResourceException:
if valification failed
sun.text back to summary

private final Class RuleBasedBreakIterator.SafeCharIterator

extends Object
implements CharacterIterator, Cloneable
Class Inheritance
All Implemented Interfaces
java.lang.Cloneable, java.text.CharacterIterator

Field Summary

Modifier and TypeField and Description
private CharacterIterator
private int
private int
private int

Constructor Summary

AccessConstructor and Description
pack-priv

Method Summary

Modifier and TypeMethod and Description
public Object
clone()

Overrides java.lang.Object.clone.

Implements java.text.CharacterIterator.clone.

Create a copy of this iterator

public char
current()

Implements java.text.CharacterIterator.current.

Gets the character at the current position (as returned by getIndex()).

public char
first()

Implements java.text.CharacterIterator.first.

Sets the position to getBeginIndex() and returns the character at that position.

public int
getBeginIndex()

Implements java.text.CharacterIterator.getBeginIndex.

Returns the start index of the text.

public int
getEndIndex()

Implements java.text.CharacterIterator.getEndIndex.

Returns the end index of the text.

public int
getIndex()

Implements java.text.CharacterIterator.getIndex.

Returns the current index.

public char
last()

Implements java.text.CharacterIterator.last.

Sets the position to getEndIndex()-1 (getEndIndex() if the text is empty) and returns the character at that position.

public char
next()

Implements java.text.CharacterIterator.next.

Increments the iterator's index by one and returns the character at the new index.

public char
previous()

Implements java.text.CharacterIterator.previous.

Decrements the iterator's index by one and returns the character at the new index.

public char
setIndex(int i)

Implements java.text.CharacterIterator.setIndex.

Sets the position to the specified position in the text and returns that character.

Inherited from java.lang.Object:
equalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Field Detail

baseback to summary
private CharacterIterator base
currentIndexback to summary
private int currentIndex
rangeLimitback to summary
private int rangeLimit
rangeStartback to summary
private int rangeStart

Constructor Detail

SafeCharIteratorback to summary
pack-priv SafeCharIterator(CharacterIterator base)

Method Detail

cloneback to summary
public Object clone()

Overrides java.lang.Object.clone.

Implements java.text.CharacterIterator.clone.

Doc from java.text.CharacterIterator.clone.

Create a copy of this iterator

Annotations
@Override
currentback to summary
public char current()

Implements java.text.CharacterIterator.current.

Doc from java.text.CharacterIterator.current.

Gets the character at the current position (as returned by getIndex()).

Annotations
@Override
firstback to summary
public char first()

Implements java.text.CharacterIterator.first.

Doc from java.text.CharacterIterator.first.

Sets the position to getBeginIndex() and returns the character at that position.

Annotations
@Override
getBeginIndexback to summary
public int getBeginIndex()

Implements java.text.CharacterIterator.getBeginIndex.

Doc from java.text.CharacterIterator.getBeginIndex.

Returns the start index of the text.

Annotations
@Override
getEndIndexback to summary
public int getEndIndex()

Implements java.text.CharacterIterator.getEndIndex.

Doc from java.text.CharacterIterator.getEndIndex.

Returns the end index of the text. This index is the index of the first character following the end of the text.

Annotations
@Override
getIndexback to summary
public int getIndex()

Implements java.text.CharacterIterator.getIndex.

Doc from java.text.CharacterIterator.getIndex.

Returns the current index.

Annotations
@Override
lastback to summary
public char last()

Implements java.text.CharacterIterator.last.

Doc from java.text.CharacterIterator.last.

Sets the position to getEndIndex()-1 (getEndIndex() if the text is empty) and returns the character at that position.

Annotations
@Override
nextback to summary
public char next()

Implements java.text.CharacterIterator.next.

Doc from java.text.CharacterIterator.next.

Increments the iterator's index by one and returns the character at the new index. If the resulting index is greater or equal to getEndIndex(), the current index is reset to getEndIndex() and a value of DONE is returned.

Annotations
@Override
previousback to summary
public char previous()

Implements java.text.CharacterIterator.previous.

Doc from java.text.CharacterIterator.previous.

Decrements the iterator's index by one and returns the character at the new index. If the current index is getBeginIndex(), the index remains at getBeginIndex() and a value of DONE is returned.

Annotations
@Override
setIndexback to summary
public char setIndex(int i)

Implements java.text.CharacterIterator.setIndex.

Doc from java.text.CharacterIterator.setIndex.

Sets the position to the specified position in the text and returns that character.

Annotations
@Override