International Components for Unicode for Java (icu4j) 74.2 · com.ibm.icu.util.StringTokenizer

ICU Enhancement

This class is ICU's replacement for java.util.Calendar.

ICU

Methods, fields, and other functionality specific to ICU are labeled 'ICU'.

The string tokenizer class allows an application to break a string into tokens by performing code point comparison. The StringTokenizer methods do not distinguish among identifiers, numbers, and quoted strings, nor do they recognize and skip comments.

The set of delimiters (the codepoints that separate tokens) may be specified either at creation time or on a per-token basis.

An instance of StringTokenizer behaves in one of three ways, depending on whether it was created with the returnDelims and coalesceDelims flags having the value true or false:

If returnDelims is false, delimiter code points serve to separate tokens. A token is a maximal sequence of consecutive code points that are not delimiters.
If returnDelims is true, delimiter code points are themselves considered to be tokens. In this case, if coalesceDelims is true, such tokens will be the maximal sequence of consecutive code points that are delimiters. If coalesceDelims is false, a token will be received for each delimiter code point.

A token is thus either one delimiter code point, a maximal sequence of consecutive code points that are delimiters, or a maximal sequence of consecutive code points that are not delimiters.

A StringTokenizer object internally maintains a current position within the string to be tokenized. Some operations advance this current position past the code point processed.

A token is returned by taking a substring of the string that was used to create the StringTokenizer object.

Example of the use of the default delimiter tokenizer.

StringTokenizer st = new StringTokenizer("this is a test");
while (st.hasMoreTokens()) {
    println(st.nextToken());
    }

prints the following output:

    this
    is
    a
    test

Example of the use of the tokenizer with user specified delimiter.

    StringTokenizer st = new StringTokenizer(
    "this is a test with supplementary characters \ud800\ud800\udc00\udc00",
        " \ud800\udc00");
    while (st.hasMoreTokens()) {
        println(st.nextToken());
    }

prints the following output:

    this
    is
    a
    test
    with
    supplementary
    characters
    \ud800
    \udc00

Field Summary

Modifier and Type	Field and Description
private static final UnicodeSet	DEFAULT_DELIMITERS_ Default set of delimiters \t\n\r\f
private boolean[]	delims
private static final UnicodeSet	EMPTY_DELIMITER_ A empty delimiter UnicodeSet, used when user specified null delimiters
private boolean	m_coalesceDelimiters_ Flag indicating whether to coalesce runs of delimiters into single tokens
private UnicodeSet	m_delimiters_ UnicodeSet containing delimiters
private int	m_length_ Length of m_source_
private int	m_nextOffset_ Current position in string to parse for tokens
private boolean	m_returnDelimiters_ Flag indicator if delimiters are to be treated as tokens too
private String	m_source_ String to parse for tokens
private int[]	m_tokenLimit_ Array of pre-calculated tokens limit indexes in source string.
private int	m_tokenOffset_ Current offset to the token array.
private int	m_tokenSize_ Size of the token array.
private int[]	m_tokenStart_ Array of pre-calculated tokens start indexes in source string terminated by -1.
private static final int	TOKEN_SIZE_ Array size increments

Constructor Summary

Access	Constructor and Description
public	StringTokenizer(String a string to be parsed. str, UnicodeSet the delimiters. delim, boolean flag indicating whether to return the delimiters as tokens. returndelims) ICU
public	StringTokenizer(String a string to be parsed. str, UnicodeSet the delimiters. delim, boolean flag indicating whether to return the delimiters as tokens. returndelims, boolean flag indicating whether to return a run of delimiters as a single token or as one token per delimiter. This only takes effect if returndelims is true. coalescedelims) Deprecated This API is ICU internal only. ICU
public	StringTokenizer(String a string to be parsed. str, UnicodeSet the delimiters. delim) ICU
public	StringTokenizer(String a string to be parsed. str, String the delimiters. delim, boolean flag indicating whether to return the delimiters as tokens. returndelims) Constructs a string tokenizer for the specified string.
public	StringTokenizer(String a string to be parsed. str, String the delimiters. delim, boolean flag indicating whether to return the delimiters as tokens. returndelims, boolean flag indicating whether to return a run of delimiters as a single token or as one token per delimiter. This only takes effect if returndelims is true. coalescedelims) Deprecated This API is ICU internal only. Constructs a string tokenizer for the specified string.
public	StringTokenizer(String a string to be parsed. str, String the delimiters. delim) Constructs a string tokenizer for the specified string.
public	StringTokenizer(String a string to be parsed str) Constructs a string tokenizer for the specified string.

Method Summary

Modifier and Type	Method and Description
pack-priv void	checkDelimiters()
public int	Returns: the number of tokens remaining in the string using the current delimiter set. countTokens() Calculates the number of times that this tokenizer's `nextToken` method can be called before it generates an exception.
private int	Returns: offset of the immediate next delimiter, otherwise (- source string length - 1) if there are no more delimiters after m_nextOffset getNextDelimiter(int to the source string offset) Gets the index of the next delimiter after offset
private int	Returns: offset of the immediate next non-delimiter, otherwise (- source string length - 1) if there are no more delimiters after m_nextOffset getNextNonDelimiter(int to the source string offset) Gets the index of the next non-delimiter after m_nextOffset_
public boolean	Returns: `true` if there are more tokens; `false` otherwise. hasMoreElements() Implements java.util.Enumeration.hasMoreElements. Returns the same value as the `hasMoreTokens` method.
public boolean	Returns: `true` if and only if there is at least one token in the string after the current position; `false` otherwise. hasMoreTokens() Tests if there are more tokens available from this tokenizer's string.
public Object	Returns: the next token in the string. nextElement() Implements java.util.Enumeration.nextElement. Returns the same value as the `nextToken` method, except that its declared return value is `Object` rather than `String`.
public String	Returns: the next token from this string tokenizer. nextToken() Returns the next token from this string tokenizer.
public String	Returns: the next token, after switching to the new delimiter set. nextToken(String the new delimiters. delim) Returns the next token in this string tokenizer's string.
public String	Returns: the next token, after switching to the new delimiter set. nextToken(UnicodeSet the new delimiters. delim) ICU

Inherited from java.lang.Object:: clone equals finalize getClass hashCode notify notifyAll toString wait wait wait

Field Detail

DEFAULT_DELIMITERS_	back to summary
private static final UnicodeSet DEFAULT_DELIMITERS_ Default set of delimiters \t\n\r\f

delims	back to summary
private boolean[] delims

EMPTY_DELIMITER_	back to summary
private static final UnicodeSet EMPTY_DELIMITER_ A empty delimiter UnicodeSet, used when user specified null delimiters

m_coalesceDelimiters_	back to summary
private boolean m_coalesceDelimiters_ Flag indicating whether to coalesce runs of delimiters into single tokens

m_delimiters_	back to summary
private UnicodeSet m_delimiters_ UnicodeSet containing delimiters

m_length_	back to summary
private int m_length_ Length of m_source_

m_nextOffset_	back to summary
private int m_nextOffset_ Current position in string to parse for tokens

m_returnDelimiters_	back to summary
private boolean m_returnDelimiters_ Flag indicator if delimiters are to be treated as tokens too

m_source_	back to summary
private String m_source_ String to parse for tokens

m_tokenLimit_	back to summary
private int[] m_tokenLimit_ Array of pre-calculated tokens limit indexes in source string. This is only set up during countTokens() and only stores the remaining tokens, not all tokens including parsed ones

m_tokenOffset_	back to summary
private int m_tokenOffset_ Current offset to the token array. If the array token is not set up yet, this value is a -1

m_tokenSize_	back to summary
private int m_tokenSize_ Size of the token array. If the array token is not set up yet, this value is a -1

m_tokenStart_	back to summary
private int[] m_tokenStart_ Array of pre-calculated tokens start indexes in source string terminated by -1. This is only set up during countTokens() and only stores the remaining tokens, not all tokens including parsed ones

TOKEN_SIZE_	back to summary
private static final int TOKEN_SIZE_ Array size increments

Constructor Detail

StringTokenizer	back to summary
public StringTokenizer(String str, UnicodeSet delim, boolean returndelims) ICU Constructs a string tokenizer for the specified string. All characters in the delim argument are the delimiters for separating tokens. If the returnDelims flag is false, the delimiter characters are skipped and only serve as separators between tokens. If the returnDelims flag is true, then the delimiter characters are also returned as tokens, one per delimiter. Parameters str:String a string to be parsed. delim:UnicodeSet the delimiters. returndelims:boolean flag indicating whether to return the delimiters as tokens. Exceptions NullPointerException: if str is null ICU Status Stable since ICU 2.4.

StringTokenizer

back to summary

public StringTokenizer(String str, UnicodeSet delim, boolean returndelims)

ICU

Constructs a string tokenizer for the specified string. All characters in the delim argument are the delimiters for separating tokens.

If the returnDelims flag is false, the delimiter characters are skipped and only serve as separators between tokens.

If the returnDelims flag is true, then the delimiter characters are also returned as tokens, one per delimiter.

Parameters

str:String: a string to be parsed.
delim:UnicodeSet: the delimiters.
returndelims:boolean: flag indicating whether to return the delimiters as tokens.

Exceptions

NullPointerException:: if str is null

ICU Status

Stable since ICU 2.4.

StringTokenizer	back to summary
public StringTokenizer(String str, UnicodeSet delim, boolean returndelims, boolean coalescedelims) Deprecated This API is ICU internal only. ICU Constructs a string tokenizer for the specified string. All characters in the delim argument are the delimiters for separating tokens. If the returnDelims flag is false, the delimiter characters are skipped and only serve as separators between tokens. If the returnDelims flag is true, then the delimiter characters are also returned as tokens. If coalescedelims is true, one token is returned for each run of delimiter characters, otherwise one token is returned per delimiter. Since surrogate pairs can be delimiters, the returned token might be two chars in length. Parameters str:String a string to be parsed. delim:UnicodeSet the delimiters. returndelims:boolean flag indicating whether to return the delimiters as tokens. coalescedelims:boolean flag indicating whether to return a run of delimiters as a single token or as one token per delimiter. This only takes effect if returndelims is true. Annotations @Deprecated Exceptions NullPointerException: if str is null ICU Status Internal ICU component.

StringTokenizer

back to summary

public StringTokenizer(String str, UnicodeSet delim, boolean returndelims, boolean coalescedelims)

Deprecated

This API is ICU internal only.

ICU

Constructs a string tokenizer for the specified string. All characters in the delim argument are the delimiters for separating tokens.

If the returnDelims flag is false, the delimiter characters are skipped and only serve as separators between tokens.

If the returnDelims flag is true, then the delimiter characters are also returned as tokens. If coalescedelims is true, one token is returned for each run of delimiter characters, otherwise one token is returned per delimiter. Since surrogate pairs can be delimiters, the returned token might be two chars in length.

Parameters

str:String: a string to be parsed.
delim:UnicodeSet: the delimiters.
returndelims:boolean: flag indicating whether to return the delimiters as tokens.
coalescedelims:boolean: flag indicating whether to return a run of delimiters as a single token or as one token per delimiter. This only takes effect if returndelims is true.

Annotations

@Deprecated

Exceptions

NullPointerException:: if str is null

ICU Status

Internal ICU component.

StringTokenizer	back to summary
public StringTokenizer(String str, UnicodeSet delim) ICU Constructs a string tokenizer for the specified string. The characters in the delim argument are the delimiters for separating tokens. Delimiter characters themselves will not be treated as tokens. Parameters str:String a string to be parsed. delim:UnicodeSet the delimiters. Exceptions NullPointerException: if str is null ICU Status Stable since ICU 2.4.

StringTokenizer

back to summary

public StringTokenizer(String str, UnicodeSet delim)

ICU

Constructs a string tokenizer for the specified string. The characters in the delim argument are the delimiters for separating tokens.

Delimiter characters themselves will not be treated as tokens.

Parameters

str:String: a string to be parsed.
delim:UnicodeSet: the delimiters.

Exceptions

NullPointerException:: if str is null

ICU Status

Stable since ICU 2.4.

StringTokenizer	back to summary
public StringTokenizer(String str, String delim, boolean returndelims) Constructs a string tokenizer for the specified string. All characters in the delim argument are the delimiters for separating tokens. If the returnDelims flag is false, the delimiter characters are skipped and only serve as separators between tokens. If the returnDelims flag is true, then the delimiter characters are also returned as tokens, one per delimiter. Parameters str:String a string to be parsed. delim:String the delimiters. returndelims:boolean flag indicating whether to return the delimiters as tokens. Exceptions NullPointerException: if str is null ICU Status Stable since ICU 2.4.

StringTokenizer

back to summary

public StringTokenizer(String str, String delim, boolean returndelims)

Constructs a string tokenizer for the specified string. All characters in the delim argument are the delimiters for separating tokens.

If the returnDelims flag is false, the delimiter characters are skipped and only serve as separators between tokens.

If the returnDelims flag is true, then the delimiter characters are also returned as tokens, one per delimiter.

Parameters

str:String: a string to be parsed.
delim:String: the delimiters.
returndelims:boolean: flag indicating whether to return the delimiters as tokens.

Exceptions

NullPointerException:: if str is null

ICU Status

Stable since ICU 2.4.

StringTokenizer	back to summary
public StringTokenizer(String str, String delim, boolean returndelims, boolean coalescedelims) Deprecated This API is ICU internal only. Constructs a string tokenizer for the specified string. All characters in the delim argument are the delimiters for separating tokens. If the returnDelims flag is false, the delimiter characters are skipped and only serve as separators between tokens. If the returnDelims flag is true, then the delimiter characters are also returned as tokens. If coalescedelims is true, one token is returned for each run of delimiter characters, otherwise one token is returned per delimiter. Since surrogate pairs can be delimiters, the returned token might be two chars in length. Parameters str:String a string to be parsed. delim:String the delimiters. returndelims:boolean flag indicating whether to return the delimiters as tokens. coalescedelims:boolean flag indicating whether to return a run of delimiters as a single token or as one token per delimiter. This only takes effect if returndelims is true. Annotations @Deprecated Exceptions NullPointerException: if str is null ICU Status Internal ICU component.

StringTokenizer

back to summary

public StringTokenizer(String str, String delim, boolean returndelims, boolean coalescedelims)

Deprecated

This API is ICU internal only.

Constructs a string tokenizer for the specified string. All characters in the delim argument are the delimiters for separating tokens.

If the returnDelims flag is false, the delimiter characters are skipped and only serve as separators between tokens.

Parameters

str:String: a string to be parsed.
delim:String: the delimiters.
returndelims:boolean: flag indicating whether to return the delimiters as tokens.
coalescedelims:boolean: flag indicating whether to return a run of delimiters as a single token or as one token per delimiter. This only takes effect if returndelims is true.

Annotations

@Deprecated

Exceptions

NullPointerException:: if str is null

ICU Status

Internal ICU component.

StringTokenizer	back to summary
public StringTokenizer(String str, String delim) Constructs a string tokenizer for the specified string. The characters in the delim argument are the delimiters for separating tokens. Delimiter characters themselves will not be treated as tokens. Parameters str:String a string to be parsed. delim:String the delimiters. Exceptions NullPointerException: if str is null ICU Status Stable since ICU 2.4.

StringTokenizer

back to summary

public StringTokenizer(String str, String delim)

Constructs a string tokenizer for the specified string. The characters in the delim argument are the delimiters for separating tokens.

Delimiter characters themselves will not be treated as tokens.

Parameters

str:String: a string to be parsed.
delim:String: the delimiters.

Exceptions

NullPointerException:: if str is null

ICU Status

Stable since ICU 2.4.

StringTokenizer	back to summary
public StringTokenizer(String str) Constructs a string tokenizer for the specified string. The tokenizer uses the default delimiter set, which is " \t\n\r\f": the space character, the tab character, the newline character, the carriage-return character, and the form-feed character. Delimiter characters themselves will not be treated as tokens. Parameters str:String a string to be parsed Exceptions NullPointerException: if str is null ICU Status Stable since ICU 2.4.

StringTokenizer

back to summary

public StringTokenizer(String str)

Constructs a string tokenizer for the specified string. The tokenizer uses the default delimiter set, which is " \t\n\r\f": the space character, the tab character, the newline character, the carriage-return character, and the form-feed character.

Delimiter characters themselves will not be treated as tokens.

Parameters

str:String: a string to be parsed

Exceptions

NullPointerException:: if str is null

ICU Status

Stable since ICU 2.4.

Method Detail

checkDelimiters	back to summary
pack-priv void checkDelimiters()

countTokens back to summary

countTokens	back to summary
public int countTokens() Calculates the number of times that this tokenizer's `nextToken` method can be called before it generates an exception. The current position is not advanced. Returns:int the number of tokens remaining in the string using the current delimiter set. ICU Status Stable since ICU 2.4. See Also `nextToken()`

public int countTokens()

Calculates the number of times that this tokenizer's nextToken method can be called before it generates an exception. The current position is not advanced.

Returns:int: the number of tokens remaining in the string using the current delimiter set.
ICU Status: Stable since ICU 2.4.
See Also: nextToken()

getNextDelimiter	back to summary
private int getNextDelimiter(int offset) Gets the index of the next delimiter after offset Parameters offset:int to the source string Returns:int offset of the immediate next delimiter, otherwise (- source string length - 1) if there are no more delimiters after m_nextOffset

getNextDelimiter

back to summary

private int getNextDelimiter(int offset)

Gets the index of the next delimiter after offset

Parameters

offset:int: to the source string

Returns:int

offset of the immediate next delimiter, otherwise (- source string length - 1) if there are no more delimiters after m_nextOffset

getNextNonDelimiter	back to summary
private int getNextNonDelimiter(int offset) Gets the index of the next non-delimiter after m_nextOffset_ Parameters offset:int to the source string Returns:int offset of the immediate next non-delimiter, otherwise (- source string length - 1) if there are no more delimiters after m_nextOffset

getNextNonDelimiter

back to summary

private int getNextNonDelimiter(int offset)

Gets the index of the next non-delimiter after m_nextOffset_

Parameters

offset:int: to the source string

Returns:int

offset of the immediate next non-delimiter, otherwise (- source string length - 1) if there are no more delimiters after m_nextOffset

hasMoreElements back to summary

hasMoreElements	back to summary
public boolean hasMoreElements() Implements java.util.Enumeration.hasMoreElements. Returns the same value as the `hasMoreTokens` method. It exists so that this class can implement the `Enumeration` interface. Returns:boolean `true` if there are more tokens; `false` otherwise. ICU Status Stable since ICU 2.4. See Also `hasMoreTokens()`

public boolean hasMoreElements()

Implements java.util.Enumeration.hasMoreElements.

Returns the same value as the hasMoreTokens method. It exists so that this class can implement the Enumeration interface.

Returns:boolean: true if there are more tokens; false otherwise.
ICU Status: Stable since ICU 2.4.
See Also: hasMoreTokens()

hasMoreTokens back to summary

hasMoreTokens	back to summary
public boolean hasMoreTokens() Tests if there are more tokens available from this tokenizer's string. If this method returns `true`, then a subsequent call to `nextToken` with no argument will successfully return a token. Returns:boolean `true` if and only if there is at least one token in the string after the current position; `false` otherwise. ICU Status Stable since ICU 2.4.

public boolean hasMoreTokens()

Tests if there are more tokens available from this tokenizer's string. If this method returns true, then a subsequent call to nextToken with no argument will successfully return a token.

Returns:boolean: true if and only if there is at least one token in the string after the current position; false otherwise.
ICU Status: Stable since ICU 2.4.

nextElement back to summary

nextElement	back to summary
public Object nextElement() Implements java.util.Enumeration.nextElement. Returns the same value as the `nextToken` method, except that its declared return value is `Object` rather than `String`. It exists so that this class can implement the `Enumeration` interface. Returns:Object the next token in the string. Exceptions NoSuchElementException: if there are no more tokens in this tokenizer's string. ICU Status Stable since ICU 2.4. See Also `nextToken()`

public Object nextElement()

Implements java.util.Enumeration.nextElement.

Returns the same value as the nextToken method, except that its declared return value is Object rather than String. It exists so that this class can implement the Enumeration interface.

Returns:Object

the next token in the string.

Exceptions

NoSuchElementException:: if there are no more tokens in this tokenizer's string.

ICU Status

Stable since ICU 2.4.

See Also

nextToken()

nextToken	back to summary
public String nextToken() Returns the next token from this string tokenizer. Returns:String the next token from this string tokenizer. Exceptions NoSuchElementException: if there are no more tokens in this tokenizer's string. ICU Status Stable since ICU 2.4.

nextToken

back to summary

public String nextToken()

Returns the next token from this string tokenizer.

Returns:String

the next token from this string tokenizer.

Exceptions

NoSuchElementException:: if there are no more tokens in this tokenizer's string.

ICU Status

Stable since ICU 2.4.

nextToken back to summary

nextToken	back to summary
public String nextToken(String delim) Returns the next token in this string tokenizer's string. First, the set of characters considered to be delimiters by this `StringTokenizer` object is changed to be the characters in the string `delim`. Then the next token in the string after the current position is returned. The current position is advanced beyond the recognized token. The new delimiter set remains the default after this call. Parameters delim:String the new delimiters. Returns:String the next token, after switching to the new delimiter set. Exceptions NoSuchElementException: if there are no more tokens in this tokenizer's string. ICU Status Stable since ICU 2.4.

public String nextToken(String delim)

Returns the next token in this string tokenizer's string. First, the set of characters considered to be delimiters by this StringTokenizer object is changed to be the characters in the string delim. Then the next token in the string after the current position is returned. The current position is advanced beyond the recognized token. The new delimiter set remains the default after this call.

Parameters

delim:String: the new delimiters.

Returns:String

the next token, after switching to the new delimiter set.

Exceptions

NoSuchElementException:: if there are no more tokens in this tokenizer's string.

ICU Status

Stable since ICU 2.4.

nextToken back to summary

nextToken	back to summary
public String nextToken(UnicodeSet delim) ICU Returns the next token in this string tokenizer's string. First, the set of characters considered to be delimiters by this `StringTokenizer` object is changed to be the characters in the string `delim`. Then the next token in the string after the current position is returned. The current position is advanced beyond the recognized token. The new delimiter set remains the default after this call. Parameters delim:UnicodeSet the new delimiters. Returns:String the next token, after switching to the new delimiter set. Exceptions NoSuchElementException: if there are no more tokens in this tokenizer's string. ICU Status Stable since ICU 2.4.

public String nextToken(UnicodeSet delim)

ICU

Parameters

delim:UnicodeSet: the new delimiters.

Returns:String

the next token, after switching to the new delimiter set.

Exceptions

NoSuchElementException:: if there are no more tokens in this tokenizer's string.

ICU Status

Stable since ICU 2.4.

public final Class StringTokenizer

Field Summary

Constructor Summary

Method Summary

Field Detail

Constructor Detail

Method Detail