Apache Commons Text 1.15.0 · org.apache.commons.text.StringTokenizer

StringTokenizer properties and options
Property	Type	Default
delim	CharSetMatcher	{ \t\n\r\f}
quote	NoneMatcher	{}
ignore	NoneMatcher	{}
emptyTokenAsNull	boolean	false
ignoreEmptyTokens	boolean	true

Modifier and Type	Field and Description
private char[]	chars The text to work on.
private static final StringTokenizer	CSV_TOKENIZER_PROTOTYPE Comma separated values tokenizer internal variable.
private StringMatcher	delimMatcher The delimiter matcher.
private boolean	emptyAsNull Whether to return empty tokens as null.
private StringMatcher	ignoredMatcher The ignored matcher.
private boolean	ignoreEmptyTokens Whether to ignore empty tokens.
private StringMatcher	quoteMatcher The quote matcher.
private int	tokenPos The current iteration position.
private String[]	tokens The parsed tokens.
private StringMatcher	trimmerMatcher The trimmer matcher.
private static final StringTokenizer	TSV_TOKENIZER_PROTOTYPE Tab separated values tokenizer internal variable.

Access	Constructor and Description
public	StringTokenizer() Constructs a tokenizer splitting on space, tab, newline and form feed as per StringTokenizer, but with no text to tokenize.
public	StringTokenizer(final char[] the string which is to be parsed, not cloned. input) Constructs a tokenizer splitting on space, tab, newline and form feed as per StringTokenizer.
public	StringTokenizer(final char[] the string which is to be parsed, not cloned. input, final char the field delimiter character. delim) Constructs a tokenizer splitting on the specified character.
public	StringTokenizer(final char[] the string which is to be parsed, not cloned. input, final char the field delimiter character. delim, final char the field quoted string character. quote) Constructs a tokenizer splitting on the specified delimiter character and handling quotes using the specified quote character.
public	StringTokenizer(final char[] the string which is to be parsed, not cloned. input, final String the field delimiter string. delim) Constructs a tokenizer splitting on the specified string.
public	StringTokenizer(final char[] the string which is to be parsed, not cloned. input, final StringMatcher the field delimiter matcher. delim) Constructs a tokenizer splitting using the specified delimiter matcher.
public	StringTokenizer(final char[] the string which is to be parsed, not cloned. input, final StringMatcher the field delimiter character. delim, final StringMatcher the field quoted string character. quote) Constructs a tokenizer splitting using the specified delimiter matcher and handling quotes using the specified quote matcher.
public	StringTokenizer(final String the string which is to be parsed. input) Constructs a tokenizer splitting on space, tab, newline and form feed as per StringTokenizer.
public	StringTokenizer(final String the string which is to be parsed. input, final char the field delimiter character. delim) Constructs a tokenizer splitting on the specified delimiter character.
public	StringTokenizer(final String the string which is to be parsed. input, final char the field delimiter character. delim, final char the field quoted string character. quote) Constructs a tokenizer splitting on the specified delimiter character and handling quotes using the specified quote character.
public	StringTokenizer(final String the string which is to be parsed. input, final String the field delimiter string. delim) Constructs a tokenizer splitting on the specified delimiter string.
public	StringTokenizer(final String the string which is to be parsed. input, final StringMatcher the field delimiter matcher. delim) Constructs a tokenizer splitting using the specified delimiter matcher.
public	StringTokenizer(final String the string which is to be parsed. input, final StringMatcher the field delimiter matcher. delim, final StringMatcher the field quoted string matcher. quote) Constructs a tokenizer splitting using the specified delimiter matcher and handling quotes using the specified quote matcher.

Method Summary

Modifier and Type	Method and Description
public void	add(final String this parameter ignored. obj) Implements java.util.ListIterator.add. Unsupported ListIterator operation.
private void	addToken(final List<String> the list to add to. list, String the token to add. tok) Adds a token to a list, paying attention to the parameters we've set.
private void	checkTokenized() Checks if tokenization has been done, and if not then do it.
public Object	Returns: a new instance of this Tokenizer which has been reset. clone() Overrides java.lang.Object.clone. Creates a new instance of this Tokenizer.
pack-priv Object	Returns: a new instance of this Tokenizer which has been reset. cloneReset() Creates a new instance of this Tokenizer.
public String	Returns: The string content being parsed. getContent() Gets the String content that the tokenizer is parsing.
private static StringTokenizer	Returns: a clone of `CSV_TOKENIZER_PROTOTYPE`. getCSVClone() Returns a clone of `CSV_TOKENIZER_PROTOTYPE`.
public static StringTokenizer	Returns: a new tokenizer instance which parses Comma Separated Value strings. getCSVInstance() Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input.
public static StringTokenizer	Returns: a new tokenizer instance which parses Comma Separated Value strings. getCSVInstance(final char[] the text to parse. input) Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input.
public static StringTokenizer	Returns: a new tokenizer instance which parses Comma Separated Value strings. getCSVInstance(final String the text to parse. input) Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input.
public StringMatcher	Returns: The delimiter matcher in use. getDelimiterMatcher() Gets the field delimiter matcher.
public StringMatcher	Returns: The ignored matcher in use. getIgnoredMatcher() Gets the ignored character matcher.
public StringMatcher	Returns: The quote matcher in use. getQuoteMatcher() Gets the quote matcher currently in use.
public String[]	Returns: The tokens as a String array. getTokenArray() Gets a copy of the full token list as an independent modifiable array.
public List<String>	Returns: The tokens as a String list. getTokenList() Gets a copy of the full token list as an independent modifiable list.
public StringMatcher	Returns: The trimmer matcher in use. getTrimmerMatcher() Gets the trimmer character matcher.
private static StringTokenizer	Returns: a clone of `TSV_TOKENIZER_PROTOTYPE`. getTSVClone() Returns a clone of `TSV_TOKENIZER_PROTOTYPE`.
public static StringTokenizer	Returns: a new tokenizer instance which parses Tab Separated Value strings. getTSVInstance() Gets a new tokenizer instance which parses Tab Separated Value strings.
public static StringTokenizer	Returns: a new tokenizer instance which parses Tab Separated Value strings. getTSVInstance(final char[] the string to parse. input) Gets a new tokenizer instance which parses Tab Separated Value strings.
public static StringTokenizer	Returns: a new tokenizer instance which parses Tab Separated Value strings. getTSVInstance(final String the string to parse. input) Gets a new tokenizer instance which parses Tab Separated Value strings.
public boolean	Returns: true if there are more tokens. hasNext() Implements java.util.ListIterator.hasNext. Tests whether there are any more tokens.
public boolean	Returns: true if there are previous tokens. hasPrevious() Implements java.util.ListIterator.hasPrevious. Tests whether there are any previous tokens that can be iterated to.
public boolean	Returns: true if empty tokens are returned as null. isEmptyTokenAsNull() Tests whether the tokenizer currently returns empty tokens as null.
public boolean	Returns: true if empty tokens are not returned. isIgnoreEmptyTokens() Tests whether the tokenizer currently ignores empty tokens.
private boolean	Returns: true if a quote is matched. isQuote(final char[] the character array being tokenized. srcChars, final int the position to check for a quote. pos, final int the length of the character array being tokenized. len, final int the start position of the matched quote, 0 if no quoting. quoteStart, final int the length of the matched quote, 0 if no quoting. quoteLen) Tests if the characters at the index specified match the quote already matched in readNextToken().
public String	Returns: The next String token. next() Implements java.util.ListIterator.next. Gets the next token.
public int	Returns: The next token index. nextIndex() Implements java.util.ListIterator.nextIndex. Gets the index of the next token to return.
public String	Returns: The next sequential token, or null when no more tokens are found. nextToken() Gets the next token from the String.
public String	Returns: The previous token. previous() Implements java.util.ListIterator.previous. Gets the token previous to the last returned token.
public int	Returns: The previous token index. previousIndex() Implements java.util.ListIterator.previousIndex. Gets the index of the previous token.
public String	Returns: The previous sequential token, or null when no more tokens are found. previousToken() Gets the previous token from the String.
private int	Returns: The starting position of the next field (the character immediately after the delimiter), or -1 if end of string found. readNextToken(final char[] the character array being tokenized. srcChars, int the first character of field. start, final int the length of the character array being tokenized. len, final TextStringBuilder a temporary work area. workArea, final List<String> the list of parsed tokens. tokenList) References Deprecated `TextStringBuilder` is deprecated or references (maybe indirectly) at least one deprecated element. Reads character by character through the String to get the next token.
private int	Returns: The starting position of the next field (the character immediately after the delimiter, or if end of string found, then the length of string. readWithQuotes(final char[] the character array being tokenized. srcChars, final int the first character of field. start, final int the length of the character array being tokenized. len, final TextStringBuilder a temporary work area. workArea, final List<String> the list of parsed tokens. tokenList, final int the start position of the matched quote, 0 if no quoting. quoteStart, final int the length of the matched quote, 0 if no quoting. quoteLen) References Deprecated `TextStringBuilder` is deprecated or references (maybe indirectly) at least one deprecated element. Reads a possibly quoted string token.
public void	remove() Implements java.util.ListIterator.remove. Throws `UnsupportedOperationException` for this unsupported ListIterator operation.
public StringTokenizer	Returns: `this` instance. reset() Resets this tokenizer, forgetting all parsing and iteration already completed.
public StringTokenizer	Returns: `this` instance. reset(final char[] the new character array to tokenize, not cloned, null sets no text to parse. input) Resets this tokenizer, giving it a new input string to parse.
public StringTokenizer	Returns: `this` instance. reset(final String the new string to tokenize, null sets no text to parse. input) Resets this tokenizer, giving it a new input string to parse.
public void	set(final String this parameter ignored. obj) Implements java.util.ListIterator.set. Throws `UnsupportedOperationException` for this unsupported ListIterator operation.
public StringTokenizer	Returns: `this` instance. setDelimiterChar(final char the delimiter character to use. delim) Sets the field delimiter character.
public StringTokenizer	Returns: `this` instance. setDelimiterMatcher(final StringMatcher the delimiter matcher to use. delim) Sets the field delimiter matcher.
public StringTokenizer	Returns: `this` instance. setDelimiterString(final String the delimiter string to use. delim) Sets the field delimiter string.
public StringTokenizer	Returns: `this` instance. setEmptyTokenAsNull(final boolean whether empty tokens are returned as null. emptyAsNull) Sets whether the tokenizer should return empty tokens as null.
public StringTokenizer	Returns: `this` instance. setIgnoredChar(final char the ignored character to use. ignored) Sets the character to ignore.
public StringTokenizer	Returns: `this` instance. setIgnoredMatcher(final StringMatcher the ignored matcher to use, null ignored. ignored) Sets the matcher for characters to ignore.
public StringTokenizer	Returns: `this` instance. setIgnoreEmptyTokens(final boolean whether empty tokens are not returned. ignoreEmptyTokens) Sets whether the tokenizer should ignore and not return empty tokens.
public StringTokenizer	Returns: `this` instance. setQuoteChar(final char the quote character to use. quote) Sets the quote character to use.
public StringTokenizer	Returns: `this` instance. setQuoteMatcher(final StringMatcher the quote matcher to use, null ignored. quote) Sets the quote matcher to use.
public StringTokenizer	Returns: `this` instance. setTrimmerMatcher(final StringMatcher the trimmer matcher to use, null ignored. trimmer) Sets the matcher for characters to trim.
public int	Returns: The number of matched tokens. size() Gets the number of tokens found in the String.
protected List<String>	Returns: The modifiable list of String tokens, unmodifiable if null array or zero count. tokenize(final char[] the character array being tokenized, may be null. srcChars, final int the start position within the character array, must be valid. offset, final int the number of characters to tokenize, must be valid. count) Internal method to performs the tokenization.
public String	Returns: The string content being parsed. toString() Overrides java.lang.Object.toString. Gets the String content that the tokenizer is parsing.

Inherited from java.lang.Object:: equals finalize getClass hashCode notify notifyAll wait wait wait

Field Detail

chars	back to summary
private char[] chars The text to work on.

CSV_TOKENIZER_PROTOTYPE	back to summary
private static final StringTokenizer CSV_TOKENIZER_PROTOTYPE Comma separated values tokenizer internal variable.

delimMatcher	back to summary
private StringMatcher delimMatcher The delimiter matcher.

emptyAsNull	back to summary
private boolean emptyAsNull Whether to return empty tokens as null.

ignoredMatcher	back to summary
private StringMatcher ignoredMatcher The ignored matcher.

ignoreEmptyTokens	back to summary
private boolean ignoreEmptyTokens Whether to ignore empty tokens.

quoteMatcher	back to summary
private StringMatcher quoteMatcher The quote matcher.

tokenPos	back to summary
private int tokenPos The current iteration position.

tokens	back to summary
private String[] tokens The parsed tokens.

trimmerMatcher	back to summary
private StringMatcher trimmerMatcher The trimmer matcher.

TSV_TOKENIZER_PROTOTYPE	back to summary
private static final StringTokenizer TSV_TOKENIZER_PROTOTYPE Tab separated values tokenizer internal variable.

Constructor Detail

StringTokenizer back to summary

StringTokenizer	back to summary
public StringTokenizer() Constructs a tokenizer splitting on space, tab, newline and form feed as per StringTokenizer, but with no text to tokenize. This constructor is normally used with `reset(String)`.

public StringTokenizer()

Constructs a tokenizer splitting on space, tab, newline and form feed as per StringTokenizer, but with no text to tokenize.

This constructor is normally used with reset(String).

StringTokenizer	back to summary
public StringTokenizer(final char[] input) Constructs a tokenizer splitting on space, tab, newline and form feed as per StringTokenizer. Parameters input:char[] the string which is to be parsed, not cloned.

StringTokenizer

back to summary

public StringTokenizer(final char[] input)

Constructs a tokenizer splitting on space, tab, newline and form feed as per StringTokenizer.

Parameters

input:char[]: the string which is to be parsed, not cloned.

StringTokenizer	back to summary
public StringTokenizer(final char[] input, final char delim) Constructs a tokenizer splitting on the specified character. Parameters input:char[] the string which is to be parsed, not cloned. delim:char the field delimiter character.

StringTokenizer

back to summary

public StringTokenizer(final char[] input, final char delim)

Constructs a tokenizer splitting on the specified character.

Parameters

input:char[]: the string which is to be parsed, not cloned.
delim:char: the field delimiter character.

StringTokenizer	back to summary
public StringTokenizer(final char[] input, final char delim, final char quote) Constructs a tokenizer splitting on the specified delimiter character and handling quotes using the specified quote character. Parameters input:char[] the string which is to be parsed, not cloned. delim:char the field delimiter character. quote:char the field quoted string character.

StringTokenizer

back to summary

public StringTokenizer(final char[] input, final char delim, final char quote)

Constructs a tokenizer splitting on the specified delimiter character and handling quotes using the specified quote character.

Parameters

input:char[]: the string which is to be parsed, not cloned.
delim:char: the field delimiter character.
quote:char: the field quoted string character.

StringTokenizer	back to summary
public StringTokenizer(final char[] input, final String delim) Constructs a tokenizer splitting on the specified string. Parameters input:char[] the string which is to be parsed, not cloned. delim:String the field delimiter string.

StringTokenizer

back to summary

public StringTokenizer(final char[] input, final String delim)

Constructs a tokenizer splitting on the specified string.

Parameters

input:char[]: the string which is to be parsed, not cloned.
delim:String: the field delimiter string.

StringTokenizer	back to summary
public StringTokenizer(final char[] input, final StringMatcher delim) Constructs a tokenizer splitting using the specified delimiter matcher. Parameters input:char[] the string which is to be parsed, not cloned. delim:StringMatcher the field delimiter matcher.

StringTokenizer

back to summary

public StringTokenizer(final char[] input, final StringMatcher delim)

Constructs a tokenizer splitting using the specified delimiter matcher.

Parameters

input:char[]: the string which is to be parsed, not cloned.
delim:StringMatcher: the field delimiter matcher.

StringTokenizer	back to summary
public StringTokenizer(final char[] input, final StringMatcher delim, final StringMatcher quote) Constructs a tokenizer splitting using the specified delimiter matcher and handling quotes using the specified quote matcher. Parameters input:char[] the string which is to be parsed, not cloned. delim:StringMatcher the field delimiter character. quote:StringMatcher the field quoted string character.

StringTokenizer

back to summary

public StringTokenizer(final char[] input, final StringMatcher delim, final StringMatcher quote)

Constructs a tokenizer splitting using the specified delimiter matcher and handling quotes using the specified quote matcher.

Parameters

input:char[]: the string which is to be parsed, not cloned.
delim:StringMatcher: the field delimiter character.
quote:StringMatcher: the field quoted string character.

StringTokenizer	back to summary
public StringTokenizer(final String input) Constructs a tokenizer splitting on space, tab, newline and form feed as per StringTokenizer. Parameters input:String the string which is to be parsed.

StringTokenizer

back to summary

public StringTokenizer(final String input)

Constructs a tokenizer splitting on space, tab, newline and form feed as per StringTokenizer.

Parameters

input:String: the string which is to be parsed.

StringTokenizer	back to summary
public StringTokenizer(final String input, final char delim) Constructs a tokenizer splitting on the specified delimiter character. Parameters input:String the string which is to be parsed. delim:char the field delimiter character.

StringTokenizer

back to summary

public StringTokenizer(final String input, final char delim)

Constructs a tokenizer splitting on the specified delimiter character.

Parameters

input:String: the string which is to be parsed.
delim:char: the field delimiter character.

StringTokenizer	back to summary
public StringTokenizer(final String input, final char delim, final char quote) Constructs a tokenizer splitting on the specified delimiter character and handling quotes using the specified quote character. Parameters input:String the string which is to be parsed. delim:char the field delimiter character. quote:char the field quoted string character.

StringTokenizer

back to summary

public StringTokenizer(final String input, final char delim, final char quote)

Constructs a tokenizer splitting on the specified delimiter character and handling quotes using the specified quote character.

Parameters

input:String: the string which is to be parsed.
delim:char: the field delimiter character.
quote:char: the field quoted string character.

StringTokenizer	back to summary
public StringTokenizer(final String input, final String delim) Constructs a tokenizer splitting on the specified delimiter string. Parameters input:String the string which is to be parsed. delim:String the field delimiter string.

StringTokenizer

back to summary

public StringTokenizer(final String input, final String delim)

Constructs a tokenizer splitting on the specified delimiter string.

Parameters

input:String: the string which is to be parsed.
delim:String: the field delimiter string.

StringTokenizer	back to summary
public StringTokenizer(final String input, final StringMatcher delim) Constructs a tokenizer splitting using the specified delimiter matcher. Parameters input:String the string which is to be parsed. delim:StringMatcher the field delimiter matcher.

StringTokenizer

back to summary

public StringTokenizer(final String input, final StringMatcher delim)

Constructs a tokenizer splitting using the specified delimiter matcher.

Parameters

input:String: the string which is to be parsed.
delim:StringMatcher: the field delimiter matcher.

StringTokenizer	back to summary
public StringTokenizer(final String input, final StringMatcher delim, final StringMatcher quote) Constructs a tokenizer splitting using the specified delimiter matcher and handling quotes using the specified quote matcher. Parameters input:String the string which is to be parsed. delim:StringMatcher the field delimiter matcher. quote:StringMatcher the field quoted string matcher.

StringTokenizer

back to summary

public StringTokenizer(final String input, final StringMatcher delim, final StringMatcher quote)

Constructs a tokenizer splitting using the specified delimiter matcher and handling quotes using the specified quote matcher.

Parameters

input:String: the string which is to be parsed.
delim:StringMatcher: the field delimiter matcher.
quote:StringMatcher: the field quoted string matcher.

Method Detail

add	back to summary
public void add(final String obj) Implements java.util.ListIterator.add. Unsupported ListIterator operation. Parameters obj:String this parameter ignored. Annotations @Override Exceptions UnsupportedOperationException: always.

add

back to summary

public void add(final String obj)

Implements java.util.ListIterator.add.

Unsupported ListIterator operation.

Parameters

obj:String: this parameter ignored.

Annotations

@Override

Exceptions

UnsupportedOperationException:: always.

addToken	back to summary
private void addToken(final List<String> list, String tok) Adds a token to a list, paying attention to the parameters we've set. Parameters list:List<String> the list to add to. tok:String the token to add.

addToken

back to summary

private void addToken(final List<String> list, String tok)

Adds a token to a list, paying attention to the parameters we've set.

Parameters

list:List<String>: the list to add to.
tok:String: the token to add.

checkTokenized	back to summary
private void checkTokenized() Checks if tokenization has been done, and if not then do it.

clone back to summary

clone	back to summary
public Object clone() Overrides java.lang.Object.clone. Creates a new instance of this Tokenizer. The new instance is reset so that it will be at the start of the token list. If a `CloneNotSupportedException` is caught, return `null`. Returns:Object a new instance of this Tokenizer which has been reset. Annotations @Override

public Object clone()

Overrides java.lang.Object.clone.

Creates a new instance of this Tokenizer. The new instance is reset so that it will be at the start of the token list. If a CloneNotSupportedException is caught, return null.

Returns:Object

a new instance of this Tokenizer which has been reset.

Annotations

@Override

cloneReset	back to summary
pack-priv Object cloneReset() throws CloneNotSupportedException Creates a new instance of this Tokenizer. The new instance is reset so that it will be at the start of the token list. Returns:Object a new instance of this Tokenizer which has been reset. Exceptions CloneNotSupportedException: if there is a problem cloning.

cloneReset

back to summary

pack-priv Object cloneReset() throws CloneNotSupportedException

Creates a new instance of this Tokenizer. The new instance is reset so that it will be at the start of the token list.

Returns:Object

a new instance of this Tokenizer which has been reset.

Exceptions

CloneNotSupportedException:: if there is a problem cloning.

getContent	back to summary
public String getContent() Gets the String content that the tokenizer is parsing. Returns:String The string content being parsed.

getContent

back to summary

public String getContent()

Gets the String content that the tokenizer is parsing.

Returns:String: The string content being parsed.

getCSVClone back to summary

getCSVClone	back to summary
private static StringTokenizer getCSVClone() Returns a clone of `CSV_TOKENIZER_PROTOTYPE`. Returns:StringTokenizer a clone of `CSV_TOKENIZER_PROTOTYPE`.

private static StringTokenizer getCSVClone()

Returns a clone of CSV_TOKENIZER_PROTOTYPE.

Returns:StringTokenizer: a clone of CSV_TOKENIZER_PROTOTYPE.

getCSVInstance	back to summary
public static StringTokenizer getCSVInstance() Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method). You must call a "reset" method to set the string which you want to parse. Returns:StringTokenizer a new tokenizer instance which parses Comma Separated Value strings.

getCSVInstance

back to summary

public static StringTokenizer getCSVInstance()

Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).

You must call a "reset" method to set the string which you want to parse.

Returns:StringTokenizer: a new tokenizer instance which parses Comma Separated Value strings.

getCSVInstance	back to summary
public static StringTokenizer getCSVInstance(final char[] input) Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method). Parameters input:char[] the text to parse. Returns:StringTokenizer a new tokenizer instance which parses Comma Separated Value strings.

getCSVInstance

back to summary

public static StringTokenizer getCSVInstance(final char[] input)

Parameters

input:char[]: the text to parse.

Returns:StringTokenizer

a new tokenizer instance which parses Comma Separated Value strings.

getCSVInstance	back to summary
public static StringTokenizer getCSVInstance(final String input) Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method). Parameters input:String the text to parse. Returns:StringTokenizer a new tokenizer instance which parses Comma Separated Value strings.

getCSVInstance

back to summary

public static StringTokenizer getCSVInstance(final String input)

Parameters

input:String: the text to parse.

Returns:StringTokenizer

a new tokenizer instance which parses Comma Separated Value strings.

getDelimiterMatcher	back to summary
public StringMatcher getDelimiterMatcher() Gets the field delimiter matcher. Returns:StringMatcher The delimiter matcher in use.

getDelimiterMatcher

back to summary

public StringMatcher getDelimiterMatcher()

Gets the field delimiter matcher.

Returns:StringMatcher: The delimiter matcher in use.

getIgnoredMatcher	back to summary
public StringMatcher getIgnoredMatcher() Gets the ignored character matcher. These characters are ignored when parsing the String, unless they are within a quoted region. The default value is not to ignore anything. Returns:StringMatcher The ignored matcher in use.

getIgnoredMatcher

back to summary

public StringMatcher getIgnoredMatcher()

Gets the ignored character matcher.

These characters are ignored when parsing the String, unless they are within a quoted region. The default value is not to ignore anything.

Returns:StringMatcher: The ignored matcher in use.

getQuoteMatcher	back to summary
public StringMatcher getQuoteMatcher() Gets the quote matcher currently in use. The quote character is used to wrap data between the tokens. This enables delimiters to be entered as data. The default value is '"' (double quote). Returns:StringMatcher The quote matcher in use.

getQuoteMatcher

back to summary

public StringMatcher getQuoteMatcher()

Gets the quote matcher currently in use.

The quote character is used to wrap data between the tokens. This enables delimiters to be entered as data. The default value is '"' (double quote).

Returns:StringMatcher: The quote matcher in use.

getTokenArray	back to summary
public String[] getTokenArray() Gets a copy of the full token list as an independent modifiable array. Returns:String[] The tokens as a String array.

getTokenArray

back to summary

public String[] getTokenArray()

Gets a copy of the full token list as an independent modifiable array.

Returns:String[]: The tokens as a String array.

getTokenList	back to summary
public List<String> getTokenList() Gets a copy of the full token list as an independent modifiable list. Returns:List<String> The tokens as a String list.

getTokenList

back to summary

public List<String> getTokenList()

Gets a copy of the full token list as an independent modifiable list.

Returns:List<String>: The tokens as a String list.

getTrimmerMatcher	back to summary
public StringMatcher getTrimmerMatcher() Gets the trimmer character matcher. These characters are trimmed off on each side of the delimiter until the token or quote is found. The default value is not to trim anything. Returns:StringMatcher The trimmer matcher in use.

getTrimmerMatcher

back to summary

public StringMatcher getTrimmerMatcher()

Gets the trimmer character matcher.

These characters are trimmed off on each side of the delimiter until the token or quote is found. The default value is not to trim anything.

Returns:StringMatcher: The trimmer matcher in use.

getTSVClone back to summary

getTSVClone	back to summary
private static StringTokenizer getTSVClone() Returns a clone of `TSV_TOKENIZER_PROTOTYPE`. Returns:StringTokenizer a clone of `TSV_TOKENIZER_PROTOTYPE`.

private static StringTokenizer getTSVClone()

Returns a clone of TSV_TOKENIZER_PROTOTYPE.

Returns:StringTokenizer: a clone of TSV_TOKENIZER_PROTOTYPE.

getTSVInstance	back to summary
public static StringTokenizer getTSVInstance() Gets a new tokenizer instance which parses Tab Separated Value strings. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method). You must call a "reset" method to set the string which you want to parse. Returns:StringTokenizer a new tokenizer instance which parses Tab Separated Value strings.

getTSVInstance

back to summary

public static StringTokenizer getTSVInstance()

Gets a new tokenizer instance which parses Tab Separated Value strings. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).

You must call a "reset" method to set the string which you want to parse.

Returns:StringTokenizer: a new tokenizer instance which parses Tab Separated Value strings.

getTSVInstance	back to summary
public static StringTokenizer getTSVInstance(final char[] input) Gets a new tokenizer instance which parses Tab Separated Value strings. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method). Parameters input:char[] the string to parse. Returns:StringTokenizer a new tokenizer instance which parses Tab Separated Value strings.

getTSVInstance

back to summary

public static StringTokenizer getTSVInstance(final char[] input)

Gets a new tokenizer instance which parses Tab Separated Value strings. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).

Parameters

input:char[]: the string to parse.

Returns:StringTokenizer

a new tokenizer instance which parses Tab Separated Value strings.

getTSVInstance	back to summary
public static StringTokenizer getTSVInstance(final String input) Gets a new tokenizer instance which parses Tab Separated Value strings. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method). Parameters input:String the string to parse. Returns:StringTokenizer a new tokenizer instance which parses Tab Separated Value strings.

getTSVInstance

back to summary

public static StringTokenizer getTSVInstance(final String input)

Gets a new tokenizer instance which parses Tab Separated Value strings. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).

Parameters

input:String: the string to parse.

Returns:StringTokenizer

a new tokenizer instance which parses Tab Separated Value strings.

hasNext	back to summary
public boolean hasNext() Implements java.util.ListIterator.hasNext. Tests whether there are any more tokens. Returns:boolean true if there are more tokens. Annotations @Override

hasNext

back to summary

public boolean hasNext()

Implements java.util.ListIterator.hasNext.

Tests whether there are any more tokens.

Returns:boolean

true if there are more tokens.

Annotations

@Override

hasPrevious	back to summary
public boolean hasPrevious() Implements java.util.ListIterator.hasPrevious. Tests whether there are any previous tokens that can be iterated to. Returns:boolean true if there are previous tokens. Annotations @Override

hasPrevious

back to summary

public boolean hasPrevious()

Implements java.util.ListIterator.hasPrevious.

Tests whether there are any previous tokens that can be iterated to.

Returns:boolean

true if there are previous tokens.

Annotations

@Override

isEmptyTokenAsNull

back to summary

public boolean isEmptyTokenAsNull()

Tests whether the tokenizer currently returns empty tokens as null. The default for this property is false.

Returns:boolean: true if empty tokens are returned as null.

isIgnoreEmptyTokens

back to summary

public boolean isIgnoreEmptyTokens()

Tests whether the tokenizer currently ignores empty tokens. The default for this property is true.

Returns:boolean: true if empty tokens are not returned.

isQuote

back to summary

private boolean isQuote(final char[] srcChars, final int pos, final int len, final int quoteStart, final int quoteLen)

Tests if the characters at the index specified match the quote already matched in readNextToken().

Parameters

srcChars:char[]: the character array being tokenized.
pos:int: the position to check for a quote.
len:int: the length of the character array being tokenized.
quoteStart:int: the start position of the matched quote, 0 if no quoting.
quoteLen:int: the length of the matched quote, 0 if no quoting.

Returns:boolean

true if a quote is matched.

back to summary

public String next()

Implements java.util.ListIterator.next.

Gets the next token.

Returns:String

The next String token.

Annotations

@Override

Exceptions

NoSuchElementException:: if there are no more elements.

nextIndex

back to summary

public int nextIndex()

Implements java.util.ListIterator.nextIndex.

Gets the index of the next token to return.

Returns:int

The next token index.

Annotations

@Override

nextToken back to summary

public String nextToken()

Gets the next token from the String. Equivalent to next() except it returns null rather than throwing NoSuchElementException when no tokens remain.

Returns:String: The next sequential token, or null when no more tokens are found.

back to summary

public String previous()

Implements java.util.ListIterator.previous.

Gets the token previous to the last returned token.

Returns:String

The previous token.

Annotations

@Override

previousIndex

back to summary

public int previousIndex()

Implements java.util.ListIterator.previousIndex.

Gets the index of the previous token.

Returns:int

The previous token index.

Annotations

@Override

previousToken

back to summary

public String previousToken()

Gets the previous token from the String.

Returns:String: The previous sequential token, or null when no more tokens are found.

readNextToken back to summary

private int readNextToken(final char[] srcChars, int start, final int len, final TextStringBuilder workArea, final List<String> tokenList)

References Deprecated

TextStringBuilder is deprecated or references (maybe indirectly) at least one deprecated element.

See corresponding docs for further information.

Reads character by character through the String to get the next token.

Parameters

srcChars:char[]: the character array being tokenized.
start:int: the first character of field.
len:int: the length of the character array being tokenized.
workArea:TextStringBuilder: a temporary work area.
tokenList:List<String>: the list of parsed tokens.

Returns:int

The starting position of the next field (the character immediately after the delimiter), or -1 if end of string found.

readWithQuotes back to summary

private int readWithQuotes(final char[] srcChars, final int start, final int len, final TextStringBuilder workArea, final List<String> tokenList, final int quoteStart, final int quoteLen)

References Deprecated

TextStringBuilder is deprecated or references (maybe indirectly) at least one deprecated element.

See corresponding docs for further information.

Reads a possibly quoted string token.

Parameters

srcChars:char[]: the character array being tokenized.
start:int: the first character of field.
len:int: the length of the character array being tokenized.
workArea:TextStringBuilder: a temporary work area.
tokenList:List<String>: the list of parsed tokens.
quoteStart:int: the start position of the matched quote, 0 if no quoting.
quoteLen:int: the length of the matched quote, 0 if no quoting.

Returns:int

The starting position of the next field (the character immediately after the delimiter, or if end of string found, then the length of string.

remove back to summary

public void remove()

Implements java.util.ListIterator.remove.

Throws UnsupportedOperationException for this unsupported ListIterator operation.

Annotations

@Override

Exceptions

UnsupportedOperationException:: Always thrown.

reset back to summary

public StringTokenizer reset()

Resets this tokenizer, forgetting all parsing and iteration already completed.

This method allows the same tokenizer to be reused for the same String.

Returns:StringTokenizer: this instance.

reset back to summary

public StringTokenizer reset(final char[] input)

Resets this tokenizer, giving it a new input string to parse. In this manner you can re-use a tokenizer with the same settings on multiple input lines.

Parameters

input:char[]: the new character array to tokenize, not cloned, null sets no text to parse.

Returns:StringTokenizer

this instance.

reset back to summary

public StringTokenizer reset(final String input)

Resets this tokenizer, giving it a new input string to parse. In this manner you can re-use a tokenizer with the same settings on multiple input lines.

Parameters

input:String: the new string to tokenize, null sets no text to parse.

Returns:StringTokenizer

this instance.

set back to summary

public void set(final String obj)

Implements java.util.ListIterator.set.

Throws UnsupportedOperationException for this unsupported ListIterator operation.

Parameters

obj:String: this parameter ignored.

Annotations

@Override

Exceptions

UnsupportedOperationException:: always.

setDelimiterChar back to summary

public StringTokenizer setDelimiterChar(final char delim)

Sets the field delimiter character.

Parameters

delim:char: the delimiter character to use.

Returns:StringTokenizer

this instance.

setDelimiterMatcher back to summary

public StringTokenizer setDelimiterMatcher(final StringMatcher delim)

Sets the field delimiter matcher.

The delimiter is used to separate one token from another.

Parameters

delim:StringMatcher: the delimiter matcher to use.

Returns:StringTokenizer

this instance.

setDelimiterString back to summary

public StringTokenizer setDelimiterString(final String delim)

Sets the field delimiter string.

Parameters

delim:String: the delimiter string to use.

Returns:StringTokenizer

this instance.

setEmptyTokenAsNull back to summary

public StringTokenizer setEmptyTokenAsNull(final boolean emptyAsNull)

Sets whether the tokenizer should return empty tokens as null. The default for this property is false.

Parameters

emptyAsNull:boolean: whether empty tokens are returned as null.

Returns:StringTokenizer

this instance.

setIgnoredChar back to summary

public StringTokenizer setIgnoredChar(final char ignored)

Sets the character to ignore.

This character is ignored when parsing the String, unless it is within a quoted region.

Parameters

ignored:char: the ignored character to use.

Returns:StringTokenizer

this instance.

setIgnoredMatcher back to summary

public StringTokenizer setIgnoredMatcher(final StringMatcher ignored)

Sets the matcher for characters to ignore.

These characters are ignored when parsing the String, unless they are within a quoted region.

Parameters

ignored:StringMatcher: the ignored matcher to use, null ignored.

Returns:StringTokenizer

this instance.

setIgnoreEmptyTokens back to summary

public StringTokenizer setIgnoreEmptyTokens(final boolean ignoreEmptyTokens)

Sets whether the tokenizer should ignore and not return empty tokens. The default for this property is true.

Parameters

ignoreEmptyTokens:boolean: whether empty tokens are not returned.

Returns:StringTokenizer

this instance.

setQuoteChar back to summary

public StringTokenizer setQuoteChar(final char quote)

Sets the quote character to use.

The quote character is used to wrap data between the tokens. This enables delimiters to be entered as data.

Parameters

quote:char: the quote character to use.

Returns:StringTokenizer

this instance.

setQuoteMatcher back to summary

public StringTokenizer setQuoteMatcher(final StringMatcher quote)

Sets the quote matcher to use.

The quote character is used to wrap data between the tokens. This enables delimiters to be entered as data.

Parameters

quote:StringMatcher: the quote matcher to use, null ignored.

Returns:StringTokenizer

this instance.

setTrimmerMatcher back to summary

public StringTokenizer setTrimmerMatcher(final StringMatcher trimmer)

Sets the matcher for characters to trim.

These characters are trimmed off on each side of the delimiter until the token or quote is found.

Parameters

trimmer:StringMatcher: the trimmer matcher to use, null ignored.

Returns:StringTokenizer

this instance.

size

back to summary

public int size()

Gets the number of tokens found in the String.

Returns:int: The number of matched tokens.

tokenize back to summary

protected List<String> tokenize(final char[] srcChars, final int offset, final int count)

Internal method to performs the tokenization.

Most users of this class do not need to call this method. This method will be called automatically by other (public) methods when required.

This method exists to allow subclasses to add code before or after the tokenization. For example, a subclass could alter the character array, offset or count to be parsed, or call the tokenizer multiple times on multiple strings. It is also be possible to filter the results.

StrTokenizer will always pass a zero offset and a count equal to the length of the array to this method, however a subclass may pass other values, or even an entirely different array.

Parameters