Top Description Fields Constructors Methods
org.apache.commons.lang3.text

public Class StrTokenizer

extends Object
implements ListIterator<String>, Cloneable
Class Inheritance
All Implemented Interfaces
java.lang.Cloneable, java.util.ListIterator, java.util.Iterator
Known Direct Subclasses
org.apache.commons.lang3.text.StrBuilder.StrBuilderTokenizer
Annotations
@Deprecated
Imports
java.util.ArrayList, .Arrays, .Collections, .List, .ListIterator, .NoSuchElementException, .StringTokenizer, org.apache.commons.lang3.ArrayUtils, .StringUtils

Deprecated

As of 3.6, use Apache Commons Text StringTokenizer.

Tokenizes a string based on delimiters (separators) and supporting quoting and ignored character concepts.

This class can split a String into many smaller strings. It aims to do a similar job to StringTokenizer, however it offers much more control and flexibility including implementing the ListIterator interface. By default, it is set up like StringTokenizer.

The input String is split into a number of tokens. Each token is separated from the next String by a delimiter. One or more delimiter characters must be specified.

Each token may be surrounded by quotes. The quote matcher specifies the quote character(s). A quote may be escaped within a quoted section by duplicating itself.

Between each token and the delimiter are potentially characters that need trimming. The trimmer matcher specifies these characters. One usage might be to trim whitespace characters.

At any point outside the quotes there might potentially be invalid characters. The ignored matcher specifies these characters to be removed. One usage might be to remove new line characters.

Empty tokens may be removed or returned as null. 

"a,b,c"         - Three tokens "a","b","c"   (comma delimiter)
" a, b , c "    - Three tokens "a","b","c"   (default CSV processing trims whitespace)
"a, ", b ,", c" - Three tokens "a, " , " b ", ", c" (quoted text untouched)
StrTokenizer properties and options
PropertyTypeDefault
delimCharSetMatcher{ \t\n\r\f}
quoteNoneMatcher{}
ignoreNoneMatcher{}
emptyTokenAsNullbooleanfalse
ignoreEmptyTokensbooleantrue
Since
2.2

Field Summary

Modifier and TypeField and Description
private char[]
chars

The text to work on.

private static final StrTokenizer
CSV_TOKENIZER_PROTOTYPE

References Deprecated StrTokenizer is deprecated or references (maybe indirectly) at least one deprecated element.
private StrMatcher
delimMatcher

References Deprecated StrMatcher is deprecated or references (maybe indirectly) at least one deprecated element.
The delimiter matcher
private boolean
emptyAsNull

Whether to return empty tokens as null

private StrMatcher
ignoredMatcher

References Deprecated StrMatcher is deprecated or references (maybe indirectly) at least one deprecated element.
The ignored matcher
private boolean
ignoreEmptyTokens

Whether to ignore empty tokens

private StrMatcher
quoteMatcher

References Deprecated StrMatcher is deprecated or references (maybe indirectly) at least one deprecated element.
The quote matcher
private int
tokenPos

The current iteration position

private String[]
tokens

The parsed tokens

private StrMatcher
trimmerMatcher

References Deprecated StrMatcher is deprecated or references (maybe indirectly) at least one deprecated element.
The trimmer matcher
private static final StrTokenizer
TSV_TOKENIZER_PROTOTYPE

References Deprecated StrTokenizer is deprecated or references (maybe indirectly) at least one deprecated element.

Constructor Summary

AccessConstructor and Description
public
StrTokenizer()

Constructs a tokenizer splitting on space, tab, newline and formfeed as per StringTokenizer, but with no text to tokenize.

public
StrTokenizer(final char[]
the string which is to be parsed, not cloned.
input)

Constructs a tokenizer splitting on space, tab, newline and formfeed as per StringTokenizer.

public
StrTokenizer(final char[]
the string which is to be parsed, not cloned.
input, final char
the field delimiter character.
delim)

Constructs a tokenizer splitting on the specified character.

public
StrTokenizer(final char[]
the string which is to be parsed, not cloned.
input, final char
the field delimiter character.
delim, final char
the field quoted string character.
quote)

Constructs a tokenizer splitting on the specified delimiter character and handling quotes using the specified quote character.

public
StrTokenizer(final char[]
the string which is to be parsed, not cloned.
input, final String
the field delimiter string.
delim)

Constructs a tokenizer splitting on the specified string.

public
StrTokenizer(final char[]
the string which is to be parsed, not cloned.
input, final StrMatcher
the field delimiter matcher.
delim)
References Deprecated StrMatcher is deprecated or references (maybe indirectly) at least one deprecated element.

Constructs a tokenizer splitting using the specified delimiter matcher.

public
StrTokenizer(final char[]
the string which is to be parsed, not cloned.
input, final StrMatcher
the field delimiter character.
delim, final StrMatcher
the field quoted string character.
quote)
References Deprecated StrMatcher is deprecated or references (maybe indirectly) at least one deprecated element.

Constructs a tokenizer splitting using the specified delimiter matcher and handling quotes using the specified quote matcher.

public
StrTokenizer(final String
the string which is to be parsed.
input)

Constructs a tokenizer splitting on space, tab, newline and formfeed as per StringTokenizer.

public
StrTokenizer(final String
the string which is to be parsed.
input, final char
the field delimiter character.
delim)

Constructs a tokenizer splitting on the specified delimiter character.

public
StrTokenizer(final String
the string which is to be parsed.
input, final char
the field delimiter character.
delim, final char
the field quoted string character.
quote)

Constructs a tokenizer splitting on the specified delimiter character and handling quotes using the specified quote character.

public
StrTokenizer(final String
the string which is to be parsed.
input, final String
the field delimiter string.
delim)

Constructs a tokenizer splitting on the specified delimiter string.

public
StrTokenizer(final String
the string which is to be parsed.
input, final StrMatcher
the field delimiter matcher.
delim)
References Deprecated StrMatcher is deprecated or references (maybe indirectly) at least one deprecated element.

Constructs a tokenizer splitting using the specified delimiter matcher.

public
StrTokenizer(final String
the string which is to be parsed.
input, final StrMatcher
the field delimiter matcher.
delim, final StrMatcher
the field quoted string matcher.
quote)
References Deprecated StrMatcher is deprecated or references (maybe indirectly) at least one deprecated element.

Constructs a tokenizer splitting using the specified delimiter matcher and handling quotes using the specified quote matcher.

Method Summary

Modifier and TypeMethod and Description
public void
add(final String
this parameter ignored.
obj)

Implements java.util.ListIterator.add.

Unsupported ListIterator operation.

private void
addToken(final List<String>
the list to add to.
list, String
the token to add.
tok)

Adds a token to a list, paying attention to the parameters we've set.

private void
checkTokenized()

Checks if tokenization has been done, and if not then do it.

public Object

Returns:

a new instance of this Tokenizer which has been reset.
clone()

Overrides java.lang.Object.clone.

Creates a new instance of this Tokenizer.

pack-priv Object

Returns:

a new instance of this Tokenizer which has been reset.
cloneReset()

Creates a new instance of this Tokenizer.

public String

Returns:

the string content being parsed.
getContent()

Gets the String content that the tokenizer is parsing.

private static StrTokenizer

Returns:

a clone of CSV_TOKENIZER_PROTOTYPE.
getCSVClone()
References Deprecated StrTokenizer is deprecated or references (maybe indirectly) at least one deprecated element.

Gets a clone of CSV_TOKENIZER_PROTOTYPE.

public static StrTokenizer

Returns:

a new tokenizer instance which parses Comma Separated Value strings.
getCSVInstance()
References Deprecated StrTokenizer is deprecated or references (maybe indirectly) at least one deprecated element.

Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input.

public static StrTokenizer

Returns:

a new tokenizer instance which parses Comma Separated Value strings.
getCSVInstance(final char[]
the text to parse.
input)
References Deprecated StrTokenizer is deprecated or references (maybe indirectly) at least one deprecated element.

Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input.

public static StrTokenizer

Returns:

a new tokenizer instance which parses Comma Separated Value strings.
getCSVInstance(final String
the text to parse.
input)
References Deprecated StrTokenizer is deprecated or references (maybe indirectly) at least one deprecated element.

Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input.

public StrMatcher

Returns:

the delimiter matcher in use.
getDelimiterMatcher()
References Deprecated StrMatcher is deprecated or references (maybe indirectly) at least one deprecated element.

Gets the field delimiter matcher.

public StrMatcher

Returns:

the ignored matcher in use.
getIgnoredMatcher()
References Deprecated StrMatcher is deprecated or references (maybe indirectly) at least one deprecated element.

Gets the ignored character matcher.

public StrMatcher

Returns:

the quote matcher in use.
getQuoteMatcher()
References Deprecated StrMatcher is deprecated or references (maybe indirectly) at least one deprecated element.

Gets the quote matcher currently in use.

public String[]

Returns:

the tokens as a String array.
getTokenArray()

Gets a copy of the full token list as an independent modifiable array.

public List<String>

Returns:

the tokens as a String array.
getTokenList()

Gets a copy of the full token list as an independent modifiable list.

public StrMatcher

Returns:

the trimmer matcher in use.
getTrimmerMatcher()
References Deprecated StrMatcher is deprecated or references (maybe indirectly) at least one deprecated element.

Gets the trimmer character matcher.

private static StrTokenizer

Returns:

a clone of TSV_TOKENIZER_PROTOTYPE.
getTSVClone()
References Deprecated StrTokenizer is deprecated or references (maybe indirectly) at least one deprecated element.

Gets a clone of TSV_TOKENIZER_PROTOTYPE.

public static StrTokenizer

Returns:

a new tokenizer instance which parses Tab Separated Value strings.
getTSVInstance()
References Deprecated StrTokenizer is deprecated or references (maybe indirectly) at least one deprecated element.

Gets a new tokenizer instance which parses Tab Separated Value strings.

public static StrTokenizer

Returns:

a new tokenizer instance which parses Tab Separated Value strings.
getTSVInstance(final char[]
the string to parse.
input)
References Deprecated StrTokenizer is deprecated or references (maybe indirectly) at least one deprecated element.

Gets a new tokenizer instance which parses Tab Separated Value strings.

public static StrTokenizer

Returns:

a new tokenizer instance which parses Tab Separated Value strings.
getTSVInstance(final String
the string to parse.
input)
References Deprecated StrTokenizer is deprecated or references (maybe indirectly) at least one deprecated element.

Gets a new tokenizer instance which parses Tab Separated Value strings.

public boolean

Returns:

true if there are more tokens.
hasNext()

Implements java.util.ListIterator.hasNext.

Checks whether there are any more tokens.

public boolean

Returns:

true if there are previous tokens.
hasPrevious()

Implements java.util.ListIterator.hasPrevious.

Checks whether there are any previous tokens that can be iterated to.

public boolean

Returns:

true if empty tokens are returned as null.
isEmptyTokenAsNull()

Gets whether the tokenizer currently returns empty tokens as null.

public boolean

Returns:

true if empty tokens are not returned.
isIgnoreEmptyTokens()

Gets whether the tokenizer currently ignores empty tokens.

private boolean

Returns:

true if a quote is matched.
isQuote(final char[]
the character array being tokenized.
srcChars, final int
the position to check for a quote.
pos, final int
the length of the character array being tokenized.
len, final int
the start position of the matched quote, 0 if no quoting.
quoteStart, final int
the length of the matched quote, 0 if no quoting.
quoteLen)

Checks if the characters at the index specified match the quote already matched in readNextToken().

public String

Returns:

the next String token.
next()

Implements java.util.ListIterator.next.

Gets the next token.

public int

Returns:

the next token index.
nextIndex()

Implements java.util.ListIterator.nextIndex.

Gets the index of the next token to return.

public String

Returns:

the next sequential token, or null when no more tokens are found.
nextToken()

Gets the next token from the String.

public String

Returns:

the previous token.
previous()

Implements java.util.ListIterator.previous.

Gets the token previous to the last returned token.

public int

Returns:

the previous token index.
previousIndex()

Implements java.util.ListIterator.previousIndex.

Gets the index of the previous token.

public String

Returns:

the previous sequential token, or null when no more tokens are found.
previousToken()

Gets the previous token from the String.

private int

Returns:

the starting position of the next field (the character immediately after the delimiter), or -1 if end of string found.
readNextToken(final char[]
the character array being tokenized.
srcChars, int
the first character of field.
start, final int
the length of the character array being tokenized.
len, final StrBuilder
a temporary work area.
workArea, final List<String>
the list of parsed tokens.
tokenList)
References Deprecated StrBuilder is deprecated or references (maybe indirectly) at least one deprecated element.

Reads character by character through the String to get the next token.

private int

Returns:

the starting position of the next field (the character immediately after the delimiter, or if end of string found, then the length of string.
readWithQuotes(final char[]
the character array being tokenized.
srcChars, final int
the first character of field.
start, final int
the length of the character array being tokenized.
len, final StrBuilder
a temporary work area.
workArea, final List<String>
the list of parsed tokens.
tokenList, final int
the start position of the matched quote, 0 if no quoting.
quoteStart, final int
the length of the matched quote, 0 if no quoting.
quoteLen)
References Deprecated StrBuilder is deprecated or references (maybe indirectly) at least one deprecated element.

Reads a possibly quoted string token.

public void
remove()

Implements java.util.ListIterator.remove.

Unsupported ListIterator operation.

public StrTokenizer

Returns:

this instance.
reset()
References Deprecated StrTokenizer is deprecated or references (maybe indirectly) at least one deprecated element.

Resets this tokenizer, forgetting all parsing and iteration already completed.

public StrTokenizer

Returns:

this instance.
reset(final char[]
the new character array to tokenize, not cloned, null sets no text to parse.
input)
References Deprecated StrTokenizer is deprecated or references (maybe indirectly) at least one deprecated element.

Reset this tokenizer, giving it a new input string to parse.

public StrTokenizer

Returns:

this instance.
reset(final String
the new string to tokenize, null sets no text to parse.
input)
References Deprecated StrTokenizer is deprecated or references (maybe indirectly) at least one deprecated element.

Reset this tokenizer, giving it a new input string to parse.

public void
set(final String
this parameter ignored.
obj)

Implements java.util.ListIterator.set.

Unsupported ListIterator operation.

public StrTokenizer

Returns:

this, to enable chaining.
setDelimiterChar(final char
the delimiter character to use.
delim)
References Deprecated StrTokenizer is deprecated or references (maybe indirectly) at least one deprecated element.

Sets the field delimiter character.

public StrTokenizer

Returns:

this, to enable chaining.
setDelimiterMatcher(final StrMatcher
the delimiter matcher to use.
delim)
References Deprecated StrMatcher and StrTokenizer are deprecated or reference (maybe indirectly) at least one deprecated element.

Sets the field delimiter matcher.

public StrTokenizer

Returns:

this, to enable chaining.
setDelimiterString(final String
the delimiter string to use.
delim)
References Deprecated StrTokenizer is deprecated or references (maybe indirectly) at least one deprecated element.

Sets the field delimiter string.

public StrTokenizer

Returns:

this, to enable chaining.
setEmptyTokenAsNull(final boolean
whether empty tokens are returned as null.
emptyAsNull)
References Deprecated StrTokenizer is deprecated or references (maybe indirectly) at least one deprecated element.

Sets whether the tokenizer should return empty tokens as null.

public StrTokenizer

Returns:

this, to enable chaining.
setIgnoredChar(final char
the ignored character to use.
ignored)
References Deprecated StrTokenizer is deprecated or references (maybe indirectly) at least one deprecated element.

Sets the character to ignore.

public StrTokenizer

Returns:

this instance.
setIgnoredMatcher(final StrMatcher
the ignored matcher to use, null ignored.
ignored)
References Deprecated StrMatcher and StrTokenizer are deprecated or reference (maybe indirectly) at least one deprecated element.

Sets the matcher for characters to ignore.

public StrTokenizer

Returns:

this instance.
setIgnoreEmptyTokens(final boolean
whether empty tokens are not returned.
ignoreEmptyTokens)
References Deprecated StrTokenizer is deprecated or references (maybe indirectly) at least one deprecated element.

Sets whether the tokenizer should ignore and not return empty tokens.

public StrTokenizer

Returns:

this instance.
setQuoteChar(final char
the quote character to use.
quote)
References Deprecated StrTokenizer is deprecated or references (maybe indirectly) at least one deprecated element.

Sets the quote character to use.

public StrTokenizer

Returns:

this instance.
setQuoteMatcher(final StrMatcher
the quote matcher to use, null ignored.
quote)
References Deprecated StrMatcher and StrTokenizer are deprecated or reference (maybe indirectly) at least one deprecated element.

Sets the quote matcher to use.

public StrTokenizer

Returns:

this instance.
setTrimmerMatcher(final StrMatcher
the trimmer matcher to use, null ignored.
trimmer)
References Deprecated StrMatcher and StrTokenizer are deprecated or reference (maybe indirectly) at least one deprecated element.

Sets the matcher for characters to trim.

public int

Returns:

the number of matched tokens.
size()

Gets the number of tokens found in the String.

protected List<String>

Returns:

the modifiable list of String tokens, unmodifiable if null array or zero count.
tokenize(final char[]
the character array being tokenized, may be null.
srcChars, final int
the start position within the character array, must be valid.
offset, final int
the number of characters to tokenize, must be valid.
count)

Internal method to performs the tokenization.

public String

Returns:

the string content being parsed.
toString()

Overrides java.lang.Object.toString.

Gets the String content that the tokenizer is parsing.

Inherited from java.lang.Object:
equalsfinalizegetClasshashCodenotifynotifyAllwaitwaitwait