Top Fields Constructors Methods
org.antlr.v4.runtime

public Class UnbufferedTokenStream<T extends Token>

extends Object
implements TokenStream
Class Inheritance
All Implemented Interfaces
org.antlr.v4.runtime.TokenStream, org.antlr.v4.runtime.IntStream
Imports
org.antlr.v4.runtime.misc.Interval, java.util.Arrays

Field Summary

Modifier and TypeField and Description
protected int
currentTokenIndex

Absolute token index.

protected Token
lastToken

This is the LT(-1) token for the current position.

protected Token
lastTokenBufferStart

When numMarkers > 0, this is the LT(-1) token for the first token in tokens.

protected int
n

The number of tokens currently in tokens.

protected int
numMarkers

Count up with mark() and down with release().

protected int
p

0..n-1 index into tokens of next token.

protected Token[]
tokens

A moving window buffer of the data being scanned.

protected TokenSource

Constructor Summary

AccessConstructor and Description
public
public
UnbufferedTokenStream(TokenSource tokenSource, int bufferSize)

Method Summary

Modifier and TypeMethod and Description
protected void
public void
consume()

Implements org.antlr.v4.runtime.IntStream.consume.

Consumes the current symbol in the stream.
protected int
fill(int n)

Add n elements to the buffer.

public Token
get(int i)

Implements org.antlr.v4.runtime.TokenStream.get.

Gets the Token at the specified index in the stream.
protected final int
public String
getSourceName()

Implements org.antlr.v4.runtime.IntStream.getSourceName.

Gets the name of the underlying symbol source.
public String
getText()

Implements org.antlr.v4.runtime.TokenStream.getText.

Return the text of all tokens in the stream.
public String
getText(RuleContext
The context providing the source interval of tokens to get text for.
ctx
)

Implements org.antlr.v4.runtime.TokenStream.getText.

Return the text of all tokens in the source interval of the specified context.
public String
getText(Token
The first token in the interval to get text for.
start
,
Token
The last token in the interval to get text for (inclusive).
stop
)

Implements org.antlr.v4.runtime.TokenStream.getText.

Return the text of all tokens in this stream between start and stop (inclusive).
public String
getText(Interval
The interval of tokens within this stream to get text for.
interval
)

Implements org.antlr.v4.runtime.TokenStream.getText.

Return the text of all tokens within the specified interval.
public TokenSource
getTokenSource()

Implements org.antlr.v4.runtime.TokenStream.getTokenSource.

Gets the underlying TokenSource which provides tokens for this stream.
public int
index()

Implements org.antlr.v4.runtime.IntStream.index.

Return the index into the stream of the input symbol referred to by LA(1).
public int
LA(int i)

Implements org.antlr.v4.runtime.IntStream.LA.

Gets the value of the symbol at offset i from the current position.
public Token
LT(int i)

Implements org.antlr.v4.runtime.TokenStream.LT.

Get the Token instance associated with the value returned by LA(k).
public int
mark()

Implements org.antlr.v4.runtime.IntStream.mark.

Return a marker that we can release later.
public void
release(int
A marker returned by a call to mark().
marker
)

Implements org.antlr.v4.runtime.IntStream.release.

This method releases a marked range created by a call to mark().
public void
seek(int
The absolute index to seek to.
index
)

Implements org.antlr.v4.runtime.IntStream.seek.

Set the input cursor to the position indicated by index.
public int
size()

Implements org.antlr.v4.runtime.IntStream.size.

Returns the total number of symbols in the stream, including a single EOF symbol.
protected void
sync(int want)

Make sure we have 'need' elements from current position p.

Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Field Detail

currentTokenIndexback to summary
protected int currentTokenIndex

Absolute token index. It's the index of the token about to be read via LT(1). Goes from 0 to the number of tokens in the entire stream, although the stream size is unknown before the end is reached.

This value is used to set the token indexes if the stream provides tokens that implement WritableToken.

lastTokenback to summary
protected Token lastToken

This is the LT(-1) token for the current position.

lastTokenBufferStartback to summary
protected Token lastTokenBufferStart

When numMarkers > 0, this is the LT(-1) token for the first token in tokens. Otherwise, this is null.

nback to summary
protected int n

The number of tokens currently in tokens.

This is not the buffer capacity, that's tokens.length.

numMarkersback to summary
protected int numMarkers

Count up with mark() and down with release(). When we release() the last mark, numMarkers reaches 0 and we reset the buffer. Copy tokens[p]..tokens[n-1] to tokens[0]..tokens[(n-1)-p].

pback to summary
protected int p

0..n-1 index into tokens of next token.

The LT(1) token is tokens[p]. If p == n, we are out of buffered tokens.

tokensback to summary
protected Token[] tokens

A moving window buffer of the data being scanned. While there's a marker, we keep adding to buffer. Otherwise, consume() resets so we start filling at index 0 again.

tokenSourceback to summary
protected TokenSource tokenSource

Constructor Detail

UnbufferedTokenStreamback to summary
public UnbufferedTokenStream(TokenSource tokenSource)
UnbufferedTokenStreamback to summary
public UnbufferedTokenStream(TokenSource tokenSource, int bufferSize)

Method Detail

addback to summary
protected void add(Token t)
consumeback to summary
public void consume()

Implements org.antlr.v4.runtime.IntStream.consume.

Doc from org.antlr.v4.runtime.IntStream.consume.

Consumes the current symbol in the stream. This method has the following effects:

  • Forward movement: The value of index() before calling this method is less than the value of index() after calling this method.
  • Ordered lookahead: The value of LA(1) before calling this method becomes the value of LA(-1) after calling this method.
Note that calling this method does not guarantee that index() is incremented by exactly 1, as that would preclude the ability to implement filtering streams (e.g. CommonTokenStream which distinguishes between "on-channel" and "off-channel" tokens).
Annotations
@Override
fillback to summary
protected int fill(int n)

Add n elements to the buffer. Returns the number of tokens actually added to the buffer. If the return value is less than n, then EOF was reached before n tokens could be added.

getback to summary
public Token get(int i)

Implements org.antlr.v4.runtime.TokenStream.get.

Doc from org.antlr.v4.runtime.TokenStream.get.

Gets the Token at the specified index in the stream. When the preconditions of this method are met, the return value is non-null.

The preconditions for this method are the same as the preconditions of IntStream#seek. If the behavior of seek(index) is unspecified for the current state and given index, then the behavior of this method is also unspecified.

The symbol referred to by index differs from seek() only in the case of filtering streams where index lies before the end of the stream. Unlike seek(), this method does not adjust index to point to a non-ignored symbol.

Annotations
@Override
getBufferStartIndexback to summary
protected final int getBufferStartIndex()
getSourceNameback to summary
public String getSourceName()

Implements org.antlr.v4.runtime.IntStream.getSourceName.

Doc from org.antlr.v4.runtime.IntStream.getSourceName.

Gets the name of the underlying symbol source. This method returns a non-null, non-empty string. If such a name is not known, this method returns UNKNOWN_SOURCE_NAME.

Annotations
@Override
getTextback to summary
public String getText()

Implements org.antlr.v4.runtime.TokenStream.getText.

Doc from org.antlr.v4.runtime.TokenStream.getText.

Return the text of all tokens in the stream. This method behaves like the following code, including potential exceptions from the calls to IntStream#size and getText(Interval), but may be optimized by the specific implementation.

TokenStream stream = ...;
String text = stream.getText(new Interval(0, stream.size()));
Returns:String

The text of all tokens in the stream.

Annotations
@Override
getTextback to summary
public String getText(RuleContext ctx)

Implements org.antlr.v4.runtime.TokenStream.getText.

Doc from org.antlr.v4.runtime.TokenStream.getText.

Return the text of all tokens in the source interval of the specified context. This method behaves like the following code, including potential exceptions from the call to getText(Interval), but may be optimized by the specific implementation.

If ctx.getSourceInterval() does not return a valid interval of tokens provided by this stream, the behavior is unspecified.

TokenStream stream = ...;
String text = stream.getText(ctx.getSourceInterval());
Parameters
ctx:RuleContext

The context providing the source interval of tokens to get text for.

Returns:String

The text of all tokens within the source interval of ctx.

Annotations
@Override
getTextback to summary
public String getText(Token start, Token stop)

Implements org.antlr.v4.runtime.TokenStream.getText.

Doc from org.antlr.v4.runtime.TokenStream.getText.

Return the text of all tokens in this stream between start and stop (inclusive).

If the specified start or stop token was not provided by this stream, or if the stop occurred before the start token, the behavior is unspecified.

For streams which ensure that the Token#getTokenIndex method is accurate for all of its provided tokens, this method behaves like the following code. Other streams may implement this method in other ways provided the behavior is consistent with this at a high level.

TokenStream stream = ...;
String text = "";
for (int i = start.getTokenIndex(); i <= stop.getTokenIndex(); i++) {
  text += stream.get(i).getText();
}
Parameters
start:Token

The first token in the interval to get text for.

stop:Token

The last token in the interval to get text for (inclusive).

Returns:String

The text of all tokens lying between the specified start and stop tokens.

Annotations
@Override
getTextback to summary
public String getText(Interval interval)

Implements org.antlr.v4.runtime.TokenStream.getText.

Doc from org.antlr.v4.runtime.TokenStream.getText.

Return the text of all tokens within the specified interval. This method behaves like the following code (including potential exceptions for violating preconditions of get, but may be optimized by the specific implementation.

TokenStream stream = ...;
String text = "";
for (int i = interval.a; i <= interval.b; i++) {
  text += stream.get(i).getText();
}
Parameters
interval:Interval

The interval of tokens within this stream to get text for.

Returns:String

The text of all tokens within the specified interval in this stream.

Annotations
@Override
getTokenSourceback to summary
public TokenSource getTokenSource()

Implements org.antlr.v4.runtime.TokenStream.getTokenSource.

Doc from org.antlr.v4.runtime.TokenStream.getTokenSource.

Gets the underlying TokenSource which provides tokens for this stream.

Annotations
@Override
indexback to summary
public int index()

Implements org.antlr.v4.runtime.IntStream.index.

Doc from org.antlr.v4.runtime.IntStream.index.

Return the index into the stream of the input symbol referred to by LA(1).

The behavior of this method is unspecified if no call to an initializing method has occurred after this stream was constructed.

Annotations
@Override
LAback to summary
public int LA(int i)

Implements org.antlr.v4.runtime.IntStream.LA.

Doc from org.antlr.v4.runtime.IntStream.LA.

Gets the value of the symbol at offset i from the current position. When i==1, this method returns the value of the current symbol in the stream (which is the next symbol to be consumed). When i==-1, this method returns the value of the previously read symbol in the stream. It is not valid to call this method with i==0, but the specific behavior is unspecified because this method is frequently called from performance-critical code.

This method is guaranteed to succeed if any of the following are true:

  • i>0
  • i==-1 and index() returns a value greater than the value of index() after the stream was constructed and LA(1) was called in that order. Specifying the current index() relative to the index after the stream was created allows for filtering implementations that do not return every symbol from the underlying source. Specifying the call to LA(1) allows for lazily initialized streams.
  • LA(i) refers to a symbol consumed within a marked region that has not yet been released.

If i represents a position at or beyond the end of the stream, this method returns EOF.

The return value is unspecified if i<0 and fewer than -i calls to consume() have occurred from the beginning of the stream before calling this method.

Annotations
@Override
LTback to summary
public Token LT(int i)

Implements org.antlr.v4.runtime.TokenStream.LT.

Doc from org.antlr.v4.runtime.TokenStream.LT.

Get the Token instance associated with the value returned by LA(k). This method has the same pre- and post-conditions as IntStream#LA. In addition, when the preconditions of this method are met, the return value is non-null and the value of LT(k).getType()==LA(k).

Annotations
@Override
markback to summary
public int mark()

Implements org.antlr.v4.runtime.IntStream.mark.

Return a marker that we can release later.

The specific marker value used for this class allows for some level of protection against misuse where seek() is called on a mark or release() is called in the wrong order.

Returns:int

Doc from org.antlr.v4.runtime.IntStream.mark.

An opaque marker which should be passed to release() when the marked range is no longer required.

Annotations
@Override
releaseback to summary
public void release(int marker)

Implements org.antlr.v4.runtime.IntStream.release.

Doc from org.antlr.v4.runtime.IntStream.release.

This method releases a marked range created by a call to mark(). Calls to release() must appear in the reverse order of the corresponding calls to mark(). If a mark is released twice, or if marks are not released in reverse order of the corresponding calls to mark(), the behavior is unspecified.

For more information and an example, see mark.

Parameters
marker:int

A marker returned by a call to mark().

Annotations
@Override
seekback to summary
public void seek(int index)

Implements org.antlr.v4.runtime.IntStream.seek.

Doc from org.antlr.v4.runtime.IntStream.seek.

Set the input cursor to the position indicated by index. If the specified index lies past the end of the stream, the operation behaves as though index was the index of the EOF symbol. After this method returns without throwing an exception, then at least one of the following will be true.

  • index() will return the index of the first symbol appearing at or after the specified index. Specifically, implementations which filter their sources should automatically adjust index forward the minimum amount required for the operation to target a non-ignored symbol.
  • LA(1) returns EOF
This operation is guaranteed to not throw an exception if index lies within a marked region. For more information on marked regions, see mark. The behavior of this method is unspecified if no call to an initializing method has occurred after this stream was constructed.
Parameters
index:int

The absolute index to seek to.

Annotations
@Override
sizeback to summary
public int size()

Implements org.antlr.v4.runtime.IntStream.size.

Doc from org.antlr.v4.runtime.IntStream.size.

Returns the total number of symbols in the stream, including a single EOF symbol.

Annotations
@Override
syncback to summary
protected void sync(int want)

Make sure we have 'need' elements from current position p. Last valid p index is tokens.length-1. p+need-1 is the tokens index 'need' elements ahead. If we need 1 element, (p+1-1)==p must be less than tokens.length.