public abstract Class Lexer

The channel number for the current token

Once we see EOF on char stream, next token will be EOF. If you have DONE : EOF ; then you see DONE EOF.

You can set the text for the current token to override what is in the input char buffer. Use setText() or can set this instance var.

The goal of all lexer rules/methods is to create a token object. This is an instance variable as multiple rules may collaborate to create a single token. nextToken will return this object after matching lexer rule(s). If you subclass to allow multiple token emissions, then set this to the last token to be matched or something nonnull so that the auto token emit mechanism will not emit another token.

What character index in the stream did the current token start at? Needed, for example, to get the text for current token. Set at the start of nextToken.

The character position of first character within the line

The line on which the first character of the token resides

The token type for the current token

By default does not support multiple emits per nextToken invocation for efficiency reasons. Subclass and override this method, nextToken, and getToken (to push tokens into a list and pull from that list rather than a single variable as this implementation does).

The standard method called to automatically emit a token at the outermost lexical rule. The token object should point into the char buffer start..stop. If there is a text override in 'text', use that to set the token's text. Override this method to emit custom Token objects or provide a new factory.

Return a list of all Token objects in input char stream. Forces load of all tokens. Does not include EOF token.

Annotations

@Nullable

What is the index of the current character of lookahead?

Implements org.antlr.v4.runtime.TokenSource.getCharPositionInLine.

Doc from org.antlr.v4.runtime.TokenSource.getCharPositionInLine.

Get the index into the current line for the current position in the input stream. The first character on a line has position 0.

Returns:int

The line number for the current position in the input stream, or -1 if the current token source does not track character positions.

Annotations

@Override

Implements abstract org.antlr.v4.runtime.Recognizer.getInputStream.

Implements org.antlr.v4.runtime.TokenSource.getInputStream.

Doc from org.antlr.v4.runtime.TokenSource.getInputStream.

Get the CharStream from which this token source is currently providing tokens.

Returns:CharStream

The CharStream associated with the current position in the input, or null if no input stream is available for the token source.

Annotations

@Override

Implements org.antlr.v4.runtime.TokenSource.getLine.

Doc from org.antlr.v4.runtime.TokenSource.getLine.

Get the line number for the current position in the input stream. The first line in the input is line 1.

Returns:int

The line number for the current position in the input stream, or 0 if the current token source does not track line numbers.

Annotations

@Override

Annotations

@Nullable

Implements org.antlr.v4.runtime.TokenSource.getSourceName.

Doc from org.antlr.v4.runtime.TokenSource.getSourceName.

Gets the name of the underlying input source. This method returns a non-null, non-empty string. If such a name is not known, this method returns IntStream#UNKNOWN_SOURCE_NAME.

Annotations

@Override

Return the text matched so far for the current token or any text override.

Override if emitting multiple tokens.

Implements org.antlr.v4.runtime.TokenSource.getTokenFactory.

Doc from org.antlr.v4.runtime.TokenSource.getTokenFactory.

Gets the TokenFactory this token source is currently using for creating Token objects from the input.

Returns:TokenFactory

The TokenFactory currently used by this token source.

Annotations

@Override

Implements abstract org.antlr.v4.runtime.Recognizer.getTokenNames.

Used to print out token names like ID during debugging and error reporting. The generated parsers implement a method that overrides this to point to their String[] tokenNames.

Annotations

@Override

@Deprecated

Implements org.antlr.v4.runtime.TokenSource.nextToken.

Return a token from this source; i.e., match a token on the char stream.

Annotations

@Override

Lexers can normally match any char in it's vocabulary after matching a token, so do the easy thing and just kill a character and hope it all works out. You can instead use the rule invocation stack to do sophisticated error recovery if you are in a fragment rule.

Set the char stream and reset the lexer

Set the complete text of this token; it wipes any previous changes to the text.

Implements org.antlr.v4.runtime.TokenSource.setTokenFactory.

Doc from org.antlr.v4.runtime.TokenSource.setTokenFactory.

Set the TokenFactory this token source should use for creating Token objects from the input.

Parameters

factory:TokenFactory

The TokenFactory to use for creating tokens.

Annotations

@Override

Instruct the lexer to skip creating a token for current lexer rule and look for another token. nextToken() knows to keep looking when a lexer rule finishes with token set to SKIP_TOKEN. Recall that if token==null at end of any token rule, it creates one for you and emits it.