This class represents the tokenizer for templates. It operates in two modes: inside and outside of expressions. It implements the TokenSource interface so it can be used with ANTLR parsers. Outside of expressions, we can return these token types: TEXT, INDENT, LDELIM (start of expression), RCURLY (end of subtemplate), and NEWLINE. Inside of an expression, this lexer returns all of the tokens needed by STParser. From the parser's point of view, it can treat a template as a simple stream of elements.

This class defines the token types and communicates these values to STParser.g via STLexer.tokens file (which must remain consistent).

Nested and Inner Type Summary

Modifier and Type	Class and Description
public static class	STLexer.STToken We build `STToken` tokens instead of relying on `CommonToken` so we can override `toString()`.

Field Summary

Modifier and Type	Field and Description
public static final int	AND
public static final int	AT
public static final int	BANG
pack-priv char	c current character
public static final int	COLON
public static final int	COMMA
public static final int	COMMENT
pack-priv char	delimiterStartChar The char which delimits the start of an expression.
pack-priv char	delimiterStopChar The char which delimits the end of an expression.
public static final int	DOT
public static final int	ELLIPSIS
public static final int	ELSE
public static final int	ELSEIF
public static final int	ENDIF
public static final char	EOF
public static final int	EOF_TYPE
public static final int	EQUALS
pack-priv ErrorManager	errMgr
public static final int	FALSE
public static final int	ID
public static final int	IF
public static final int	INDENT
pack-priv CharStream	input
public static final int	LBRACK
public static final int	LCURLY
public static final int	LDELIM
public static final int	LPAREN
public static final int	NEWLINE
public static final int	OR
public static final int	PIPE
public static final int	RBRACK
public static final int	RCURLY
public static final int	RDELIM
public static final int	REGION_END
public static final int	RPAREN
pack-priv boolean	scanningInsideExpr This keeps track of the current mode of the lexer.
public static final int	SEMI
public static final Token	SKIP
public static final int	SLASH
pack-priv int	startCharIndex When we started token, track initial coordinates so we can properly build token objects.
pack-priv int	startCharPositionInLine
pack-priv int	startLine
public static final int	STRING
public int	subtemplateDepth To be able to properly track the inside/outside mode, we need to track how deeply nested we are in some templates.
public static final int	SUPER
pack-priv Token	templateToken template embedded in a group file?
public static final int	TEXT
pack-priv List<Token>	tokens Our lexer routines might have to emit more than a single token.
public static final int	TRUE

Constructor Summary

Access	Constructor and Description
public	STLexer(CharStream input)
public	STLexer(ErrorManager errMgr, CharStream input, Token templateToken)
public	STLexer(ErrorManager errMgr, CharStream input, Token templateToken, char delimiterStartChar, char delimiterStopChar)

Method Summary

Modifier and Type	Method and Description
public Token	_nextToken()
pack-priv Token	COMMENT()
protected void	consume()
public void	emit(Token token)
pack-priv Token	ESCAPE()
public String	getSourceName() Implements org.antlr.runtime.TokenSource.getSourceName. Where are you getting tokens from?
protected Token	inside()
public static boolean	isIDLetter(char c)
public static boolean	isIDStartLetter(char c)
public static boolean	isUnicodeLetter(char c)
public static boolean	isWS(char c)
pack-priv void	LINEBREAK()
public void	match(char x) Consume if `x` is next character on the input stream.
pack-priv Token	mID() ID : ('a'..'z'\|'A'..'Z'\|'_'\|'/') ('a'..'z'\|'A'..'Z'\|'0'..'9'\|'_'\|'/')* ;
pack-priv Token	mSTRING() STRING : '"' ( '\\' '"' \| '\\' ~'"' \| ~('\\'\|'"') )* '"' ;
pack-priv Token	mTEXT()
public Token	newToken(int ttype)
public Token	newToken(int ttype, String text, int pos)
public Token	newToken(int ttype, String text)
public Token	newTokenFromPreviousChar(int ttype)
public Token	nextToken() Implements org.antlr.runtime.TokenSource.nextToken. Return a Token object from your input stream (usually a CharStream).
protected Token	outside()
public static String	str(int c)
pack-priv Token	subTemplate()
pack-priv Token	UNICODE()
pack-priv void	WS()

Inherited from java.lang.Object:: clone equals finalize getClass hashCode notify notifyAll toString wait wait wait

Field Detail

AND	back to summary
public static final int AND

AT	back to summary
public static final int AT

BANG	back to summary
public static final int BANG

c	back to summary
pack-priv char c current character

COLON	back to summary
public static final int COLON

COMMA	back to summary
public static final int COMMA

COMMENT	back to summary
public static final int COMMENT

delimiterStartChar	back to summary
pack-priv char delimiterStartChar The char which delimits the start of an expression.

delimiterStopChar	back to summary
pack-priv char delimiterStopChar The char which delimits the end of an expression.

DOT	back to summary
public static final int DOT

ELLIPSIS	back to summary
public static final int ELLIPSIS

ELSE	back to summary
public static final int ELSE

ELSEIF	back to summary
public static final int ELSEIF

ENDIF	back to summary
public static final int ENDIF

EOF	back to summary
public static final char EOF

EOF_TYPE	back to summary
public static final int EOF_TYPE

EQUALS	back to summary
public static final int EQUALS

errMgr	back to summary
pack-priv ErrorManager errMgr

FALSE	back to summary
public static final int FALSE

ID	back to summary
public static final int ID

IF	back to summary
public static final int IF

INDENT	back to summary
public static final int INDENT

input	back to summary
pack-priv CharStream input

LBRACK	back to summary
public static final int LBRACK

LCURLY	back to summary
public static final int LCURLY

LDELIM	back to summary
public static final int LDELIM

LPAREN	back to summary
public static final int LPAREN

NEWLINE	back to summary
public static final int NEWLINE

OR	back to summary
public static final int OR

PIPE	back to summary
public static final int PIPE

RBRACK	back to summary
public static final int RBRACK

RCURLY	back to summary
public static final int RCURLY

RDELIM	back to summary
public static final int RDELIM

REGION_END	back to summary
public static final int REGION_END

RPAREN	back to summary
public static final int RPAREN

scanningInsideExpr	back to summary
pack-priv boolean scanningInsideExpr This keeps track of the current mode of the lexer. Are we inside or outside an ST expression?

SEMI	back to summary
public static final int SEMI

SKIP	back to summary
public static final Token SKIP

SLASH	back to summary
public static final int SLASH

startCharIndex	back to summary
pack-priv int startCharIndex When we started token, track initial coordinates so we can properly build token objects.

startCharPositionInLine	back to summary
pack-priv int startCharPositionInLine

startLine	back to summary
pack-priv int startLine

STRING	back to summary
public static final int STRING

subtemplateDepth	back to summary
public int subtemplateDepth To be able to properly track the inside/outside mode, we need to track how deeply nested we are in some templates. Otherwise, we know whether a `'}'` and the outermost subtemplate to send this back to outside mode.

SUPER	back to summary
public static final int SUPER

templateToken	back to summary
pack-priv Token templateToken template embedded in a group file? this is the template

TEXT	back to summary
public static final int TEXT

tokens	back to summary
pack-priv List<Token> tokens Our lexer routines might have to emit more than a single token. We buffer everything through this list.

TRUE	back to summary
public static final int TRUE

Constructor Detail

STLexer	back to summary
public STLexer(CharStream input)

STLexer	back to summary
public STLexer(ErrorManager errMgr, CharStream input, Token templateToken)

STLexer	back to summary
public STLexer(ErrorManager errMgr, CharStream input, Token templateToken, char delimiterStartChar, char delimiterStopChar)

Method Detail

_nextToken	back to summary
public Token _nextToken()

COMMENT	back to summary
pack-priv Token COMMENT()

consume	back to summary
protected void consume()

emit	back to summary
public void emit(Token token)

ESCAPE	back to summary
pack-priv Token ESCAPE()

getSourceName

back to summary

public String getSourceName()

Implements org.antlr.runtime.TokenSource.getSourceName.

Doc from org.antlr.runtime.TokenSource.getSourceName.

Where are you getting tokens from? normally the implication will simply ask lexers input stream.

Annotations

@Override

inside	back to summary
protected Token inside()

isIDLetter	back to summary
public static boolean isIDLetter(char c)

isIDStartLetter	back to summary
public static boolean isIDStartLetter(char c)

isUnicodeLetter	back to summary
public static boolean isUnicodeLetter(char c)

isWS	back to summary
public static boolean isWS(char c)

LINEBREAK	back to summary
pack-priv void LINEBREAK()

match	back to summary
public void match(char x) Consume if `x` is next character on the input stream.

mID	back to summary
pack-priv Token mID() ID : ('a'..'z'\|'A'..'Z'\|'_'\|'/') ('a'..'z'\|'A'..'Z'\|'0'..'9'\|'_'\|'/')* ;

mSTRING	back to summary
pack-priv Token mSTRING() STRING : '"' ( '\\' '"' \| '\\' ~'"' \| ~('\\'\|'"') )* '"' ;

mTEXT	back to summary
pack-priv Token mTEXT()

newToken	back to summary
public Token newToken(int ttype)

newToken	back to summary
public Token newToken(int ttype, String text, int pos)

newToken	back to summary
public Token newToken(int ttype, String text)

newTokenFromPreviousChar	back to summary
public Token newTokenFromPreviousChar(int ttype)

nextToken

back to summary

public Token nextToken()

Implements org.antlr.runtime.TokenSource.nextToken.

Doc from org.antlr.runtime.TokenSource.nextToken.

Return a Token object from your input stream (usually a CharStream). Do not fail/return upon lexing error; keep chewing on the characters until you get a good one; errors are not passed through to the parser.

Annotations

@Override

outside	back to summary
protected Token outside()

str	back to summary
public static String str(int c)

subTemplate	back to summary
pack-priv Token subTemplate()

UNICODE	back to summary
pack-priv Token UNICODE()

WS	back to summary
pack-priv void WS()

org.stringtemplate.v4.compiler back to summary

public Class STLexer.STToken

Class Inheritance

extends CommonToken

java.lang.Object
org.antlr.runtime.CommonToken
org.stringtemplate.v4.compiler.STLexer.STToken

We build STToken tokens instead of relying on CommonToken so we can override toString(). It just converts token types to token names like 23 to "LDELIM".

Field Summary

Inherited from org.antlr.runtime.CommonToken:: channel charPositionInLine index input line start stop text type

Constructor Summary

Access	Constructor and Description
public	STToken(CharStream input, int type, int start, int stop)
public	STToken(int type, String text)

Method Summary

Modifier and Type	Method and Description
public String	toString() Overrides org.antlr.runtime.CommonToken.toString. Returns a string representation of the object.

Inherited from org.antlr.runtime.CommonToken:: getChannel getCharPositionInLine getInputStream getLine getStartIndex getStopIndex getText getTokenIndex getType setChannel setCharPositionInLine setInputStream setLine setStartIndex setStopIndex setText setTokenIndex setType

Constructor Detail

STToken	back to summary
public STToken(CharStream input, int type, int start, int stop)

STToken	back to summary
public STToken(int type, String text)

Method Detail

toString

back to summary

public String toString()

Overrides org.antlr.runtime.CommonToken.toString.

Doc from java.lang.Object.toString.

Returns a string representation of the object.

Returns:String

a string representation of the object

Annotations

@Override