Modifier and Type	Class and Description
protected static class	JavaTokenizer.BasicComment Scan a documentation comment; determine if a deprecated tag is present.

Modifier and Type	Field and Description
protected final boolean	enableLineDocComments Whether "///" comments are recognized as documentation comments.
protected int	errPos The position where a lexical error occurred;
protected final ScannerFactory	fac Origin scanner factory.
protected boolean	hasEscapeSequences true if contains escape sequences, set by nextToken().
protected boolean	isTextBlock true if is a text block, set by nextToken().
protected final Log	log Hides com.sun.tools.javac.parser.UnicodeReader.log. The log to be used for error reporting.
protected Name	name The token's name, set by nextToken().
private final Names	names The names factory.
private static final int	NOT_FOUND Sentinel for non-value.
private final Preview	preview The preview language setting.
protected int	radix The token's radix, set by nextToken().
protected final StringBuilder	sb Buffer for building literals, used by nextToken().
private static final boolean	scannerDebug If true then prints token information after each nextToken().
private final Source	source The source language setting.
protected Tokens.TokenKind	tk The token kind, set by nextToken().
private final Tokens	tokens The token factory.

Access	Constructor and Description
protected	JavaTokenizer(ScannerFactory the factory which created this Scanner. fac, CharBuffer the input character buffer. cb) Construct a Java token scanner from the input character buffer.
protected	JavaTokenizer(ScannerFactory factory which created this Scanner fac, char[] input character array array, int length of the meaningful content in the array length) Construct a Java token scanner from the input character array.

Method Summary

Modifier and Type	Method and Description
protected boolean	Returns: true if either ch1 or ch2 matches current character. acceptOneOfThenPut(char first character to match. ch1, char second character to match. ch2) If either ch1 or ch2 matches the current character then add current character to the literal buffer and then advance.
protected boolean	Returns: true if ch matches current character. acceptThenPut(char character to match. ch) If the specified character ch matches the current character then add current character to the literal buffer and then advance.
pack-priv List<Tokens.Comment>	Returns: new list with comment prepended to the existing list. appendComment(List<Tokens.Comment> existing list of comments. comments, Tokens.Comment comment to append. comment) Appends a comment to the list of comments preceding the current token.
private void	checkIdent() Determines if the sequence in the literal buffer is a token (keyword, operator.)
protected void	checkSourceLevel(int position in input buffer. pos, Source.Feature feature to verify. feature) Check the source level for a lexical feature.
public int	Returns: position in the input buffer of where the error occurred. errPos() Return the position where a lexical error occurred.
public void	errPos(int position in the input buffer of where the error occurred. pos) Set the position where a lexical error occurred.
public Position.LineMap	Returns: a LineMap getLineMap() Build a map for translating between line numbers and positions in the input.
private boolean	Returns: true if ch can be part of an operator. isSpecial(char character to check. ch) Return true if ch can be part of an operator.
protected void	lexError(int position in input buffer. pos, JCDiagnostic.Error error key to report. key) Report an error at the given position using the provided arguments.
protected Tokens.Comment	Returns: the constructed BasicComment. processComment(int position of the opening / in the input buffer. pos, int position + 1 of the closing / in the input buffer. endPos, Tokens.Comment.CommentStyle style of comment. style) Called when a complete comment has been scanned.
protected void	processLineTerminator(int position in input buffer of first character in sequence. pos, int position + 1 in input buffer of last character in sequence. endPos) Called when a line terminator has been processed.
protected void	processWhiteSpace(int position in input buffer of first whitespace character. pos, int position + 1 in input buffer of last whitespace character. endPos) Called when a complete whitespace run has been scanned.
protected void	put(char character to add. ch) Add a character to the literal buffer.
protected void	put() Add current character or codepoint to the literal buffer.
protected void	put(String string) Add a string to the literal buffer.
protected void	putCodePoint(int codepoint to add. codePoint) Add a codepoint to the literal buffer.
protected char	putThenNext() Add current character or codepoint to the literal buffer then return next character.
public Tokens.Token	readToken() Read token (main entrypoint.)
private void	scanDigits(int position of the first character in literal. pos, int radix of numeric literal. digitRadix) Scan sequence of digits.
private void	scanFraction(int position of the first character in literal. pos) Read fractional part of floating point number.
private void	scanFractionAndSuffix(int position of the first character in literal. pos) Read fractional part and 'd' or 'f' suffix of floating point number.
private void	scanHexExponentAndSuffix(int position of the first character in literal. pos) Read fractional part of hexadecimal floating point number.
private void	scanHexFractionAndSuffix(int position of the first character in literal. pos, boolean seendigit) Read fractional part and 'd' or 'f' suffix of hexadecimal floating point number.
private void	scanIdent() Read an identifier.
private void	scanLitChar(int position of the first character in literal. pos, boolean true if is a string literal isString) Processes the current character and places in the literal buffer.
private void	scanNumber(int position of the first character in literal. pos, int the radix of the number; one of 2, 8, 10, 16. radix) Read a number.
private void	scanOperator() Read longest possible sequence of special characters and convert to token.
private void	scanString(int position of the first character in literal. pos) Scan a string literal or text block.
private void	skipIllegalUnderscores() Skip over underscores and report as a error if found.
private void	skipLineTerminator() Skip and process a line terminator sequence.

Field Detail

enableLineDocComments	back to summary
protected final boolean enableLineDocComments Whether "///" comments are recognized as documentation comments.

errPos	back to summary
protected int errPos The position where a lexical error occurred;

fac	back to summary
protected final ScannerFactory fac Origin scanner factory.

hasEscapeSequences	back to summary
protected boolean hasEscapeSequences true if contains escape sequences, set by nextToken().

isTextBlock	back to summary
protected boolean isTextBlock true if is a text block, set by nextToken().

log	back to summary
protected final Log log Hides com.sun.tools.javac.parser.UnicodeReader.log. The log to be used for error reporting. Copied from scanner factory.

log

back to summary

protected final Log log

Hides com.sun.tools.javac.parser.UnicodeReader.log.

The log to be used for error reporting. Copied from scanner factory.

name	back to summary
protected Name name The token's name, set by nextToken().

names	back to summary
private final Names names The names factory. Copied from scanner factory.

NOT_FOUND	back to summary
private static final int NOT_FOUND Sentinel for non-value.

preview	back to summary
private final Preview preview The preview language setting. Copied from scanner factory.

radix	back to summary
protected int radix The token's radix, set by nextToken().

sb	back to summary
protected final StringBuilder sb Buffer for building literals, used by nextToken().

scannerDebug	back to summary
private static final boolean scannerDebug If true then prints token information after each nextToken().

source	back to summary
private final Source source The source language setting. Copied from scanner factory.

tk	back to summary
protected Tokens.TokenKind tk The token kind, set by nextToken().

tokens	back to summary
private final Tokens tokens The token factory. Copied from scanner factory.

Constructor Detail

JavaTokenizer	back to summary
protected JavaTokenizer(ScannerFactory fac, CharBuffer cb) Construct a Java token scanner from the input character buffer. Parameters fac:ScannerFactory the factory which created this Scanner. cb:CharBuffer the input character buffer.

JavaTokenizer

back to summary

protected JavaTokenizer(ScannerFactory fac, CharBuffer cb)

Construct a Java token scanner from the input character buffer.

Parameters

fac:ScannerFactory: the factory which created this Scanner.
cb:CharBuffer: the input character buffer.

JavaTokenizer	back to summary
protected JavaTokenizer(ScannerFactory fac, char[] array, int length) Construct a Java token scanner from the input character array. Parameters fac:ScannerFactory factory which created this Scanner array:char[] input character array length:int length of the meaningful content in the array

JavaTokenizer

back to summary

protected JavaTokenizer(ScannerFactory fac, char[] array, int length)

Construct a Java token scanner from the input character array.

Parameters

fac:ScannerFactory: factory which created this Scanner
array:char[]: input character array
length:int: length of the meaningful content in the array

Method Detail

acceptOneOfThenPut	back to summary
protected boolean acceptOneOfThenPut(char ch1, char ch2) If either ch1 or ch2 matches the current character then add current character to the literal buffer and then advance. Parameters ch1:char first character to match. ch2:char second character to match. Returns:boolean true if either ch1 or ch2 matches current character.

acceptOneOfThenPut

back to summary

protected boolean acceptOneOfThenPut(char ch1, char ch2)

If either ch1 or ch2 matches the current character then add current character to the literal buffer and then advance.

Parameters

ch1:char: first character to match.
ch2:char: second character to match.

Returns:boolean

true if either ch1 or ch2 matches current character.

acceptThenPut	back to summary
protected boolean acceptThenPut(char ch) If the specified character ch matches the current character then add current character to the literal buffer and then advance. Parameters ch:char character to match. Returns:boolean true if ch matches current character.

acceptThenPut

back to summary

protected boolean acceptThenPut(char ch)

If the specified character ch matches the current character then add current character to the literal buffer and then advance.

Parameters

ch:char: character to match.

Returns:boolean

true if ch matches current character.

appendComment	back to summary
pack-priv List<Tokens.Comment> appendComment(List<Tokens.Comment> comments, Tokens.Comment comment) Appends a comment to the list of comments preceding the current token. Parameters comments:List<Tokens.Comment> existing list of comments. comment:Tokens.Comment comment to append. Returns:List<Tokens.Comment> new list with comment prepended to the existing list.

appendComment

back to summary

pack-priv List<Tokens.Comment> appendComment(List<Tokens.Comment> comments, Tokens.Comment comment)

Appends a comment to the list of comments preceding the current token.

Parameters

comments:List<Tokens.Comment>: existing list of comments.
comment:Tokens.Comment: comment to append.

Returns:List<Tokens.Comment>

new list with comment prepended to the existing list.

checkIdent	back to summary
private void checkIdent() Determines if the sequence in the literal buffer is a token (keyword, operator.)

checkSourceLevel	back to summary
protected void checkSourceLevel(int pos, Source.Feature feature) Check the source level for a lexical feature. Parameters pos:int position in input buffer. feature:Source.Feature feature to verify.

checkSourceLevel

back to summary

protected void checkSourceLevel(int pos, Source.Feature feature)

Check the source level for a lexical feature.

Parameters

pos:int: position in input buffer.
feature:Source.Feature: feature to verify.

errPos	back to summary
public int errPos() Return the position where a lexical error occurred. Returns:int position in the input buffer of where the error occurred.

errPos

back to summary

public int errPos()

Return the position where a lexical error occurred.

Returns:int: position in the input buffer of where the error occurred.

errPos	back to summary
public void errPos(int pos) Set the position where a lexical error occurred. Parameters pos:int position in the input buffer of where the error occurred.

errPos

back to summary

public void errPos(int pos)

Set the position where a lexical error occurred.

Parameters

pos:int: position in the input buffer of where the error occurred.

getLineMap	back to summary
public Position.LineMap getLineMap() Build a map for translating between line numbers and positions in the input. Returns:Position.LineMap a LineMap

getLineMap

back to summary

public Position.LineMap getLineMap()

Build a map for translating between line numbers and positions in the input.

Returns:Position.LineMap: a LineMap

isSpecial	back to summary
private boolean isSpecial(char ch) Return true if ch can be part of an operator. Parameters ch:char character to check. Returns:boolean true if ch can be part of an operator.

isSpecial

back to summary

private boolean isSpecial(char ch)

Return true if ch can be part of an operator.

Parameters

ch:char: character to check.

Returns:boolean

true if ch can be part of an operator.

lexError	back to summary
protected void lexError(int pos, JCDiagnostic.Error key) Report an error at the given position using the provided arguments. Parameters pos:int position in input buffer. key:JCDiagnostic.Error error key to report.

lexError

back to summary

protected void lexError(int pos, JCDiagnostic.Error key)

Report an error at the given position using the provided arguments.

Parameters

pos:int: position in input buffer.
key:JCDiagnostic.Error: error key to report.

processComment	back to summary
protected Tokens.Comment processComment(int pos, int endPos, Tokens.Comment.CommentStyle style) Called when a complete comment has been scanned. pos and endPos will mark the comment boundary. Parameters pos:int position of the opening / in the input buffer. endPos:int position + 1 of the closing / in the input buffer. style:Tokens.Comment.CommentStyle style of comment. Returns:Tokens.Comment the constructed BasicComment.

processComment

back to summary

protected Tokens.Comment processComment(int pos, int endPos, Tokens.Comment.CommentStyle style)

Called when a complete comment has been scanned. pos and endPos will mark the comment boundary.

Parameters

pos:int: position of the opening / in the input buffer.
endPos:int: position + 1 of the closing / in the input buffer.
style:Tokens.Comment.CommentStyle: style of comment.

Returns:Tokens.Comment

the constructed BasicComment.

processLineTerminator	back to summary
protected void processLineTerminator(int pos, int endPos) Called when a line terminator has been processed. Parameters pos:int position in input buffer of first character in sequence. endPos:int position + 1 in input buffer of last character in sequence.

processLineTerminator

back to summary

protected void processLineTerminator(int pos, int endPos)

Called when a line terminator has been processed.

Parameters

pos:int: position in input buffer of first character in sequence.
endPos:int: position + 1 in input buffer of last character in sequence.

processWhiteSpace	back to summary
protected void processWhiteSpace(int pos, int endPos) Called when a complete whitespace run has been scanned. pos and endPos will mark the whitespace boundary. (Spec 3.6) Parameters pos:int position in input buffer of first whitespace character. endPos:int position + 1 in input buffer of last whitespace character.

processWhiteSpace

back to summary

protected void processWhiteSpace(int pos, int endPos)

Called when a complete whitespace run has been scanned. pos and endPos will mark the whitespace boundary. (Spec 3.6)

Parameters

pos:int: position in input buffer of first whitespace character.
endPos:int: position + 1 in input buffer of last whitespace character.

put	back to summary
protected void put(char ch) Add a character to the literal buffer. Parameters ch:char character to add.

put

back to summary

protected void put(char ch)

Add a character to the literal buffer.

Parameters

ch:char: character to add.

put	back to summary
protected void put() Add current character or codepoint to the literal buffer.

put	back to summary
protected void put(String string) Add a string to the literal buffer.

putCodePoint	back to summary
protected void putCodePoint(int codePoint) Add a codepoint to the literal buffer. Parameters codePoint:int codepoint to add.

putCodePoint

back to summary

protected void putCodePoint(int codePoint)

Add a codepoint to the literal buffer.

Parameters

codePoint:int: codepoint to add.

putThenNext	back to summary
protected char putThenNext() Add current character or codepoint to the literal buffer then return next character.

readToken	back to summary
public Tokens.Token readToken() Read token (main entrypoint.)

scanDigits	back to summary
private void scanDigits(int pos, int digitRadix) Scan sequence of digits. Parameters pos:int position of the first character in literal. digitRadix:int radix of numeric literal.

scanDigits

back to summary

private void scanDigits(int pos, int digitRadix)

Scan sequence of digits.

Parameters

pos:int: position of the first character in literal.
digitRadix:int: radix of numeric literal.

scanFraction	back to summary
private void scanFraction(int pos) Read fractional part of floating point number. Parameters pos:int position of the first character in literal.

scanFraction

back to summary

private void scanFraction(int pos)

Read fractional part of floating point number.

Parameters

pos:int: position of the first character in literal.

scanFractionAndSuffix	back to summary
private void scanFractionAndSuffix(int pos) Read fractional part and 'd' or 'f' suffix of floating point number. Parameters pos:int position of the first character in literal.

scanFractionAndSuffix

back to summary

private void scanFractionAndSuffix(int pos)

Read fractional part and 'd' or 'f' suffix of floating point number.

Parameters

pos:int: position of the first character in literal.

scanHexExponentAndSuffix	back to summary
private void scanHexExponentAndSuffix(int pos) Read fractional part of hexadecimal floating point number. Parameters pos:int position of the first character in literal.

scanHexExponentAndSuffix

back to summary

private void scanHexExponentAndSuffix(int pos)

Read fractional part of hexadecimal floating point number.

Parameters

pos:int: position of the first character in literal.

scanHexFractionAndSuffix	back to summary
private void scanHexFractionAndSuffix(int pos, boolean seendigit) Read fractional part and 'd' or 'f' suffix of hexadecimal floating point number. Parameters pos:int position of the first character in literal.

scanHexFractionAndSuffix

back to summary

private void scanHexFractionAndSuffix(int pos, boolean seendigit)

Read fractional part and 'd' or 'f' suffix of hexadecimal floating point number.

Parameters

pos:int: position of the first character in literal.

scanIdent	back to summary
private void scanIdent() Read an identifier. (Spec. 3.8)

scanLitChar	back to summary
private void scanLitChar(int pos, boolean isString) Processes the current character and places in the literal buffer. If the current character is a backslash then the next character is assumed to be a proper escape character. Actual conversion of escape sequences takes place during at the end of readToken. Parameters pos:int position of the first character in literal. isString:boolean true if is a string literal

scanLitChar

back to summary

private void scanLitChar(int pos, boolean isString)

Processes the current character and places in the literal buffer. If the current character is a backslash then the next character is assumed to be a proper escape character. Actual conversion of escape sequences takes place during at the end of readToken.

Parameters

pos:int: position of the first character in literal.
isString:boolean: true if is a string literal

scanNumber	back to summary
private void scanNumber(int pos, int radix) Read a number. (Spec. 3.10) Parameters pos:int position of the first character in literal. radix:int the radix of the number; one of 2, 8, 10, 16.

scanNumber

back to summary

private void scanNumber(int pos, int radix)

Read a number. (Spec. 3.10)

Parameters

pos:int: position of the first character in literal.
radix:int: the radix of the number; one of 2, 8, 10, 16.

scanOperator	back to summary
private void scanOperator() Read longest possible sequence of special characters and convert to token.

scanString

back to summary

private void scanString(int pos)

Scan a string literal or text block.

Parameters

pos:int: position of the first character in literal.

skipIllegalUnderscores	back to summary
private void skipIllegalUnderscores() Skip over underscores and report as a error if found.

skipLineTerminator	back to summary
private void skipLineTerminator() Skip and process a line terminator sequence.

com.sun.tools.javac.parser back to summary

protected Class JavaTokenizer.BasicComment

Class Inheritance

extends PositionTrackingReader
implements Comment

java.lang.Object
com.sun.tools.javac.parser.UnicodeReader
com.sun.tools.javac.parser.UnicodeReader.PositionTrackingReader
com.sun.tools.javac.parser.JavaTokenizer.BasicComment

All Implemented Interfaces

com.sun.tools.javac.parser.Tokens.Comment

Known Direct Subclasses

com.sun.tools.javac.parser.JavadocTokenizer.JavadocComment

Scan a documentation comment; determine if a deprecated tag is present. Called once the initial /, * have been skipped, positioned at the second * (which is treated as the beginning of the first line). Stops positioned at the closing '/'.

Field Summary

Modifier and Type	Field and Description
pack-priv final Tokens.Comment.CommentStyle	cs Style of comment
protected boolean	deprecatedFlag true if comment contains @deprecated at beginning of a line.
pack-priv JCDiagnostic.DiagnosticPosition	pos
protected boolean	scanned true if comment has been fully scanned.

Constructor Summary

Access	Constructor and Description
protected	BasicComment(Tokens.Comment.CommentStyle comment style cs, UnicodeReader existing reader reader, int start of meaningful content in buffer. pos, int end of meaningful content in buffer. endPos) Constructor.

Method Summary

Modifier and Type	Method and Description
pack-priv int	Returns: minimum indent to remove getJavadocLineCommentIndent() Determine how much indent to remove from a JavaDoc line comment.
public JCDiagnostic.DiagnosticPosition	Returns: diagnostic position getPos() Implements com.sun.tools.javac.parser.Tokens.Comment.getPos. Return the diagnostic position of this comment.
public int	Returns: buffer position in original buffer. getSourcePos(int buffer position in comment. pos) Implements com.sun.tools.javac.parser.Tokens.Comment.getSourcePos. Return buffer position in original buffer mapped from buffer position in comment.
public Tokens.Comment.CommentStyle	Returns: style of comment. getStyle() Implements com.sun.tools.javac.parser.Tokens.Comment.getStyle. Return style of comment.
public String	Returns: comment body text. getText() Implements com.sun.tools.javac.parser.Tokens.Comment.getText. Return comment body text minus comment adornments or null if not scanned.
public boolean	Returns: true if comment contains @deprecated. isDeprecated() Implements com.sun.tools.javac.parser.Tokens.Comment.isDeprecated. true if comment contains @deprecated at beginning of a line.
protected void	putLine(UnicodeReader line reader line) Put the line into the buffer.
protected void	scanDocComment() Scan document comment for content.
public Tokens.Comment	Returns: comment with removed whitespace or this comment stripIndent() Implements com.sun.tools.javac.parser.Tokens.Comment.stripIndent. Return a version of this comment with incidental whitespace removed, or this comment if the operation is not supported.
pack-priv UnicodeReader	Returns: new line reader if detected otherwise original line reader. trimEndOfComment(UnicodeReader line reader line) Remove closing star(s) slash from comment.
pack-priv UnicodeReader	Returns: modified line reader trimJavadocComment(UnicodeReader line reader line) Trim the first part of the JavaDoc block comment.
pack-priv UnicodeReader	Returns: modified line reader trimJavadocLineComment(UnicodeReader line reader line, int how much indentation to remove indent) Trim the first part of a JavaDoc line comment.