Top Inners Fields Constructors Methods
com.ibm.icu.text

pack-priv Class TransliteratorParser

extends Object
Class Inheritance
Imports
java.text.ParsePosition, java.util.ArrayList, .HashMap, .List, .Map, com.ibm.icu.impl.IllegalIcuArgumentException, .PatternProps, .Utility, com.ibm.icu.lang.UCharacter, com.ibm.icu.text.RuleBasedTransliterator.Data

Nested and Inner Type Summary

Modifier and TypeClass and Description
private class
TransliteratorParser.ParseData

This class implements the SymbolTable interface.

private static class
TransliteratorParser.RuleArray

RuleBody subclass for a String[] array.

private abstract static class
TransliteratorParser.RuleBody

A private abstract class representing the interface to rule source code that is broken up into lines.

private static class
TransliteratorParser.RuleHalf

A class representing one side of a rule.

Field Summary

Modifier and TypeField and Description
private static final char
private static final char
private static final char
private static final char
private static final char
public UnicodeSet
compoundFilter

PUBLIC data member containing the parsed compound filter, if any.

private static final char
private static final char
private RuleBasedTransliterator.Data
curData

The current data object for which we are parsing rules

private static final char
private static final char
public List<RuleBasedTransliterator.Data>
dataVector

PUBLIC data member.

private int
private static final char
private static final String
private int
dotStandIn

The stand-in character for the 'dot' set, represented by '.' in patterns.

private static final char
private static final char
private static final char
private static final char
private static final char
private static final String
private static final String
private static final int
public List<String>
idBlockVector

PUBLIC data member.

private static UnicodeSet
private static UnicodeSet
private static UnicodeSet
private static final char
private static final char
private static final String
private TransliteratorParser.ParseData
parseData

Temporary symbol table used during parsing.

private static final char
private static final char
private static final char
private static final char
private static final char
private List<StringMatcher>
segmentObjects

Vector of StringMatcher objects for segments.

private StringBuffer
segmentStandins

String of standins for segments.

private String
undefinedVariableName

When we encounter an undefined variable, we do not immediately signal an error, in case we are defining this variable, e.g., "$a = [a-z];".

private static final char
private char
variableLimit

The last available stand-in for variables.

private Map<String, char[]>
variableNames

Temporary table of variable names.

private char
variableNext

The next available stand-in for variables.

private List<Object>
variablesVector

Temporary vector of set variables.

private static final char

Constructor Summary

AccessConstructor and Description
public
TransliteratorParser()

Constructor.

Method Summary

Modifier and TypeMethod and Description
private void
appendVariableDef(String name, StringBuffer buf)

Append the value of the given variable name to the given StringBuffer.

private void
checkVariableRange(int ch, String rule, int start)

Assert that the given character is NOT within the variable range.

pack-priv char
generateStandInFor(Object obj)

Generate and return a stand-in for a new UnicodeMatcher or UnicodeReplacer.

pack-priv char
getDotStandIn()

Return the stand-in for the dot set.

public char
getSegmentStandin(int seg)

Return the standin for segment seg (1-based).

public void
parse(String rules, int dir)

Parse a set of rules.

private int

Returns:

the position index after the final ';' of the pragma, or -1 on failure.
parsePragma
(String rule, int
offset to the first non-whitespace character of the rule.
pos
,
int
pointer past the last character of the rule.
limit
)

Parse a pragma.

private int
parseRule(String rule, int pos, int limit)

MAIN PARSER.

pack-priv void
parseRules(TransliteratorParser.RuleBody ruleArray, int dir)

Parse an array of zero or more rules.

private final char
parseSet(String rule, ParsePosition pos)

Parse a UnicodeSet out, store it, and return the stand-in character used to represent it.

private void
pragmaMaximumBackup(int backup)

Set the maximum backup to 'backup', in response to a pragma statement.

private void
pragmaNormalizeRules(Normalizer.Mode mode)

Deprecated as a consequence of Normalizer.Mode being deprecated.
Begin normalizing all rules using the given mode, in response to a pragma statement.
pack-priv static boolean
resemblesPragma(String rule, int
offset to the first non-whitespace character of the rule.
pos
,
int
pointer past the last character of the rule.
limit
)

Return true if the given rule looks like a pragma.

pack-priv static final int
ruleEnd(String rule, int start, int limit)

public void
setSegmentObject(int seg, StringMatcher obj)

Set the object for segment seg (1-based).

private void
setVariableRange(int start, int end)

Set the variable range to [start, end] (inclusive).

pack-priv static final void
syntaxError(String
error description
msg
,
String
pattern string
rule
,
int
position of first character of current rule
start
)

Throw an exception indicating a syntax error.

Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Field Detail

ALT_FORWARD_RULE_OPback to summary
private static final char ALT_FORWARD_RULE_OP
ALT_FUNCTIONback to summary
private static final char ALT_FUNCTION
ALT_FWDREV_RULE_OPback to summary
private static final char ALT_FWDREV_RULE_OP
ALT_REVERSE_RULE_OPback to summary
private static final char ALT_REVERSE_RULE_OP
ANCHOR_STARTback to summary
private static final char ANCHOR_START
compoundFilterback to summary
public UnicodeSet compoundFilter

PUBLIC data member containing the parsed compound filter, if any.

CONTEXT_ANTEback to summary
private static final char CONTEXT_ANTE
CONTEXT_POSTback to summary
private static final char CONTEXT_POST
curDataback to summary
private RuleBasedTransliterator.Data curData

The current data object for which we are parsing rules

CURSOR_OFFSETback to summary
private static final char CURSOR_OFFSET
CURSOR_POSback to summary
private static final char CURSOR_POS
dataVectorback to summary
public List<RuleBasedTransliterator.Data> dataVector

PUBLIC data member. A Vector of RuleBasedTransliterator.Data objects, one for each discrete group of rules in the rule set

directionback to summary
private int direction
DOTback to summary
private static final char DOT
DOT_SETback to summary
private static final String DOT_SET
dotStandInback to summary
private int dotStandIn

The stand-in character for the 'dot' set, represented by '.' in patterns. This is allocated the first time it is needed, and reused thereafter.

END_OF_RULEback to summary
private static final char END_OF_RULE
ESCAPEback to summary
private static final char ESCAPE
FORWARD_RULE_OPback to summary
private static final char FORWARD_RULE_OP
FUNCTIONback to summary
private static final char FUNCTION
FWDREV_RULE_OPback to summary
private static final char FWDREV_RULE_OP
HALF_ENDERSback to summary
private static final String HALF_ENDERS
ID_TOKENback to summary
private static final String ID_TOKEN
ID_TOKEN_LENback to summary
private static final int ID_TOKEN_LEN
idBlockVectorback to summary
public List<String> idBlockVector

PUBLIC data member. A Vector of Strings containing all of the ID blocks in the rule set

ILLEGAL_FUNCback to summary
private static UnicodeSet ILLEGAL_FUNC
ILLEGAL_SEGback to summary
private static UnicodeSet ILLEGAL_SEG
ILLEGAL_TOPback to summary
private static UnicodeSet ILLEGAL_TOP
KLEENE_STARback to summary
private static final char KLEENE_STAR
ONE_OR_MOREback to summary
private static final char ONE_OR_MORE
OPERATORSback to summary
private static final String OPERATORS
parseDataback to summary
private TransliteratorParser.ParseData parseData

Temporary symbol table used during parsing.

QUOTEback to summary
private static final char QUOTE
REVERSE_RULE_OPback to summary
private static final char REVERSE_RULE_OP
RULE_COMMENT_CHARback to summary
private static final char RULE_COMMENT_CHAR
SEGMENT_CLOSEback to summary
private static final char SEGMENT_CLOSE
SEGMENT_OPENback to summary
private static final char SEGMENT_OPEN
segmentObjectsback to summary
private List<StringMatcher> segmentObjects

Vector of StringMatcher objects for segments. Used during the parsing of a single rule. segmentStandins.charAt(0) is the standin for "$1" and corresponds to StringMatcher object segmentObjects.elementAt(0), etc.

segmentStandinsback to summary
private StringBuffer segmentStandins

String of standins for segments. Used during the parsing of a single rule. segmentStandins.charAt(0) is the standin for "$1" and corresponds to StringMatcher object segmentObjects.elementAt(0), etc.

undefinedVariableNameback to summary
private String undefinedVariableName

When we encounter an undefined variable, we do not immediately signal an error, in case we are defining this variable, e.g., "$a = [a-z];". Instead, we save the name of the undefined variable, and substitute in the placeholder char variableLimit - 1, and decrement variableLimit.

VARIABLE_DEF_OPback to summary
private static final char VARIABLE_DEF_OP
variableLimitback to summary
private char variableLimit

The last available stand-in for variables. This is discovered dynamically. At any point during parsing, available variables are variableNext..variableLimit-1. During variable definition we use the special value variableLimit-1 as a placeholder.

variableNamesback to summary
private Map<String, char[]> variableNames

Temporary table of variable names. When parsing is complete, this is copied into data.variableNames.

variableNextback to summary
private char variableNext

The next available stand-in for variables. This starts at some point in the private use area (discovered dynamically) and increments up toward variableLimit. At any point during parsing, available variables are variableNext..variableLimit-1.

variablesVectorback to summary
private List<Object> variablesVector

Temporary vector of set variables. When parsing is complete, this is copied into the array data.variables. As with data.variables, element 0 corresponds to character data.variablesBase.

ZERO_OR_ONEback to summary
private static final char ZERO_OR_ONE

Constructor Detail

TransliteratorParserback to summary
public TransliteratorParser()

Constructor.

Method Detail

appendVariableDefback to summary
private void appendVariableDef(String name, StringBuffer buf)

Append the value of the given variable name to the given StringBuffer.

Exceptions
IllegalIcuArgumentException:
if the name is unknown.
checkVariableRangeback to summary
private void checkVariableRange(int ch, String rule, int start)

Assert that the given character is NOT within the variable range. If it is, signal an error. This is necessary to ensure that the variable range does not overlap characters used in a rule.

generateStandInForback to summary
pack-priv char generateStandInFor(Object obj)

Generate and return a stand-in for a new UnicodeMatcher or UnicodeReplacer. Store the object.

getDotStandInback to summary
pack-priv char getDotStandIn()

Return the stand-in for the dot set. It is allocated the first time and reused thereafter.

getSegmentStandinback to summary
public char getSegmentStandin(int seg)

Return the standin for segment seg (1-based).

parseback to summary
public void parse(String rules, int dir)

Parse a set of rules. After the parse completes, examine the public data members for results.

parsePragmaback to summary
private int parsePragma(String rule, int pos, int limit)

Parse a pragma. This method assumes resemblesPragma() has already returned true.

Parameters
pos:int

offset to the first non-whitespace character of the rule.

limit:int

pointer past the last character of the rule.

Returns:int

the position index after the final ';' of the pragma, or -1 on failure.

parseRuleback to summary
private int parseRule(String rule, int pos, int limit)

MAIN PARSER. Parse the next rule in the given rule string, starting at pos. Return the index after the last character parsed. Do not parse characters at or after limit.

Important Note

The character at pos must be a non-whitespace character that is not the comment character. This method handles quoting, escaping, and whitespace removal. It parses the end-of-rule character. It recognizes context and cursor indicators. Once it does a lexical breakdown of the rule at pos, it creates a rule object and adds it to our rule list. This method is tightly coupled to the inner class RuleHalf.

parseRulesback to summary
pack-priv void parseRules(TransliteratorParser.RuleBody ruleArray, int dir)

Parse an array of zero or more rules. The strings in the array are treated as if they were concatenated together, with rule terminators inserted between array elements if not present already. Any previous rules are discarded. Typically this method is called exactly once, during construction. The member this.data will be set to null if there are no rules.

Exceptions
IllegalIcuArgumentException:
if there is a syntax error in the rules
parseSetback to summary
private final char parseSet(String rule, ParsePosition pos)

Parse a UnicodeSet out, store it, and return the stand-in character used to represent it.

pragmaMaximumBackupback to summary
private void pragmaMaximumBackup(int backup)

Set the maximum backup to 'backup', in response to a pragma statement.

pragmaNormalizeRulesback to summary
private void pragmaNormalizeRules(Normalizer.Mode mode)

Deprecated

as a consequence of Normalizer.Mode being deprecated. See corresponding docs for further information.

Begin normalizing all rules using the given mode, in response to a pragma statement.

resemblesPragmaback to summary
pack-priv static boolean resemblesPragma(String rule, int pos, int limit)

Return true if the given rule looks like a pragma.

Parameters
pos:int

offset to the first non-whitespace character of the rule.

limit:int

pointer past the last character of the rule.

ruleEndback to summary
pack-priv static final int ruleEnd(String rule, int start, int limit)
setSegmentObjectback to summary
public void setSegmentObject(int seg, StringMatcher obj)

Set the object for segment seg (1-based).

setVariableRangeback to summary
private void setVariableRange(int start, int end)

Set the variable range to [start, end] (inclusive).

syntaxErrorback to summary
pack-priv static final void syntaxError(String msg, String rule, int start)

Throw an exception indicating a syntax error. Search the rule string for the probable end of the rule. Of course, if the error is that the end of rule marker is missing, then the rule end will not be found. In any case the rule start will be correctly reported.

Parameters
msg:String

error description

rule:String

pattern string

start:int

position of first character of current rule

com.ibm.icu.text back to summary

private Class TransliteratorParser.ParseData

extends Object
implements SymbolTable
Class Inheritance
All Implemented Interfaces
com.ibm.icu.text.SymbolTable

This class implements the SymbolTable interface. It is used during parsing to give UnicodeSet access to variables that have been defined so far. Note that it uses variablesVector, _not_ data.variables.

Constructor Summary

AccessConstructor and Description
private

Method Summary

Modifier and TypeMethod and Description
public boolean
isMatcher(int ch)

Return true if the given character is a matcher standin or a plain character (non standin).

public boolean
isReplacer(int ch)

Return true if the given character is a replacer standin or a plain character (non standin).

public char[]
lookup(String
the symbolic name to lookup
name
)

Implements com.ibm.icu.text.SymbolTable.lookup.

Implement SymbolTable API.
public UnicodeMatcher
lookupMatcher(int
a 32-bit code point from 0 to 0x10FFFF inclusive.
ch
)

Implements com.ibm.icu.text.SymbolTable.lookupMatcher.

Implement SymbolTable API.
public String
parseReference(String
the text to parse for the name
text
,
ParsePosition
on entry, the index of the first character to parse. This is the character following the SYMBOL_REF character. On exit, the index after the last parsed character. If the parse failed, pos is unchanged on exit.
pos
,
int
the index after the last character to be parsed.
limit
)

Implements com.ibm.icu.text.SymbolTable.parseReference.

Implement SymbolTable API.
Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Constructor Detail

ParseDataback to summary
private ParseData()

Method Detail

isMatcherback to summary
public boolean isMatcher(int ch)

Return true if the given character is a matcher standin or a plain character (non standin).

isReplacerback to summary
public boolean isReplacer(int ch)

Return true if the given character is a replacer standin or a plain character (non standin).

lookupback to summary
public char[] lookup(String name)

Implements com.ibm.icu.text.SymbolTable.lookup.

Implement SymbolTable API.

Parameters
name:String

Doc from com.ibm.icu.text.SymbolTable.lookup.

the symbolic name to lookup

Returns:char[]

Doc from com.ibm.icu.text.SymbolTable.lookup.

a char array containing the name's value, or null if there is no mapping for s.

Annotations
@Override
lookupMatcherback to summary
public UnicodeMatcher lookupMatcher(int ch)

Implements com.ibm.icu.text.SymbolTable.lookupMatcher.

Implement SymbolTable API.

Parameters
ch:int

Doc from com.ibm.icu.text.SymbolTable.lookupMatcher.

a 32-bit code point from 0 to 0x10FFFF inclusive.

Returns:UnicodeMatcher

Doc from com.ibm.icu.text.SymbolTable.lookupMatcher.

the UnicodeMatcher object represented by the given character, or null if there is no mapping for ch.

Annotations
@Override
parseReferenceback to summary
public String parseReference(String text, ParsePosition pos, int limit)

Implements com.ibm.icu.text.SymbolTable.parseReference.

Implement SymbolTable API. Parse out a symbol reference name.

Parameters
text:String

Doc from com.ibm.icu.text.SymbolTable.parseReference.

the text to parse for the name

pos:ParsePosition

Doc from com.ibm.icu.text.SymbolTable.parseReference.

on entry, the index of the first character to parse. This is the character following the SYMBOL_REF character. On exit, the index after the last parsed character. If the parse failed, pos is unchanged on exit.

limit:int

Doc from com.ibm.icu.text.SymbolTable.parseReference.

the index after the last character to be parsed.

Returns:String

Doc from com.ibm.icu.text.SymbolTable.parseReference.

the parsed name, or null if there is no valid symbolic name at the given position.

Annotations
@Override
com.ibm.icu.text back to summary

private Class TransliteratorParser.RuleArray

extends RuleBody
Class Inheritance

RuleBody subclass for a String[] array.

Field Summary

Modifier and TypeField and Description
pack-priv String[]
pack-priv int
i

Constructor Summary

AccessConstructor and Description
public
RuleArray(String[] array)

Method Summary

Modifier and TypeMethod and Description
public String
handleNextLine()

Implements abstract com.ibm.icu.text.TransliteratorParser.RuleBody.handleNextLine.

Subclass method to return the next line of the source.
public void
reset()

Implements abstract com.ibm.icu.text.TransliteratorParser.RuleBody.reset.

Reset to the first line of the source.
Inherited from com.ibm.icu.text.TransliteratorParser.RuleBody:
nextLine

Field Detail

arrayback to summary
pack-priv String[] array
iback to summary
pack-priv int i

Constructor Detail

RuleArrayback to summary
public RuleArray(String[] array)

Method Detail

handleNextLineback to summary
public String handleNextLine()

Implements abstract com.ibm.icu.text.TransliteratorParser.RuleBody.handleNextLine.

Doc from com.ibm.icu.text.TransliteratorParser.RuleBody.handleNextLine.

Subclass method to return the next line of the source.

Annotations
@Override
resetback to summary
public void reset()

Implements abstract com.ibm.icu.text.TransliteratorParser.RuleBody.reset.

Doc from com.ibm.icu.text.TransliteratorParser.RuleBody.reset.

Reset to the first line of the source.

Annotations
@Override
com.ibm.icu.text back to summary

private abstract Class TransliteratorParser.RuleBody

extends Object
Class Inheritance
Known Direct Subclasses
com.ibm.icu.text.TransliteratorParser.RuleArray

A private abstract class representing the interface to rule source code that is broken up into lines. Handles the folding of lines terminated by a backslash. This folding is limited; it does not account for comments, quotes, or escapes, so its use to be limited.

Constructor Summary

AccessConstructor and Description
private

Method Summary

Modifier and TypeMethod and Description
pack-priv abstract String
handleNextLine()

Subclass method to return the next line of the source.

pack-priv String
nextLine()

Retrieve the next line of the source, or return null if none.

pack-priv abstract void
reset()

Reset to the first line of the source.

Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Constructor Detail

RuleBodyback to summary
private RuleBody()

Method Detail

handleNextLineback to summary
pack-priv abstract String handleNextLine()

Subclass method to return the next line of the source.

nextLineback to summary
pack-priv String nextLine()

Retrieve the next line of the source, or return null if none. Folds lines terminated by a backslash into the next line, without regard for comments, quotes, or escapes.

resetback to summary
pack-priv abstract void reset()

Reset to the first line of the source.

com.ibm.icu.text back to summary

private Class TransliteratorParser.RuleHalf

extends Object
Class Inheritance

A class representing one side of a rule. This class knows how to parse half of a rule. It is tightly coupled to the method TransliteratorParser.parseRule().

Field Summary

Modifier and TypeField and Description
public boolean
public boolean
public int
public int
public int
private int
private int
nextSegmentNumber

The segment number from 1..n of the next '(' we see during parsing; 1-based.

public int
public String

Constructor Summary

AccessConstructor and Description
private

Method Summary

Modifier and TypeMethod and Description
public boolean
isValidInput(TransliteratorParser parser)

Return true if this half looks like valid input, that is, does not contain functions or other special output-only elements.

public boolean
isValidOutput(TransliteratorParser parser)

Return true if this half looks like valid output, that is, does not contain quantifiers or other special input-only elements.

public int

Returns:

the index after the terminating character, or if limit was reached, limit
parse
(String rule, int pos, int limit, TransliteratorParser parser)

Parse one side of a rule, stopping at either the limit, the END_OF_RULE character, or an operator.

private int

Returns:

the index after the terminating character, or if limit was reached, limit
parseSection
(String rule, int pos, int limit, TransliteratorParser parser, StringBuffer
buffer into which to accumulate the rule pattern characters, either literal characters from the rule or standins for UnicodeMatcher objects including segments.
buf
,
UnicodeSet
the set of special characters that is illegal during this parse.
illegal
,
boolean
if true, then we've already seen a '(' and pos on entry points right after it. Accumulate everything up to the closing ')', put it in a segment matcher object, generate a standin for it, and add the standin to buf. As a side effect, update the segments vector with a reference to the segment matcher. This works recursively for nested segments. If isSegment is false, just accumulate characters into buf.
isSegment
)

Parse a section of one side of a rule, stopping at either the limit, the END_OF_RULE character, an operator, or a segment close character.

pack-priv void
removeContext()

Remove context.

Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Field Detail

anchorEndback to summary
public boolean anchorEnd
anchorStartback to summary
public boolean anchorStart
anteback to summary
public int ante
cursorback to summary
public int cursor
cursorOffsetback to summary
public int cursorOffset
cursorOffsetPosback to summary
private int cursorOffsetPos
nextSegmentNumberback to summary
private int nextSegmentNumber

The segment number from 1..n of the next '(' we see during parsing; 1-based.

postback to summary
public int post
textback to summary
public String text

Constructor Detail

RuleHalfback to summary
private RuleHalf()

Method Detail

isValidInputback to summary
public boolean isValidInput(TransliteratorParser parser)

Return true if this half looks like valid input, that is, does not contain functions or other special output-only elements.

isValidOutputback to summary
public boolean isValidOutput(TransliteratorParser parser)

Return true if this half looks like valid output, that is, does not contain quantifiers or other special input-only elements.

parseback to summary
public int parse(String rule, int pos, int limit, TransliteratorParser parser)

Parse one side of a rule, stopping at either the limit, the END_OF_RULE character, or an operator.

Returns:int

the index after the terminating character, or if limit was reached, limit

parseSectionback to summary
private int parseSection(String rule, int pos, int limit, TransliteratorParser parser, StringBuffer buf, UnicodeSet illegal, boolean isSegment)

Parse a section of one side of a rule, stopping at either the limit, the END_OF_RULE character, an operator, or a segment close character. This method parses both a top-level rule half and a segment within such a rule half. It calls itself recursively to parse segments and nested segments.

Parameters
buf:StringBuffer

buffer into which to accumulate the rule pattern characters, either literal characters from the rule or standins for UnicodeMatcher objects including segments.

illegal:UnicodeSet

the set of special characters that is illegal during this parse.

isSegment:boolean

if true, then we've already seen a '(' and pos on entry points right after it. Accumulate everything up to the closing ')', put it in a segment matcher object, generate a standin for it, and add the standin to buf. As a side effect, update the segments vector with a reference to the segment matcher. This works recursively for nested segments. If isSegment is false, just accumulate characters into buf.

Returns:int

the index after the terminating character, or if limit was reached, limit

removeContextback to summary
pack-priv void removeContext()

Remove context.