Top Description Inners Fields Constructors Methods
org.antlr.runtime

public Class TokenRewriteStream

extends CommonTokenStream
Class Inheritance
Imports
java.util.*

Useful for dumping out the input stream after doing some augmentation or other manipulations. You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a String. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the toString() method(s) check to see if there is an operation at the current index. If so, the operation is done and then normal String rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :) Since the operations are done lazily at toString-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1. Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example, CharStream input = new ANTLRFileStream("input"); TLexer lex = new TLexer(input); TokenRewriteStream tokens = new TokenRewriteStream(lex); T parser = new T(tokens); parser.startRule(); Then in the rules, you can execute Token t,u; ... input.insertAfter(t, "text to put after t");} input.insertAfter(u, "text after u");} System.out.println(tokens.toString()); Actually, you have to cast the 'input' to a TokenRewriteStream. :( You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer: tokens.insertAfter("pass1", t, "text to put after t");} tokens.insertAfter("pass2", u, "text after u");} System.out.println(tokens.toString("pass1")); System.out.println(tokens.toString("pass2")); If you don't use named rewrite streams, a "default" stream is used as the first example shows.

Nested and Inner Type Summary

Modifier and TypeClass and Description
pack-priv class
pack-priv class
TokenRewriteStream.ReplaceOp

I'm going to try replacing range from x..y with (y-x)+1 ReplaceOp instructions.

public class

Field Summary

Modifier and TypeField and Description
public static final String
protected Map<String, Integer>
lastRewriteTokenIndexes

Map String (program name) → Integer index

public static final int
public static final int
protected Map<String, List<TokenRewriteStream.RewriteOperation>>
programs

You may have multiple, named streams of rewrite operations.

Inherited from org.antlr.runtime.CommonTokenStream:
channel

Constructor Summary

AccessConstructor and Description
public
public
public
TokenRewriteStream(TokenSource tokenSource, int channel)

Method Summary

Modifier and TypeMethod and Description
protected String
public void
delete(int index)

public void
delete(int from, int to)

public void
delete(Token indexT)

public void
delete(Token from, Token to)

public void
delete(String programName, int from, int to)

public void
delete(String programName, Token from, Token to)

public void
public void
deleteProgram(String programName)

Reset the program so that no instructions exist

protected <T extends TokenRewriteStream.RewriteOperation> List<? extends T>
protected <T extends TokenRewriteStream.RewriteOperation> List<? extends T>
getKindOfOps(List<? extends TokenRewriteStream.RewriteOperation> rewrites, Class<T> kind, int before)

Get all operations before an index of a particular kind

public int
protected int
protected List<TokenRewriteStream.RewriteOperation>
protected void
init()

private List<TokenRewriteStream.RewriteOperation>
public void
public void
insertAfter(int index, Object text)

public void
insertAfter(String programName, Token t, Object text)

public void
insertAfter(String programName, int index, Object text)

public void
public void
insertBefore(int index, Object text)

public void
insertBefore(String programName, Token t, Object text)

public void
insertBefore(String programName, int index, Object text)

protected Map<Integer, ? extends TokenRewriteStream.RewriteOperation>
reduceToSingleOperationPerIndex(List<? extends TokenRewriteStream.RewriteOperation> rewrites)

We need to combine operations and report invalid operations (like overlapping replaces that are not completed nested).

public void
replace(int index, Object text)

public void
replace(int from, int to, Object text)

public void
replace(Token indexT, Object text)

public void
replace(Token from, Token to, Object text)

public void
replace(String programName, int from, int to, Object text)

public void
replace(String programName, Token from, Token to, Object text)

public void
rollback(int instructionIndex)

public void
rollback(String programName, int instructionIndex)

Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream.

protected void
setLastRewriteTokenIndex(String programName, int i)

public String
public String
toDebugString(int start, int end)

public String
public String
toOriginalString(int start, int end)

public String
toString()

Overrides org.antlr.runtime.BufferedTokenStream.toString.

Grab *all* tokens from stream and return string
public String
toString(String programName)

public String
toString(int start, int end)

Overrides org.antlr.runtime.BufferedTokenStream.toString.

Implements org.antlr.runtime.TokenStream.toString.

Return the text of all tokens from start to stop, inclusive.
public String
toString(String programName, int start, int end)

Inherited from org.antlr.runtime.CommonTokenStream:
consumegetNumberOfOnChannelTokensLBLTresetsetTokenSourcesetupskipOffTokenChannelsskipOffTokenChannelsReverse

Field Detail

DEFAULT_PROGRAM_NAMEback to summary
public static final String DEFAULT_PROGRAM_NAME
lastRewriteTokenIndexesback to summary
protected Map<String, Integer> lastRewriteTokenIndexes

Map String (program name) → Integer index

MIN_TOKEN_INDEXback to summary
public static final int MIN_TOKEN_INDEX
PROGRAM_INIT_SIZEback to summary
public static final int PROGRAM_INIT_SIZE
programsback to summary
protected Map<String, List<TokenRewriteStream.RewriteOperation>> programs

You may have multiple, named streams of rewrite operations. I'm calling these things "programs." Maps String (name) → rewrite (List)

Constructor Detail

TokenRewriteStreamback to summary
public TokenRewriteStream()
TokenRewriteStreamback to summary
public TokenRewriteStream(TokenSource tokenSource)
TokenRewriteStreamback to summary
public TokenRewriteStream(TokenSource tokenSource, int channel)

Method Detail

catOpTextback to summary
protected String catOpText(Object a, Object b)
deleteback to summary
public void delete(int index)
deleteback to summary
public void delete(int from, int to)
deleteback to summary
public void delete(Token indexT)
deleteback to summary
public void delete(Token from, Token to)
deleteback to summary
public void delete(String programName, int from, int to)
deleteback to summary
public void delete(String programName, Token from, Token to)
deleteProgramback to summary
public void deleteProgram()
deleteProgramback to summary
public void deleteProgram(String programName)

Reset the program so that no instructions exist

getKindOfOpsback to summary
protected <T extends TokenRewriteStream.RewriteOperation> List<? extends T> getKindOfOps(List<? extends TokenRewriteStream.RewriteOperation> rewrites, Class<T> kind)
getKindOfOpsback to summary
protected <T extends TokenRewriteStream.RewriteOperation> List<? extends T> getKindOfOps(List<? extends TokenRewriteStream.RewriteOperation> rewrites, Class<T> kind, int before)

Get all operations before an index of a particular kind

getLastRewriteTokenIndexback to summary
public int getLastRewriteTokenIndex()
getLastRewriteTokenIndexback to summary
protected int getLastRewriteTokenIndex(String programName)
getProgramback to summary
protected List<TokenRewriteStream.RewriteOperation> getProgram(String name)
initback to summary
protected void init()
initializeProgramback to summary
private List<TokenRewriteStream.RewriteOperation> initializeProgram(String name)
insertAfterback to summary
public void insertAfter(Token t, Object text)
insertAfterback to summary
public void insertAfter(int index, Object text)
insertAfterback to summary
public void insertAfter(String programName, Token t, Object text)
insertAfterback to summary
public void insertAfter(String programName, int index, Object text)
insertBeforeback to summary
public void insertBefore(Token t, Object text)
insertBeforeback to summary
public void insertBefore(int index, Object text)
insertBeforeback to summary
public void insertBefore(String programName, Token t, Object text)
insertBeforeback to summary
public void insertBefore(String programName, int index, Object text)
reduceToSingleOperationPerIndexback to summary
protected Map<Integer, ? extends TokenRewriteStream.RewriteOperation> reduceToSingleOperationPerIndex(List<? extends TokenRewriteStream.RewriteOperation> rewrites)

We need to combine operations and report invalid operations (like overlapping replaces that are not completed nested). Inserts to same index need to be combined etc... Here are the cases: I.i.u I.j.v leave alone, nonoverlapping I.i.u I.i.v combine: Iivu R.i-j.u R.x-y.v | i-j in x-y delete first R R.i-j.u R.i-j.v delete first R R.i-j.u R.x-y.v | x-y in i-j ERROR R.i-j.u R.x-y.v | boundaries overlap ERROR Delete special case of replace (text==null): D.i-j.u D.x-y.v | boundaries overlap combine to max(min)..max(right) I.i.u R.x-y.v | i in (x+1)-y delete I (since insert before we're not deleting i) I.i.u R.x-y.v | i not in (x+1)-y leave alone, nonoverlapping R.x-y.v I.i.u | i in x-y ERROR R.x-y.v I.x.u R.x-y.uv (combine, delete I) R.x-y.v I.i.u | i not in x-y leave alone, nonoverlapping I.i.u = insert u before op @ index i R.x-y.u = replace x-y indexed tokens with u First we need to examine replaces. For any replace op: 1. wipe out any insertions before op within that range. 2. Drop any replace op before that is contained completely within that range. 3. Throw exception upon boundary overlap with any previous replace. Then we can deal with inserts: 1. for any inserts to same index, combine even if not adjacent. 2. for any prior replace with same left boundary, combine this insert with replace and delete this replace. 3. throw exception if index in same range as previous replace Don't actually delete; make op null in list. Easier to walk list. Later we can throw as we add to index → op map. Note that I.2 R.2-2 will wipe out I.2 even though, technically, the inserted stuff would be before the replace range. But, if you add tokens in front of a method body '{' and then delete the method body, I think the stuff before the '{' you added should disappear too. Return a map from token index to operation.

replaceback to summary
public void replace(int index, Object text)
replaceback to summary
public void replace(int from, int to, Object text)
replaceback to summary
public void replace(Token indexT, Object text)
replaceback to summary
public void replace(Token from, Token to, Object text)
replaceback to summary
public void replace(String programName, int from, int to, Object text)
replaceback to summary
public void replace(String programName, Token from, Token to, Object text)
rollbackback to summary
public void rollback(int instructionIndex)
rollbackback to summary
public void rollback(String programName, int instructionIndex)

Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream.

setLastRewriteTokenIndexback to summary
protected void setLastRewriteTokenIndex(String programName, int i)
toDebugStringback to summary
public String toDebugString()
toDebugStringback to summary
public String toDebugString(int start, int end)
toOriginalStringback to summary
public String toOriginalString()
toOriginalStringback to summary
public String toOriginalString(int start, int end)
toStringback to summary
public String toString()

Overrides org.antlr.runtime.BufferedTokenStream.toString.

Doc from org.antlr.runtime.BufferedTokenStream.toString.

Grab *all* tokens from stream and return string

Returns:String

Doc from java.lang.Object.toString.

a string representation of the object.

Annotations
@Override
toStringback to summary
public String toString(String programName)
toStringback to summary
public String toString(int start, int end)

Overrides org.antlr.runtime.BufferedTokenStream.toString.

Implements org.antlr.runtime.TokenStream.toString.

Doc from org.antlr.runtime.TokenStream.toString.

Return the text of all tokens from start to stop, inclusive. If the stream does not buffer all the tokens then it can just return "" or null; Users should not access $ruleLabel.text in an action of course in that case.

Annotations
@Override
toStringback to summary
public String toString(String programName, int start, int end)
org.antlr.runtime back to summary

pack-priv Class TokenRewriteStream.InsertBeforeOp

extends RewriteOperation
Class Inheritance

Field Summary

Inherited from org.antlr.runtime.TokenRewriteStream.RewriteOperation:
indexinstructionIndextext

Constructor Summary

AccessConstructor and Description
public
InsertBeforeOp(int index, Object text)

Method Summary

Modifier and TypeMethod and Description
public int
execute(StringBuffer buf)

Overrides org.antlr.runtime.TokenRewriteStream.RewriteOperation.execute.

Execute the rewrite operation by possibly adding to the buffer.
Inherited from org.antlr.runtime.TokenRewriteStream.RewriteOperation:
toString

Constructor Detail

InsertBeforeOpback to summary
public InsertBeforeOp(int index, Object text)

Method Detail

executeback to summary
public int execute(StringBuffer buf)

Overrides org.antlr.runtime.TokenRewriteStream.RewriteOperation.execute.

Doc from org.antlr.runtime.TokenRewriteStream.RewriteOperation.execute.

Execute the rewrite operation by possibly adding to the buffer. Return the index of the next token to operate on.

Annotations
@Override
org.antlr.runtime back to summary

pack-priv Class TokenRewriteStream.ReplaceOp

extends RewriteOperation
Class Inheritance

I'm going to try replacing range from x..y with (y-x)+1 ReplaceOp instructions.

Field Summary

Modifier and TypeField and Description
protected int
Inherited from org.antlr.runtime.TokenRewriteStream.RewriteOperation:
indexinstructionIndextext

Constructor Summary

AccessConstructor and Description
public
ReplaceOp(int from, int to, Object text)

Method Summary

Modifier and TypeMethod and Description
public int
execute(StringBuffer buf)

Overrides org.antlr.runtime.TokenRewriteStream.RewriteOperation.execute.

Execute the rewrite operation by possibly adding to the buffer.
public String
toString()

Overrides org.antlr.runtime.TokenRewriteStream.RewriteOperation.toString.

Returns a string representation of the object.

Field Detail

lastIndexback to summary
protected int lastIndex

Constructor Detail

ReplaceOpback to summary
public ReplaceOp(int from, int to, Object text)

Method Detail

executeback to summary
public int execute(StringBuffer buf)

Overrides org.antlr.runtime.TokenRewriteStream.RewriteOperation.execute.

Doc from org.antlr.runtime.TokenRewriteStream.RewriteOperation.execute.

Execute the rewrite operation by possibly adding to the buffer. Return the index of the next token to operate on.

Annotations
@Override
toStringback to summary
public String toString()

Overrides org.antlr.runtime.TokenRewriteStream.RewriteOperation.toString.

Doc from java.lang.Object.toString.

Returns a string representation of the object.

Returns:String

a string representation of the object.

Annotations
@Override
org.antlr.runtime back to summary

public Class TokenRewriteStream.RewriteOperation

extends Object
Class Inheritance
Known Direct Subclasses
org.antlr.runtime.TokenRewriteStream.InsertBeforeOp, org.antlr.runtime.TokenRewriteStream.ReplaceOp

Field Summary

Modifier and TypeField and Description
protected int
index

Token buffer index.

protected int
instructionIndex

What index into rewrites List are we?

protected Object

Constructor Summary

AccessConstructor and Description
protected
RewriteOperation(int index)

protected
RewriteOperation(int index, Object text)

Method Summary

Modifier and TypeMethod and Description
public int
execute(StringBuffer buf)

Execute the rewrite operation by possibly adding to the buffer.

public String
toString()

Overrides java.lang.Object.toString.

Returns a string representation of the object.
Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAllwaitwaitwait

Field Detail

indexback to summary
protected int index

Token buffer index.

instructionIndexback to summary
protected int instructionIndex

What index into rewrites List are we?

textback to summary
protected Object text

Constructor Detail

RewriteOperationback to summary
protected RewriteOperation(int index)
RewriteOperationback to summary
protected RewriteOperation(int index, Object text)

Method Detail

executeback to summary
public int execute(StringBuffer buf)

Execute the rewrite operation by possibly adding to the buffer. Return the index of the next token to operate on.

toStringback to summary
public String toString()

Overrides java.lang.Object.toString.

Doc from java.lang.Object.toString.

Returns a string representation of the object.

Returns:String

a string representation of the object.

Annotations
@Override