Top Description Inners Fields Constructors Methods
org.apache.avro.io

public Class BlockingBinaryEncoder

extends BufferedBinaryEncoder
Class Inheritance
Imports
java.io.IOException, .OutputStream, java.nio.ByteBuffer, java.util.Arrays, org.apache.avro.AvroTypeException, .Schema
A BinaryEncoder implementation that writes large arrays and maps as a sequence of blocks. So long as individual primitive values fit in memory, arbitrarily long arrays and maps may be written and subsequently read without exhausting memory. Values are buffered until the specified block size would be exceeded, minimizing block overhead.

Use EncoderFactory#blockingBinaryEncoder(OutputStream, BinaryEncoder) to construct and configure.

BlockingBinaryEncoder buffers writes, data may not appear on the output until flush() is called.

BlockingBinaryEncoder is not thread-safe

See Also
BinaryEncoder, EncoderFactory, Encoder

Nested and Inner Type Summary

Modifier and TypeClass and Description
private static class
BlockingBinaryEncoder.BlockedValue

Field Summary

Modifier and TypeField and Description
private BlockingBinaryEncoder.BlockedValue[]
blockStack

The state stack.

private byte[]
buf

Hides org.apache.avro.io.BufferedBinaryEncoder.buf.

The buffer to hold the bytes before being written into the underlying stream.
private byte[]
headerBuffer

private int
pos

Hides org.apache.avro.io.BufferedBinaryEncoder.pos.

Index into the location in buf, where next byte can be written.
private static final int
STACK_STEP

private int
stackTop

Constructor Summary

AccessConstructor and Description
pack-priv
BlockingBinaryEncoder(OutputStream out, int blockBufferSize, int binaryEncoderBufferSize)

Method Summary

Modifier and TypeMethod and Description
public int
bytesBuffered()

Overrides org.apache.avro.io.BufferedBinaryEncoder.bytesBuffered.

Returns the number of bytes currently buffered by this encoder.
private boolean
check()

private void
compact()

Only call if you're there are REGULAR-state values on the stack.

pack-priv BlockingBinaryEncoder
configure(OutputStream out, int blockBufferSize, int binaryEncoderBufferSize)

private void
doWriteBytes(byte[] bytes, int start, int len)

private void
endBlockedValue()

private void
ensureBounds(int l)

Hides org.apache.avro.io.BufferedBinaryEncoder.ensureBounds.

Ensures that the buffer has at least num bytes free to write to between its current position and the end.
private void
expandStack()

private void
finishOverflow()

Called when we've finished writing the last item in an overflow buffer.

public void
flush()

Overrides org.apache.avro.io.BufferedBinaryEncoder.flush.

Implements java.io.Flushable.flush.

Flushes this stream by writing any buffered output to the underlying stream.
public void
setItemCount(long
The number of startItem() calls to follow.
itemCount)

Overrides org.apache.avro.io.BinaryEncoder.setItemCount.

Call this method before writing a batch of items in an array or a map.
public void
startItem()

Overrides org.apache.avro.io.BinaryEncoder.startItem.

Start a new item of an array or map.
private void
write(byte[] b, int off, int len)

public void
writeArrayEnd()

Overrides org.apache.avro.io.BinaryEncoder.writeArrayEnd.

Call this method to finish writing an array.
public void
writeArrayStart()

Overrides org.apache.avro.io.BinaryEncoder.writeArrayStart.

Call this method to start writing an array.
public void
writeBoolean(boolean b)

Overrides org.apache.avro.io.BufferedBinaryEncoder.writeBoolean.

Write a boolean value.
public void
writeDouble(double d)

Overrides org.apache.avro.io.BufferedBinaryEncoder.writeDouble.

Write a double.
public void
writeFixed(byte[]
The contents to write
bytes, int
The position within bytes where the contents start.
start, int
The number of bytes to write.
len)

Overrides org.apache.avro.io.BufferedBinaryEncoder.writeFixed.

Writes a fixed size binary object.
public void
writeFixed(ByteBuffer bytes)

Overrides org.apache.avro.io.BufferedBinaryEncoder.writeFixed.

Writes a fixed from a ByteBuffer.
public void
writeFloat(float f)

Overrides org.apache.avro.io.BufferedBinaryEncoder.writeFloat.

Write a float.
public void
writeIndex(int unionIndex)

Overrides org.apache.avro.io.BinaryEncoder.writeIndex.

Call this method to write the tag of a union.
public void
writeInt(int n)

Overrides org.apache.avro.io.BufferedBinaryEncoder.writeInt.

Writes a 32-bit integer.
public void
writeLong(long n)

Overrides org.apache.avro.io.BufferedBinaryEncoder.writeLong.

Write a 64-bit integer.
public void
writeMapEnd()

Overrides org.apache.avro.io.BinaryEncoder.writeMapEnd.

Call this method to terminate the inner-most, currently-opened map.
public void
writeMapStart()

Overrides org.apache.avro.io.BinaryEncoder.writeMapStart.

Call this to start a new map.
protected void
writeZero()

Overrides org.apache.avro.io.BufferedBinaryEncoder.writeZero.

Write a zero byte to the underlying output.
Inherited from org.apache.avro.io.BufferedBinaryEncoder:
configure

Field Detail

blockStackback to summary
private BlockingBinaryEncoder.BlockedValue[] blockStack

The state stack.

bufback to summary
private byte[] buf

Hides org.apache.avro.io.BufferedBinaryEncoder.buf.

The buffer to hold the bytes before being written into the underlying stream.

headerBufferback to summary
private byte[] headerBuffer
posback to summary
private int pos

Hides org.apache.avro.io.BufferedBinaryEncoder.pos.

Index into the location in buf, where next byte can be written.

STACK_STEPback to summary
private static final int STACK_STEP
stackTopback to summary
private int stackTop

Constructor Detail

BlockingBinaryEncoderback to summary
pack-priv BlockingBinaryEncoder(OutputStream out, int blockBufferSize, int binaryEncoderBufferSize)

Method Detail

bytesBufferedback to summary
public int bytesBuffered()

Overrides org.apache.avro.io.BufferedBinaryEncoder.bytesBuffered.

Doc from org.apache.avro.io.BinaryEncoder.bytesBuffered.

Returns the number of bytes currently buffered by this encoder. If this Encoder does not buffer, this will always return zero.

Call flush() to empty the buffer to the underlying output.

Annotations
@Override
checkback to summary
private boolean check()
compactback to summary
private void compact() throws IOException

Only call if you're there are REGULAR-state values on the stack.

configureback to summary
pack-priv BlockingBinaryEncoder configure(OutputStream out, int blockBufferSize, int binaryEncoderBufferSize)
doWriteBytesback to summary
private void doWriteBytes(byte[] bytes, int start, int len) throws IOException
endBlockedValueback to summary
private void endBlockedValue() throws IOException
ensureBoundsback to summary
private void ensureBounds(int l) throws IOException

Hides org.apache.avro.io.BufferedBinaryEncoder.ensureBounds.

Doc from org.apache.avro.io.BufferedBinaryEncoder.ensureBounds.

Ensures that the buffer has at least num bytes free to write to between its current position and the end. This will not expand the buffer larger than its current size, for writes larger than or near to the size of the buffer, we flush the buffer and write directly to the output, bypassing the buffer.

expandStackback to summary
private void expandStack()
finishOverflowback to summary
private void finishOverflow() throws IOException

Called when we've finished writing the last item in an overflow buffer. When this is finished, the top of the stack will be an empty block in the "regular" state.

flushback to summary
public void flush() throws IOException

Overrides org.apache.avro.io.BufferedBinaryEncoder.flush.

Implements java.io.Flushable.flush.

Doc from java.io.Flushable.flush.

Flushes this stream by writing any buffered output to the underlying stream.

Annotations
@Override
Exceptions
IOException:
If an I/O error occurs
setItemCountback to summary
public void setItemCount(long itemCount) throws IOException

Overrides org.apache.avro.io.BinaryEncoder.setItemCount.

Doc from org.apache.avro.io.Encoder.setItemCount.

Call this method before writing a batch of items in an array or a map. Then for each item, call startItem() followed by any of the other write methods of Encoder. The number of calls to startItem() must be equal to the count specified in setItemCount. Once a batch is completed you can start another batch with setItemCount.

Parameters
itemCount:long

The number of startItem() calls to follow.

Annotations
@Override
startItemback to summary
public void startItem() throws IOException

Overrides org.apache.avro.io.BinaryEncoder.startItem.

Doc from org.apache.avro.io.Encoder.startItem.

Start a new item of an array or map. See writeArrayStart for usage information.

Annotations
@Override
writeback to summary
private void write(byte[] b, int off, int len) throws IOException
writeArrayEndback to summary
public void writeArrayEnd() throws IOException

Overrides org.apache.avro.io.BinaryEncoder.writeArrayEnd.

Doc from org.apache.avro.io.Encoder.writeArrayEnd.

Call this method to finish writing an array. See writeArrayStart for usage information.

Annotations
@Override
writeArrayStartback to summary
public void writeArrayStart() throws IOException

Overrides org.apache.avro.io.BinaryEncoder.writeArrayStart.

Doc from org.apache.avro.io.Encoder.writeArrayStart.

Call this method to start writing an array. When starting to serialize an array, call writeArrayStart. Then, before writing any data for any item call setItemCount followed by a sequence of startItem() and the item itself. The number of startItem() should match the number specified in setItemCount. When actually writing the data of the item, you can call any Encoder method (e.g., writeLong). When all items of the array have been written, call writeArrayEnd. As an example, let's say you want to write an array of records, the record consisting of an Long field and a Boolean field. Your code would look something like this: 

out.writeArrayStart();
out.setItemCount(list.size());
for (Record r : list) {
  out.startItem();
  out.writeLong(r.longField);
  out.writeBoolean(r.boolField);
}
out.writeArrayEnd();
Annotations
@Override
writeBooleanback to summary
public void writeBoolean(boolean b) throws IOException

Overrides org.apache.avro.io.BufferedBinaryEncoder.writeBoolean.

Doc from org.apache.avro.io.Encoder.writeBoolean.

Write a boolean value.

Annotations
@Override
writeDoubleback to summary
public void writeDouble(double d) throws IOException

Overrides org.apache.avro.io.BufferedBinaryEncoder.writeDouble.

Doc from org.apache.avro.io.Encoder.writeDouble.

Write a double.

Annotations
@Override
writeFixedback to summary
public void writeFixed(byte[] bytes, int start, int len) throws IOException

Overrides org.apache.avro.io.BufferedBinaryEncoder.writeFixed.

Doc from org.apache.avro.io.Encoder.writeFixed.

Writes a fixed size binary object.

Parameters
bytes:byte[]

The contents to write

start:int

The position within bytes where the contents start.

len:int

The number of bytes to write.

Annotations
@Override
writeFixedback to summary
public void writeFixed(ByteBuffer bytes) throws IOException

Overrides org.apache.avro.io.BufferedBinaryEncoder.writeFixed.

Doc from org.apache.avro.io.Encoder.writeFixed.

Writes a fixed from a ByteBuffer.

Annotations
@Override
writeFloatback to summary
public void writeFloat(float f) throws IOException

Overrides org.apache.avro.io.BufferedBinaryEncoder.writeFloat.

Doc from org.apache.avro.io.Encoder.writeFloat.

Write a float.

Annotations
@Override
writeIndexback to summary
public void writeIndex(int unionIndex) throws IOException

Overrides org.apache.avro.io.BinaryEncoder.writeIndex.

Doc from org.apache.avro.io.Encoder.writeIndex.

Call this method to write the tag of a union. As an example of usage, let's say you want to write a union, whose second branch is a record consisting of an Long field and a Boolean field. Your code would look something like this: 

out.writeIndex(1);
out.writeLong(record.longField);
out.writeBoolean(record.boolField);
Annotations
@Override
writeIntback to summary
public void writeInt(int n) throws IOException

Overrides org.apache.avro.io.BufferedBinaryEncoder.writeInt.

Doc from org.apache.avro.io.Encoder.writeInt.

Writes a 32-bit integer.

Annotations
@Override
writeLongback to summary
public void writeLong(long n) throws IOException

Overrides org.apache.avro.io.BufferedBinaryEncoder.writeLong.

Doc from org.apache.avro.io.Encoder.writeLong.

Write a 64-bit integer.

Annotations
@Override
writeMapEndback to summary
public void writeMapEnd() throws IOException

Overrides org.apache.avro.io.BinaryEncoder.writeMapEnd.

Doc from org.apache.avro.io.Encoder.writeMapEnd.

Call this method to terminate the inner-most, currently-opened map. See writeArrayStart for more details.

Annotations
@Override
writeMapStartback to summary
public void writeMapStart() throws IOException

Overrides org.apache.avro.io.BinaryEncoder.writeMapStart.

Doc from org.apache.avro.io.Encoder.writeMapStart.

Call this to start a new map. See writeArrayStart for details on usage. As an example of usage, let's say you want to write a map of records, the record consisting of an Long field and a Boolean field. Your code would look something like this: 

out.writeMapStart();
out.setItemCount(list.size());
for (Map.Entry entry : map.entrySet()) {
  out.startItem();
  out.writeString(entry.getKey());
  out.writeLong(entry.getValue().longField);
  out.writeBoolean(entry.getValue().boolField);
}
out.writeMapEnd();
Annotations
@Override
writeZeroback to summary
protected void writeZero() throws IOException

Overrides org.apache.avro.io.BufferedBinaryEncoder.writeZero.

Doc from org.apache.avro.io.BinaryEncoder.writeZero.

Write a zero byte to the underlying output.

Annotations
@Override
org.apache.avro.io back to summary

private Class BlockingBinaryEncoder.BlockedValue

extends Object
Class Inheritance

Nested and Inner Type Summary

Modifier and TypeClass and Description
public static enum
BlockingBinaryEncoder.BlockedValue.State

Field Summary

Modifier and TypeField and Description
public int
items

Number of items in this blocked value that are stored in the buffer.

public long
itemsLeftToWrite

Number of items left to write

public int
lastFullItem

The index one past the last byte for the previous item.

public int
start

The location in the buffer where this blocked value starts

public BlockingBinaryEncoder.BlockedValue.State
state

The state of this BlockedValue

public Schema.Type
type

The type of this blocked value (ARRAY or MAP).

Constructor Summary

AccessConstructor and Description
public
BlockedValue()

Create a ROOT instance.

Method Summary

Modifier and TypeMethod and Description
public boolean
check(BlockingBinaryEncoder.BlockedValue prev, int pos)

Check invariants of this and also the BlockedValue containing this.

Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Field Detail

itemsback to summary
public int items

Number of items in this blocked value that are stored in the buffer.

itemsLeftToWriteback to summary
public long itemsLeftToWrite

Number of items left to write

lastFullItemback to summary
public int lastFullItem

The index one past the last byte for the previous item. If this is the first item, this is same as start.

startback to summary
public int start

The location in the buffer where this blocked value starts

stateback to summary
public BlockingBinaryEncoder.BlockedValue.State state

The state of this BlockedValue

typeback to summary
public Schema.Type type

The type of this blocked value (ARRAY or MAP).

Constructor Detail

BlockedValueback to summary
public BlockedValue()

Create a ROOT instance.

Method Detail

checkback to summary
public boolean check(BlockingBinaryEncoder.BlockedValue prev, int pos)

Check invariants of this and also the BlockedValue containing this.

org.apache.avro.io back to summary

public final Enum BlockingBinaryEncoder.BlockedValue.State

extends Enum<BlockingBinaryEncoder.BlockedValue.State>
Class Inheritance

Field Summary

Modifier and TypeField and Description
public static final BlockingBinaryEncoder.BlockedValue.State
OVERFLOW

Represents a blocked-value whose current block is in the overflow state.

public static final BlockingBinaryEncoder.BlockedValue.State
REGULAR

Represents the "regular" case, i.e., a blocked-value whose current block is fully contained in the buffer.

public static final BlockingBinaryEncoder.BlockedValue.State
ROOT

The bottom element of our stack represents being _outside_ of a blocked value.

Constructor Summary

AccessConstructor and Description
private
State()

Field Detail

OVERFLOWback to summary
public static final BlockingBinaryEncoder.BlockedValue.State OVERFLOW

Represents a blocked-value whose current block is in the overflow state. In this case, BlockedValue#start is zero. The header for such a block has _already been written_ (we've written out a header indicating that the block has a single item, and we put a "zero" down for the byte-count to indicate that we don't know the physical length of the buffer. Any blocks _containing_ this block must be in the OVERFLOW state.

REGULARback to summary
public static final BlockingBinaryEncoder.BlockedValue.State REGULAR

Represents the "regular" case, i.e., a blocked-value whose current block is fully contained in the buffer. In this case, BlockedValue#start points to the start of the blocks _data_ -- but no room has been left for a header! When this block is terminated, it's data will have to be moved over a bit to make room for the header.

ROOTback to summary
public static final BlockingBinaryEncoder.BlockedValue.State ROOT

The bottom element of our stack represents being _outside_ of a blocked value.

Constructor Detail

Stateback to summary
private State()

Method Detail

valueOfback to summary
public static BlockingBinaryEncoder.BlockedValue.State valueOf(String name)
valuesback to summary
public static BlockingBinaryEncoder.BlockedValue.State[] values()