Top Description Fields Constructors Methods
org.xerial.snappy

public Class SnappyOutputStream

extends OutputStream
Class Inheritance
Known Direct Subclasses
org.xerial.snappy.SnappyHadoopCompatibleOutputStream
Imports
org.xerial.snappy.buffer.BufferAllocatorFactory, .BufferAllocator, .CachedBufferAllocator, java.io.IOException, .OutputStream
This class implements a stream filter for writing compressed data using Snappy.

The input data is blocked into 32kb size (in default), and each block is compressed and then passed to the given OutputStream.

The output data format is:
  1. snappy codec header defined in SnappyCodec (8 bytes)
  2. codec version (4 bytes integer, at least 1) and compatible version (4 bytes integer)
  3. compressed block 1 : a pair of (compressed data size [4 byte integer. Big-endian], compressed data...)
  4. compressed block 2
  5. ...

Note that the compressed data created by SnappyOutputStream cannot be uncompressed by Snappy#uncompress(byte[]) since the output formats of Snappy#compress(byte[]) and SnappyOutputStream are different. Use SnappyInputStream for uncompress the data generated by SnappyOutputStream.

Author
leo

Field Summary

Modifier and TypeField and Description
private final int
blockSize

private boolean
closed

pack-priv static final int
DEFAULT_BLOCK_SIZE

private boolean
headerWritten

protected byte[]
inputBuffer

private final BufferAllocator
inputBufferAllocator

private int
inputCursor

public static final int
MAX_BLOCK_SIZE

pack-priv static final int
MIN_BLOCK_SIZE

protected final OutputStream
out

protected byte[]
outputBuffer

private final BufferAllocator
outputBufferAllocator

private int
outputCursor

Constructor Summary

AccessConstructor and Description
public
SnappyOutputStream(OutputStream out)

public
SnappyOutputStream(OutputStream out, int
byte size of the internal buffer size
blockSize)

public
SnappyOutputStream(OutputStream out, int blockSize, BufferAllocatorFactory bufferAllocatorFactory)

Method Summary

Modifier and TypeMethod and Description
public void
close()

Overrides java.io.OutputStream.close.

Implements java.io.Closeable.close.

close the stream

protected void
compressInput()

protected void
dumpOutput()

public void
flush()

Overrides java.io.OutputStream.flush.

Implements java.io.Flushable.flush.

Flushes this output stream and forces any buffered output bytes to be written out.

private boolean
hasSufficientOutputBufferFor(int inputSize)

public void
rawWrite(Object
array data of any type (e.g., byte[], float[], long[], ...)
array, int byteOffset, int byteLength)

Compress the raw byte array data.

pack-priv static int
readInt(byte[] buffer, int pos)

public void
write(byte[]
the data.
b, int
the start offset in the data.
byteOffset, int
the number of bytes to write.
byteLength)

Overrides java.io.OutputStream.write.

Writes len bytes from the specified byte array starting at offset off to this output stream.

public void
write(long[]
input array
d, int
offset in the array
off, int
the number of elements in the array to copy
len)

Compress the input long array data

public void
write(double[]
input array
f, int
offset in the array
off, int
the number of elements in the array to copy
len)

Compress the input double array data

public void
write(float[]
input array
f, int
offset in the array
off, int
the number of elements in the array to copy
len)

Compress the input float array data

public void
write(int[]
input array
f, int
offset in the array
off, int
the number of elements in the array to copy
len)

Compress the input int array data

public void
write(short[]
input array
f, int
offset in the array
off, int
the number of elements in the array to copy
len)

Compress the input short array data

public void
write(long[] d)

Compress the input array data

public void
write(double[] f)

Compress the input array data

public void
write(float[] f)

Compress the input array data

public void
write(int[] f)

Compress the input array data

public void
write(short[] f)

Compress the input array data

public void
write(int
the byte.
b)

Implements abstract java.io.OutputStream.write.

Writes the specified byte to this output stream.

protected void
writeBlockPreemble()

Optionally write a preemble before a block.

protected void
writeCurrentDataSize()

protected int
writeHeader()

pack-priv static void
writeInt(byte[] dst, int offset, int v)

Inherited from java.io.OutputStream:
nullOutputStreamwrite

Field Detail

blockSizeback to summary
private final int blockSize
closedback to summary
private boolean closed
DEFAULT_BLOCK_SIZEback to summary
pack-priv static final int DEFAULT_BLOCK_SIZE
headerWrittenback to summary
private boolean headerWritten
inputBufferback to summary
protected byte[] inputBuffer
inputBufferAllocatorback to summary
private final BufferAllocator inputBufferAllocator
inputCursorback to summary
private int inputCursor
MAX_BLOCK_SIZEback to summary
public static final int MAX_BLOCK_SIZE
MIN_BLOCK_SIZEback to summary
pack-priv static final int MIN_BLOCK_SIZE
outback to summary
protected final OutputStream out
outputBufferback to summary
protected byte[] outputBuffer
outputBufferAllocatorback to summary
private final BufferAllocator outputBufferAllocator
outputCursorback to summary
private int outputCursor

Constructor Detail

SnappyOutputStreamback to summary
public SnappyOutputStream(OutputStream out)
SnappyOutputStreamback to summary
public SnappyOutputStream(OutputStream out, int blockSize)
Parameters
blockSize:int

byte size of the internal buffer size

Exceptions
IllegalArgumentException:
when blockSize is larger than 512 MiB
SnappyOutputStreamback to summary
public SnappyOutputStream(OutputStream out, int blockSize, BufferAllocatorFactory bufferAllocatorFactory)

Method Detail

closeback to summary
public void close() throws IOException

Overrides java.io.OutputStream.close.

Implements java.io.Closeable.close.

close the stream

Annotations
@Override
Exceptions
IOException:

Doc from java.io.OutputStream.close.

if an I/O error occurs.

compressInputback to summary
protected void compressInput() throws IOException
dumpOutputback to summary
protected void dumpOutput() throws IOException
flushback to summary
public void flush() throws IOException

Overrides java.io.OutputStream.flush.

Implements java.io.Flushable.flush.

Doc from java.io.OutputStream.flush.

Flushes this output stream and forces any buffered output bytes to be written out. The general contract of flush is that calling it is an indication that, if any bytes previously written have been buffered by the implementation of the output stream, such bytes should immediately be written to their intended destination.

If the intended destination of this stream is an abstraction provided by the underlying operating system, for example a file, then flushing the stream guarantees only that bytes previously written to the stream are passed to the operating system for writing; it does not guarantee that they are actually written to a physical device such as a disk drive.

Annotations
@Override
Exceptions
IOException:
if an I/O error occurs.
hasSufficientOutputBufferForback to summary
private boolean hasSufficientOutputBufferFor(int inputSize)
rawWriteback to summary
public void rawWrite(Object array, int byteOffset, int byteLength) throws IOException

Compress the raw byte array data.

Parameters
array:Object

array data of any type (e.g., byte[], float[], long[], ...)

readIntback to summary
pack-priv static int readInt(byte[] buffer, int pos)
writeback to summary
public void write(byte[] b, int byteOffset, int byteLength) throws IOException

Overrides java.io.OutputStream.write.

Doc from java.io.OutputStream.write.

Writes len bytes from the specified byte array starting at offset off to this output stream. The general contract for write(b, off, len) is that some of the bytes in the array b are written to the output stream in order; element b[off] is the first byte written and b[off+len-1] is the last byte written by this operation.

If b is null, a NullPointerException is thrown.

If off is negative, or len is negative, or off+len is greater than the length of the array b, then an IndexOutOfBoundsException is thrown.

Parameters
b:byte[]

the data.

byteOffset:int

the start offset in the data.

byteLength:int

the number of bytes to write.

Annotations
@Override
Exceptions
IOException:
if an I/O error occurs. In particular, an IOException is thrown if the output stream is closed.
writeback to summary
public void write(long[] d, int off, int len) throws IOException

Compress the input long array data

Parameters
d:long[]

input array

off:int

offset in the array

len:int

the number of elements in the array to copy

writeback to summary
public void write(double[] f, int off, int len) throws IOException

Compress the input double array data

Parameters
f:double[]

input array

off:int

offset in the array

len:int

the number of elements in the array to copy

writeback to summary
public void write(float[] f, int off, int len) throws IOException

Compress the input float array data

Parameters
f:float[]

input array

off:int

offset in the array

len:int

the number of elements in the array to copy

writeback to summary
public void write(int[] f, int off, int len) throws IOException

Compress the input int array data

Parameters
f:int[]

input array

off:int

offset in the array

len:int

the number of elements in the array to copy

writeback to summary
public void write(short[] f, int off, int len) throws IOException

Compress the input short array data

Parameters
f:short[]

input array

off:int

offset in the array

len:int

the number of elements in the array to copy

writeback to summary
public void write(long[] d) throws IOException

Compress the input array data

writeback to summary
public void write(double[] f) throws IOException

Compress the input array data

writeback to summary
public void write(float[] f) throws IOException

Compress the input array data

writeback to summary
public void write(int[] f) throws IOException

Compress the input array data

writeback to summary
public void write(short[] f) throws IOException

Compress the input array data

writeback to summary
public void write(int b) throws IOException

Implements abstract java.io.OutputStream.write.

Writes the specified byte to this output stream. The general contract for write is that one byte is written to the output stream. The byte to be written is the eight low-order bits of the argument b. The 24 high-order bits of b are ignored.

Parameters
b:int

Doc from java.io.OutputStream.write.

the byte.

Annotations
@Override
Exceptions
IOException:

Doc from java.io.OutputStream.write.

if an I/O error occurs. In particular, an IOException may be thrown if the output stream has been closed.

writeBlockPreembleback to summary
protected void writeBlockPreemble()

Optionally write a preemble before a block. Hadoop requires the actual block data size being written. This base implementation does nothing. Derive implementation can call writeCurrentDataSize().

writeCurrentDataSizeback to summary
protected void writeCurrentDataSize()
writeHeaderback to summary
protected int writeHeader()
writeIntback to summary
pack-priv static void writeInt(byte[] dst, int offset, int v)