Modifier and Type	Class and Description
public static class	DataFileWriter.AppendWriteException Thrown by `append(Object)` when an exception occurs while writing a datum to the buffer.
private class	DataFileWriter.BufferedFileOutputStream

Modifier and Type	Field and Description
private long	blockCount
private NonCopyingByteArrayOutputStream	buffer
private BinaryEncoder	bufOut
private Codec	codec
private DatumWriter<D>	dout
private boolean	flushOnEveryBlock
private boolean	isOpen
private final Map<String, byte[]>	meta
private DataFileWriter<D>.BufferedFileOutputStream	out
private Schema	schema
private byte[]	sync
private int	syncInterval
private OutputStream	underlyingStream
private BinaryEncoder	vout

Access	Constructor and Description
public	DataFileWriter(DatumWriter<D> dout) Construct a writer, not yet open.

Method Summary

Modifier and Type	Method and Description
public void	append(D datum) Append a datum to the file.
public void	appendAllFrom(DataFileStream<D> otherFile, boolean recompress) Appends data from another file.
public void	appendEncoded(ByteBuffer datum) Expert: Append a pre-encoded datum to the file.
public DataFileWriter<D>	appendTo(File file) Open a writer appending to an existing file.
public DataFileWriter<D>	appendTo(SeekableInput reading the existing file. in, OutputStream positioned at the end of the existing file. out) Open a writer appending to an existing file.
private void	assertNotOpen()
private void	assertOpen()
private int	bufferInUse()
public void	close() Implements java.io.Closeable.close. Flush and close the file.
public DataFileWriter<D>	create(Schema schema, File file) Open a new file for data matching a schema with a random sync.
public DataFileWriter<D>	create(Schema schema, OutputStream outs) Open a new file for data matching a schema with a random sync.
public DataFileWriter<D>	create(Schema schema, OutputStream outs, byte[] sync) Open a new file for data matching a schema with an explicit sync.
public void	flush() Implements java.io.Flushable.flush. Calls sync() and then flushes the current state of the file.
public void	fSync() If this writer was instantiated using a File or using an Syncable instance, this method flushes all buffers for this writer to disk.
private static byte[]	generateSync()
private void	init(OutputStream outs)
public boolean	Returns: - true if this writer flushes the block to the stream every time a sync marker is written. Else returns false. isFlushOnEveryBlock()
public static boolean	isReservedMeta(String key)
private void	resetBufferTo(int size)
public DataFileWriter<D>	setCodec(CodecFactory c) Configures this writer to use the given codec.
public void	setFlushOnEveryBlock(boolean - If set to false, this writer will not flush the block to the stream until flush() is explicitly called. flushOnEveryBlock) Set whether this writer should flush the block to the stream every time a sync marker is written.
public DataFileWriter<D>	setMeta(String key, byte[] value) Set a metadata property.
public DataFileWriter<D>	setMeta(String key, String value) Set a metadata property.
public DataFileWriter<D>	setMeta(String key, long value) Set a metadata property.
private DataFileWriter<D>	setMetaInternal(String key, byte[] value)
private DataFileWriter<D>	setMetaInternal(String key, String value)
public DataFileWriter<D>	Returns: this DataFileWriter setSyncInterval(int the approximate number of uncompressed bytes to write in each block syncInterval) Set the synchronization interval for this file, in bytes.
public long	sync() Return the current position as a value that may be passed to `DataFileReader#seek(long)`.
private void	writeBlock()
private void	writeIfBlockFull()

Inherited from java.lang.Object:: clone equals finalize getClass hashCode notify notifyAll toString wait wait wait

Field Detail

blockCount	back to summary
private long blockCount

buffer	back to summary
private NonCopyingByteArrayOutputStream buffer

bufOut	back to summary
private BinaryEncoder bufOut

codec	back to summary
private Codec codec

dout	back to summary
private DatumWriter<D> dout

flushOnEveryBlock	back to summary
private boolean flushOnEveryBlock

isOpen	back to summary
private boolean isOpen

meta	back to summary
private final Map<String, byte[]> meta

out	back to summary
private DataFileWriter<D>.BufferedFileOutputStream out

schema	back to summary
private Schema schema

sync	back to summary
private byte[] sync

syncInterval	back to summary
private int syncInterval

underlyingStream	back to summary
private OutputStream underlyingStream

vout	back to summary
private BinaryEncoder vout

Constructor Detail

DataFileWriter	back to summary
public DataFileWriter(DatumWriter<D> dout) Construct a writer, not yet open.

Method Detail

append back to summary

append	back to summary
public void append(D datum) throws IOException Append a datum to the file. See Also `AppendWriteException`

public void append(D datum) throws IOException

Append a datum to the file.

See Also: AppendWriteException

appendAllFrom	back to summary
public void appendAllFrom(DataFileStream<D> otherFile, boolean recompress) throws IOException Appends data from another file. otherFile must have the same schema. Data blocks will be copied without de-serializing data. If the codecs of the two files are compatible, data blocks are copied directly without decompression. If the codecs are not compatible, blocks from otherFile are uncompressed and then compressed using this file's codec. If the recompress flag is set all blocks are decompressed and then compressed using this file's codec. This is useful when the two files have compatible compression codecs but different codec options. For example, one might append a file compressed with deflate at compression level 1 to a file with deflate at compression level 7. If recompress is false, blocks will be copied without changing the compression level. If true, they will be converted to the new compression level.

appendAllFrom

back to summary

public void appendAllFrom(DataFileStream<D> otherFile, boolean recompress) throws IOException

Appends data from another file. otherFile must have the same schema. Data blocks will be copied without de-serializing data. If the codecs of the two files are compatible, data blocks are copied directly without decompression. If the codecs are not compatible, blocks from otherFile are uncompressed and then compressed using this file's codec.

If the recompress flag is set all blocks are decompressed and then compressed using this file's codec. This is useful when the two files have compatible compression codecs but different codec options. For example, one might append a file compressed with deflate at compression level 1 to a file with deflate at compression level 7. If recompress is false, blocks will be copied without changing the compression level. If true, they will be converted to the new compression level.

appendEncoded	back to summary
public void appendEncoded(ByteBuffer datum) throws IOException Expert: Append a pre-encoded datum to the file. No validation is performed to check that the encoding conforms to the file's schema. Appending non-conforming data may result in an unreadable file.

appendTo	back to summary
public DataFileWriter<D> appendTo(File file) throws IOException Open a writer appending to an existing file.

appendTo	back to summary
public DataFileWriter<D> appendTo(SeekableInput in, OutputStream out) throws IOException Open a writer appending to an existing file. Since 1.9.0 this method does not close in. Parameters in:SeekableInput reading the existing file. out:OutputStream positioned at the end of the existing file.

appendTo

back to summary

public DataFileWriter<D> appendTo(SeekableInput in, OutputStream out) throws IOException

Open a writer appending to an existing file. Since 1.9.0 this method does not close in.

Parameters

in:SeekableInput: reading the existing file.
out:OutputStream: positioned at the end of the existing file.

assertNotOpen	back to summary
private void assertNotOpen()

assertOpen	back to summary
private void assertOpen()

bufferInUse	back to summary
private int bufferInUse()

close	back to summary
public void close() throws IOException Implements java.io.Closeable.close. Flush and close the file. Annotations @Override Exceptions IOException: Doc from java.io.Closeable.close. if an I/O error occurs

back to summary

public void close() throws IOException

Implements java.io.Closeable.close.

Flush and close the file.

Annotations

@Override

Exceptions

IOException:

Doc from java.io.Closeable.close.

if an I/O error occurs

create	back to summary
public DataFileWriter<D> create(Schema schema, File file) throws IOException Open a new file for data matching a schema with a random sync.

create	back to summary
public DataFileWriter<D> create(Schema schema, OutputStream outs) throws IOException Open a new file for data matching a schema with a random sync.

create	back to summary
public DataFileWriter<D> create(Schema schema, OutputStream outs, byte[] sync) throws IOException Open a new file for data matching a schema with an explicit sync.

flush	back to summary
public void flush() throws IOException Implements java.io.Flushable.flush. Calls sync() and then flushes the current state of the file. Annotations @Override Exceptions IOException: Doc from java.io.Flushable.flush. If an I/O error occurs

flush

back to summary

public void flush() throws IOException

Implements java.io.Flushable.flush.

Calls sync() and then flushes the current state of the file.

Annotations

@Override

Exceptions

IOException:

Doc from java.io.Flushable.flush.

If an I/O error occurs

fSync	back to summary
public void fSync() throws IOException If this writer was instantiated using a File or using an Syncable instance, this method flushes all buffers for this writer to disk. In other cases, this method behaves exactly like flush().

generateSync	back to summary
private static byte[] generateSync()

init	back to summary
private void init(OutputStream outs) throws IOException

isFlushOnEveryBlock	back to summary
public boolean isFlushOnEveryBlock() Returns:boolean - true if this writer flushes the block to the stream every time a sync marker is written. Else returns false.

isReservedMeta	back to summary
public static boolean isReservedMeta(String key)

resetBufferTo	back to summary
private void resetBufferTo(int size) throws IOException

setCodec	back to summary
public DataFileWriter<D> setCodec(CodecFactory c) Configures this writer to use the given codec. May not be reset after writes have begun.

setFlushOnEveryBlock	back to summary
public void setFlushOnEveryBlock(boolean flushOnEveryBlock) Set whether this writer should flush the block to the stream every time a sync marker is written. By default, the writer will flush the buffer each time a sync marker is written (if the block size limit is reached or the sync() is called. Parameters flushOnEveryBlock:boolean - If set to false, this writer will not flush the block to the stream until flush() is explicitly called.

setFlushOnEveryBlock

back to summary

public void setFlushOnEveryBlock(boolean flushOnEveryBlock)

Set whether this writer should flush the block to the stream every time a sync marker is written. By default, the writer will flush the buffer each time a sync marker is written (if the block size limit is reached or the sync() is called.

Parameters

flushOnEveryBlock:boolean: - If set to false, this writer will not flush the block to the stream until flush() is explicitly called.

setMeta	back to summary
public DataFileWriter<D> setMeta(String key, byte[] value) Set a metadata property.

setMeta	back to summary
public DataFileWriter<D> setMeta(String key, String value) Set a metadata property.

setMeta	back to summary
public DataFileWriter<D> setMeta(String key, long value) Set a metadata property.

setMetaInternal	back to summary
private DataFileWriter<D> setMetaInternal(String key, byte[] value)

setMetaInternal	back to summary
private DataFileWriter<D> setMetaInternal(String key, String value)

setSyncInterval	back to summary
public DataFileWriter<D> setSyncInterval(int syncInterval) Set the synchronization interval for this file, in bytes. Valid values range from 32 to 2^30 Suggested values are between 2K and 2M The stream is flushed by default at the end of each synchronization interval. If setFlushOnEveryBlock(boolean) is called with param set to false, then the block may not be flushed to the stream after the sync marker is written. In this case, the flush() must be called to flush the stream. Invalid values throw IllegalArgumentException Parameters syncInterval:int the approximate number of uncompressed bytes to write in each block Returns:DataFileWriter<D> this DataFileWriter

setSyncInterval

back to summary

public DataFileWriter<D> setSyncInterval(int syncInterval)

Set the synchronization interval for this file, in bytes. Valid values range from 32 to 2^30 Suggested values are between 2K and 2M The stream is flushed by default at the end of each synchronization interval. If setFlushOnEveryBlock(boolean) is called with param set to false, then the block may not be flushed to the stream after the sync marker is written. In this case, the flush() must be called to flush the stream. Invalid values throw IllegalArgumentException

Parameters

syncInterval:int: the approximate number of uncompressed bytes to write in each block

Returns:DataFileWriter<D>

this DataFileWriter

sync back to summary

sync	back to summary
public long sync() throws IOException Return the current position as a value that may be passed to `DataFileReader#seek(long)`. Forces the end of the current block, emitting a synchronization marker. By default, this will also flush the block to the stream. If setFlushOnEveryBlock(boolean) is called with param set to false, then this method may not flush the block. In this case, the flush() must be called to flush the stream.

public long sync() throws IOException

Return the current position as a value that may be passed to DataFileReader#seek(long). Forces the end of the current block, emitting a synchronization marker. By default, this will also flush the block to the stream. If setFlushOnEveryBlock(boolean) is called with param set to false, then this method may not flush the block. In this case, the flush() must be called to flush the stream.

writeBlock	back to summary
private void writeBlock() throws IOException

writeIfBlockFull	back to summary
private void writeIfBlockFull() throws IOException

org.apache.avro.file back to summary

public Class DataFileWriter.AppendWriteException

Class Inheritance

extends RuntimeException

java.lang.Object
java.lang.Throwable
java.lang.Exception
java.lang.RuntimeException
org.apache.avro.file.DataFileWriter.AppendWriteException

Thrown by append(Object) when an exception occurs while writing a datum to the buffer. When this is thrown, the file is unaltered and may continue to be appended to.

Constructor Summary

Access	Constructor and Description
public	AppendWriteException(Exception e)

Constructor Detail

AppendWriteException	back to summary
public AppendWriteException(Exception e)

org.apache.avro.file back to summary

private Class DataFileWriter.BufferedFileOutputStream

Class Inheritance

extends BufferedOutputStream

java.lang.Object
java.io.OutputStream
java.io.FilterOutputStream
java.io.BufferedOutputStream
org.apache.avro.file.DataFileWriter.BufferedFileOutputStream

Nested and Inner Type Summary

Modifier and Type	Class and Description
private class	DataFileWriter.BufferedFileOutputStream.PositionFilter

Field Summary

Modifier and Type	Field and Description
private long	position

Inherited from java.io.BufferedOutputStream:: buf count

Constructor Summary

Access	Constructor and Description
public	BufferedFileOutputStream(OutputStream out)

Method Summary

Modifier and Type	Method and Description
public synchronized void	flush() Overrides java.io.BufferedOutputStream.flush. Implements java.io.Flushable.flush. Flushes this buffered output stream.
public long	tell()

Inherited from java.io.BufferedOutputStream:: write write

Field Detail

position	back to summary
private long position

Constructor Detail

BufferedFileOutputStream	back to summary
public BufferedFileOutputStream(OutputStream out) throws IOException

Method Detail

flush

back to summary

public synchronized void flush() throws IOException

Overrides java.io.BufferedOutputStream.flush.

Implements java.io.Flushable.flush.

Doc from java.io.BufferedOutputStream.flush.

Flushes this buffered output stream. This forces any buffered output bytes to be written out to the underlying output stream.

Annotations

@Override

Exceptions

IOException:: if an I/O error occurs.

tell	back to summary
public long tell()

org.apache.avro.file back to summary

private Class DataFileWriter.BufferedFileOutputStream.PositionFilter

Class Inheritance

extends FilterOutputStream

java.lang.Object
java.io.OutputStream
java.io.FilterOutputStream
org.apache.avro.file.DataFileWriter.BufferedFileOutputStream.PositionFilter

Field Summary

Inherited from java.io.FilterOutputStream:: out

Constructor Summary

Access	Constructor and Description
public	PositionFilter(OutputStream out)

Method Summary

Modifier and Type	Method and Description
public void	write(byte[] the data. b, int the start offset in the data. off, int the number of bytes to write. len) Overrides java.io.FilterOutputStream.write. Writes `len` bytes from the specified `byte` array starting at offset `off` to this output stream.

Inherited from java.io.FilterOutputStream:: close flush write write

Constructor Detail

PositionFilter	back to summary
public PositionFilter(OutputStream out) throws IOException

Method Detail

write back to summary

public void write(byte[] b, int off, int len) throws IOException

Overrides java.io.FilterOutputStream.write.

Doc from java.io.FilterOutputStream.write.

Writes len bytes from the specified byte array starting at offset off to this output stream.

Parameters

b:byte[]: the data.
off:int: the start offset in the data.
len:int: the number of bytes to write.

Annotations

@Override

Exceptions

IOException:: if an I/O error occurs.

public Class DataFileWriter<D>

Nested and Inner Type Summary

Field Summary

Constructor Summary

Method Summary

Field Detail

Constructor Detail

Method Detail

public Class DataFileWriter.AppendWriteException

Constructor Summary

Constructor Detail

private Class DataFileWriter.BufferedFileOutputStream

Nested and Inner Type Summary

Field Summary

Constructor Summary

Method Summary

Field Detail

Constructor Detail

Method Detail

private Class DataFileWriter.BufferedFileOutputStream.PositionFilter

Field Summary

Constructor Summary

Method Summary

Constructor Detail

Method Detail