Top Description Inners Fields Constructors Methods
com.amazon.ion.impl

pack-priv Class IonUTF8

extends Object
Class Inheritance
Imports
com.amazon.ion.IonException, java.io.Closeable, .IOException, .OutputStream

this class holds the various constants and helper functions Ion uses to "understand" UTF-8 encoded input. This may duplicate values that are available in Java's Character class, but the goal is to fully control and isolate the UTF-8 conversion here. The helper functions are intended to avoid the various bit twiddling errors that can easily occur when working with bits. As these functions are all static final methods, and very short, the Java compilers should find them very easy to in-line. (in previous testing I have observed that the Sun JVM heavily in-lines such methods).

Nested and Inner Type Summary

Modifier and TypeClass and Description
public static class
public static class

Field Summary

Modifier and TypeField and Description
private static final int
private static final int
private static final int
private static final int
private static final int
private static final int
private static final int
private static final int
private static final int
private static final int
private static final int
private static final int
private static final int
private static final int
private static final int
private static final int
private static final int
private static final int

Constructor Summary

AccessConstructor and Description
pack-priv

Method Summary

Modifier and TypeMethod and Description
public static final int

Returns:

number of bytes written to the output array
convertToUTF8Bytes
(int
scalar to convert
unicodeScalar
,
byte[]
user output array to fill with UTF8 bytes
outputBytes
,
int
first array element to fill
offset
,
int
maximum number of array elements to fill
maxLength
)

this helper converts the unicodeScalar to a sequence of utf8 bytes and copies those bytes into the supplied outputBytes array.

public static final int
fourByteScalar(int b1, int b2, int b3, int b4)

public static final int
getAs4BytesReversed(int unicodeScalar)

public static final byte
getByte1Of2(int unicodeScalar)

public static final byte
getByte1Of3(int unicodeScalar)

public static final byte
getByte1Of4(int unicodeScalar)

public static final byte
getByte2Of2(int unicodeScalar)

public static final byte
getByte2Of3(int unicodeScalar)

public static final byte
getByte2Of4(int unicodeScalar)

public static final byte
getByte3Of3(int unicodeScalar)

public static final byte
getByte3Of4(int unicodeScalar)

public static final byte
getByte4Of4(int unicodeScalar)

public static final int
getScalarFrom4BytesReversed(int utf8BytesReversed)

public static final int

Returns:

Unicode scalar
getScalarFromBytes
(byte[]
UTF8 bytes in an array
bytes
,
int
initial array element to decode from
offset
,
int
maximum number of bytes to consume from the array
maxLength
)

this helper converts the bytes starting at offset from UTF8 to a Unicode scalar.

public static final int

Returns:

number of bytes needed to decode the next scalar
getScalarReadLengthFromBytes
(byte[]
UTF8 bytes in user supplied array
bytes
,
int
first array element to read from
offset
,
int
maximum number of bytes to read from the array
maxLength
)

this helper calculates the number of bytes it will consume from the supplied byte array (bytes) if getScalarFromBytes is called with the same parameters.

public static final int
getUnicodeScalarFromSurrogates(int highSurrogate, int lowSurrogate)

public static final int
getUTF8ByteCount(int unicodeScalar)

public static final int
public static final char
highSurrogate(int unicodeScalar)

public static final boolean
public static final boolean
isFourByteScalar(int unicodeScalar)

public static final boolean
public static final boolean
public static final boolean
public static final boolean
isOneByteScalar(int unicodeScalar)

public static final boolean
public static final boolean
isStartByte(int b)

public static final boolean
isSurrogate(int b)

public static final boolean
isThreeByteScalar(int unicodeScalar)

public static final boolean
public static final boolean
isTwoByteScalar(int unicodeScalar)

public static final boolean
public static final char
lowSurrogate(int unicodeScalar)

public static final boolean
needsSurrogateEncoding(int unicodeScalar)

public static final int
packBytesAfter1(int unicodeScalar, int utf8Len)

converts a unicode code point to a 0-3 bytes of UTF8 encoded data and a length - note this doesn't pack a 1 byte character and it returns the start character.

public static final int
threeByteScalar(int b1, int b2, int b3)

public static final char
twoByteScalar(int b1, int b2)

Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Field Detail

HIGH_SURROGATEback to summary
private static final int HIGH_SURROGATE
LOW_SURROGATEback to summary
private static final int LOW_SURROGATE
MAXIMUM_UTF16_1_CHAR_CODE_POINTback to summary
private static final int MAXIMUM_UTF16_1_CHAR_CODE_POINT
SURROGATE_MASKback to summary
private static final int SURROGATE_MASK
SURROGATE_OFFSETback to summary
private static final int SURROGATE_OFFSET
UNICODE_CONTINUATION_BYTE_HEADERback to summary
private static final int UNICODE_CONTINUATION_BYTE_HEADER
UNICODE_CONTINUATION_BYTE_MASKback to summary
private static final int UNICODE_CONTINUATION_BYTE_MASK
UNICODE_FOUR_BYTE_HEADERback to summary
private static final int UNICODE_FOUR_BYTE_HEADER
UNICODE_FOUR_BYTE_MASKback to summary
private static final int UNICODE_FOUR_BYTE_MASK
UNICODE_MAX_FOUR_BYTE_SCALARback to summary
private static final int UNICODE_MAX_FOUR_BYTE_SCALAR
UNICODE_MAX_ONE_BYTE_SCALARback to summary
private static final int UNICODE_MAX_ONE_BYTE_SCALAR
UNICODE_MAX_THREE_BYTE_SCALARback to summary
private static final int UNICODE_MAX_THREE_BYTE_SCALAR
UNICODE_MAX_TWO_BYTE_SCALARback to summary
private static final int UNICODE_MAX_TWO_BYTE_SCALAR
UNICODE_THREE_BYTE_HEADERback to summary
private static final int UNICODE_THREE_BYTE_HEADER
UNICODE_THREE_BYTE_MASKback to summary
private static final int UNICODE_THREE_BYTE_MASK
UNICODE_THREE_BYTES_OR_FEWER_MASKback to summary
private static final int UNICODE_THREE_BYTES_OR_FEWER_MASK
UNICODE_TWO_BYTE_HEADERback to summary
private static final int UNICODE_TWO_BYTE_HEADER
UNICODE_TWO_BYTE_MASKback to summary
private static final int UNICODE_TWO_BYTE_MASK

Constructor Detail

IonUTF8back to summary
pack-priv IonUTF8()

Method Detail

convertToUTF8Bytesback to summary
public static final int convertToUTF8Bytes(int unicodeScalar, byte[] outputBytes, int offset, int maxLength)

this helper converts the unicodeScalar to a sequence of utf8 bytes and copies those bytes into the supplied outputBytes array. If there is insufficient room in the array to hold the generated bytes it will throw an ArrayIndexOutOfBoundsException. It does not check for the validity of the passed in unicodeScalar thoroughly, however it will throw an InvalidUnicodeCodePoint if the value is less than negative or the UTF8 encoding would exceed 4 bytes.

Parameters
unicodeScalar:int

scalar to convert

outputBytes:byte[]

user output array to fill with UTF8 bytes

offset:int

first array element to fill

maxLength:int

maximum number of array elements to fill

Returns:int

number of bytes written to the output array

fourByteScalarback to summary
public static final int fourByteScalar(int b1, int b2, int b3, int b4)
getAs4BytesReversedback to summary
public static final int getAs4BytesReversed(int unicodeScalar)
getByte1Of2back to summary
public static final byte getByte1Of2(int unicodeScalar)
getByte1Of3back to summary
public static final byte getByte1Of3(int unicodeScalar)
getByte1Of4back to summary
public static final byte getByte1Of4(int unicodeScalar)
getByte2Of2back to summary
public static final byte getByte2Of2(int unicodeScalar)
getByte2Of3back to summary
public static final byte getByte2Of3(int unicodeScalar)
getByte2Of4back to summary
public static final byte getByte2Of4(int unicodeScalar)
getByte3Of3back to summary
public static final byte getByte3Of3(int unicodeScalar)
getByte3Of4back to summary
public static final byte getByte3Of4(int unicodeScalar)
getByte4Of4back to summary
public static final byte getByte4Of4(int unicodeScalar)
getScalarFrom4BytesReversedback to summary
public static final int getScalarFrom4BytesReversed(int utf8BytesReversed)
getScalarFromBytesback to summary
public static final int getScalarFromBytes(byte[] bytes, int offset, int maxLength)

this helper converts the bytes starting at offset from UTF8 to a Unicode scalar. This does not check for valid Unicode scalar ranges but simply handle the UTF8 decoding. getScalarReadLengthFromBytes can be used to determine how many bytes would be converted (consumed) in this process if the same parameters are passed in. This will throw ArrayIndexOutOfBoundsException if the array has too few bytes to fully decode the scalar. It will throw InvalidUnicodeCodePoint if the first byte isn't a valid UTF8 initial byte with a length of 4 or less.

Parameters
bytes:byte[]

UTF8 bytes in an array

offset:int

initial array element to decode from

maxLength:int

maximum number of bytes to consume from the array

Returns:int

Unicode scalar

getScalarReadLengthFromBytesback to summary
public static final int getScalarReadLengthFromBytes(byte[] bytes, int offset, int maxLength)

this helper calculates the number of bytes it will consume from the supplied byte array (bytes) if getScalarFromBytes is called with the same parameters. This is in place of having getScalarFromBytes return an object with the bytes consumed and the resultant scalar. This will throw ArrayIndexOutOfBoundsException if there are fewer bytes remaining (maxLength) than needed for a valid UTF8 sequence starting at offset.

Parameters
bytes:byte[]

UTF8 bytes in user supplied array

offset:int

first array element to read from

maxLength:int

maximum number of bytes to read from the array

Returns:int

number of bytes needed to decode the next scalar

getUnicodeScalarFromSurrogatesback to summary
public static final int getUnicodeScalarFromSurrogates(int highSurrogate, int lowSurrogate)
getUTF8ByteCountback to summary
public static final int getUTF8ByteCount(int unicodeScalar)
getUTF8LengthFromFirstByteback to summary
public static final int getUTF8LengthFromFirstByte(int firstByte)
highSurrogateback to summary
public static final char highSurrogate(int unicodeScalar)
isContinueByteUTF8back to summary
public static final boolean isContinueByteUTF8(int b)
isFourByteScalarback to summary
public static final boolean isFourByteScalar(int unicodeScalar)
isFourByteUTF8back to summary
public static final boolean isFourByteUTF8(int b)
isHighSurrogateback to summary
public static final boolean isHighSurrogate(int b)
isLowSurrogateback to summary
public static final boolean isLowSurrogate(int b)
isOneByteScalarback to summary
public static final boolean isOneByteScalar(int unicodeScalar)
isOneByteUTF8back to summary
public static final boolean isOneByteUTF8(int b)
isStartByteback to summary
public static final boolean isStartByte(int b)
isSurrogateback to summary
public static final boolean isSurrogate(int b)
isThreeByteScalarback to summary
public static final boolean isThreeByteScalar(int unicodeScalar)
isThreeByteUTF8back to summary
public static final boolean isThreeByteUTF8(int b)
isTwoByteScalarback to summary
public static final boolean isTwoByteScalar(int unicodeScalar)
isTwoByteUTF8back to summary
public static final boolean isTwoByteUTF8(int b)
lowSurrogateback to summary
public static final char lowSurrogate(int unicodeScalar)
needsSurrogateEncodingback to summary
public static final boolean needsSurrogateEncoding(int unicodeScalar)
packBytesAfter1back to summary
public static final int packBytesAfter1(int unicodeScalar, int utf8Len)

converts a unicode code point to a 0-3 bytes of UTF8 encoded data and a length - note this doesn't pack a 1 byte character and it returns the start character. this is the unpacking routine while (_utf8_pretch_byte_count > 0 && offset < limit) { _utf8_pretch_byte_count--; buffer[offset++] = (byte)((_utf8_pretch_bytes >> (_utf8_pretch_byte_count*8)) & 0xff); }

threeByteScalarback to summary
public static final int threeByteScalar(int b1, int b2, int b3)
twoByteScalarback to summary
public static final char twoByteScalar(int b1, int b2)
com.amazon.ion.impl back to summary

public Class IonUTF8.InvalidUnicodeCodePoint

extends IonException
Class Inheritance

Field Summary

Modifier and TypeField and Description
private static final long

Constructor Summary

AccessConstructor and Description
public
public
public
public

Method Summary

Inherited from com.amazon.ion.IonException:
causeOfType

Field Detail

serialVersionUIDback to summary
private static final long serialVersionUID

Hides com.amazon.ion.IonException.serialVersionUID.

Constructor Detail

InvalidUnicodeCodePointback to summary
public InvalidUnicodeCodePoint()
InvalidUnicodeCodePointback to summary
public InvalidUnicodeCodePoint(String msg)
InvalidUnicodeCodePointback to summary
public InvalidUnicodeCodePoint(Exception e)
InvalidUnicodeCodePointback to summary
public InvalidUnicodeCodePoint(String msg, Exception e)
com.amazon.ion.impl back to summary

public final Class IonUTF8.UTF8ToChar

extends OutputStream
implements Closeable
Class Inheritance
All Implemented Interfaces
java.io.Closeable, java.lang.AutoCloseable

Field Summary

Modifier and TypeField and Description
private final Appendable
private int
private int
private int
private int
private int

Constructor Summary

AccessConstructor and Description
public
UTF8ToChar(Appendable charStream)

Method Summary

Modifier and TypeMethod and Description
public final void
close()

Overrides java.io.OutputStream.close.

Implements java.io.Closeable.close.

Closes this stream and releases any system resources associated with it.

public final Appendable
private void
public final void
write(int
the byte.
b
)

Implements abstract java.io.OutputStream.write.

Writes the specified byte to this output stream.

public final void
write(byte[]
the data.
bytes
)

Overrides java.io.OutputStream.write.

Writes b.length bytes from the specified byte array to this output stream.

public final void
write(byte[]
the data.
bytes
,
int
the start offset in the data.
off
,
int
the number of bytes to write.
len
)

Overrides java.io.OutputStream.write.

Writes len bytes from the specified byte array starting at offset off to this output stream.

private final void
private final void
private final void
Inherited from java.io.OutputStream:
flushnullOutputStream

Field Detail

_char_streamback to summary
private final Appendable _char_stream
_expected_countback to summary
private int _expected_count
_pending_c1back to summary
private int _pending_c1
_pending_c2back to summary
private int _pending_c2
_pending_c3back to summary
private int _pending_c3
_pending_countback to summary
private int _pending_count

Constructor Detail

UTF8ToCharback to summary
public UTF8ToChar(Appendable charStream)

Method Detail

closeback to summary
public final void close() throws IOException

Overrides java.io.OutputStream.close.

Implements java.io.Closeable.close.

Doc from java.io.Closeable.close.

Closes this stream and releases any system resources associated with it. If the stream is already closed then invoking this method has no effect.

As noted in AutoCloseable#close(), cases where the close may fail require careful attention. It is strongly advised to relinquish the underlying resources and to internally mark the Closeable as closed, prior to throwing the IOException.

Annotations
@Override
Exceptions
IOException:
if an I/O error occurs
getAppendableback to summary
public final Appendable getAppendable()
throwBadContinuationByteback to summary
private void throwBadContinuationByte() throws IOException
writeback to summary
public final void write(int b) throws IOException

Implements abstract java.io.OutputStream.write.

Doc from java.io.OutputStream.write.

Writes the specified byte to this output stream. The general contract for write is that one byte is written to the output stream. The byte to be written is the eight low-order bits of the argument b. The 24 high-order bits of b are ignored.

Parameters
b:int

the byte.

Annotations
@Override
Exceptions
IOException:
if an I/O error occurs. In particular, an IOException may be thrown if the output stream has been closed.
writeback to summary
public final void write(byte[] bytes) throws IOException

Overrides java.io.OutputStream.write.

Doc from java.io.OutputStream.write.

Writes b.length bytes from the specified byte array to this output stream. The general contract for write(b) is that it should have exactly the same effect as the call write(b, 0, b.length).

Parameters
bytes:byte[]

the data.

Annotations
@Override
Exceptions
IOException:
if an I/O error occurs.
writeback to summary
public final void write(byte[] bytes, int off, int len) throws IOException

Overrides java.io.OutputStream.write.

Doc from java.io.OutputStream.write.

Writes len bytes from the specified byte array starting at offset off to this output stream. The general contract for write(b, off, len) is that some of the bytes in the array b are written to the output stream in order; element b[off] is the first byte written and b[off+len-1] is the last byte written by this operation.

If b is null, a NullPointerException is thrown.

If off is negative, or len is negative, or off+len is greater than the length of the array b, then an IndexOutOfBoundsException is thrown.

Parameters
bytes:byte[]

the data.

off:int

the start offset in the data.

len:int

the number of bytes to write.

Annotations
@Override
Exceptions
IOException:
if an I/O error occurs. In particular, an IOException is thrown if the output stream is closed.
write_helper_continueback to summary
private final void write_helper_continue(int b) throws IOException
write_helper_start_sequenceback to summary
private final void write_helper_start_sequence(int b) throws IOException
write_helper_write_charback to summary
private final void write_helper_write_char(int b) throws IOException