Top Description Inners Fields Constructors Methods
com.google.protobuf

pack-priv final Class Utf8

extends Object
Class Inheritance
Static Imports
com.google.protobuf.UnsafeUtil.addressOffset, .UnsafeUtil.hasUnsafeArrayOperations, .UnsafeUtil.hasUnsafeByteBufferOperations, java.lang.Character.MAX_SURROGATE, .Character.MIN_HIGH_SURROGATE, .Character.MIN_LOW_SURROGATE, .Character.MIN_SUPPLEMENTARY_CODE_POINT, .Character.MIN_SURROGATE, .Character.isSurrogatePair, .Character.toCodePoint
A set of low-level, high-performance static utility methods related to the UTF-8 character encoding. This class has no dependencies outside of the core JDK libraries.

There are several variants of UTF-8. The one implemented by this class is the restricted definition of UTF-8 introduced in Unicode 3.1, which mandates the rejection of "overlong" byte sequences as well as rejection of 3-byte surrogate codepoint byte sequences. Note that the UTF-8 decoder included in Oracle's JDK has been modified to also reject "overlong" byte sequences, but (as of 2011) still accepts 3-byte surrogate codepoint byte sequences.

The byte sequences considered valid by this class are exactly those that can be roundtrip converted to Strings and back to bytes using the UTF-8 charset, without loss: 

Arrays.equals(bytes, new String(bytes, Internal.UTF_8).getBytes(Internal.UTF_8))

See the Unicode Standard,
Table 3-6. UTF-8 Bit Distribution,
Table 3-7. Well Formed UTF-8 Byte Sequences.

This class supports decoding of partial byte sequences, so that the bytes in a complete UTF-8 byte sequences can be stored in multiple segments. Methods typically return MALFORMED if the partial byte sequence is definitely not well-formed, COMPLETE if it is well-formed in the absence of additional input, or if the byte sequence apparently terminated in the middle of a character, an opaque integer "state" value containing enough information to decode the character when passed to a subsequent invocation of a partial decoding method.

Author
[email protected] (Martin Buchholz)

Nested and Inner Type Summary

Modifier and TypeClass and Description
private static class
Utf8.DecodeUtil

Utility methods for decoding bytes into String.

pack-priv abstract static class
Utf8.Processor

A processor of UTF-8 strings, providing methods for checking validity and encoding.

pack-priv static class
Utf8.SafeProcessor

Processor implementation that does not use any sun.misc.Unsafe methods.

pack-priv static class
Utf8.UnpairedSurrogateException

pack-priv static class
Utf8.UnsafeProcessor

Processor that uses sun.misc.Unsafe where possible to improve performance.

Field Summary

Modifier and TypeField and Description
private static final long
ASCII_MASK_LONG

A mask used when performing unsafe reads to determine if a long value contains any non-ASCII characters (i.e. any byte >= 0x80).

public static final int
COMPLETE

State value indicating that the byte sequence is well-formed and complete (no further bytes are needed to complete a character).

public static final int
MALFORMED

State value indicating that the byte sequence is definitely not well-formed.

pack-priv static final int
MAX_BYTES_PER_CHAR

Maximum number of bytes per Java UTF-16 char in UTF-8.

private static final Utf8.Processor
processor

UTF-8 is a runtime hot spot so we attempt to provide heavily optimized implementations depending on what is available on the platform.

private static final int
UNSAFE_COUNT_ASCII_THRESHOLD

Used by Unsafe UTF-8 string validation logic to determine the minimum string length above which to employ an optimized algorithm for counting ASCII characters.

Constructor Summary

AccessConstructor and Description
private
Utf8()

Method Summary

Modifier and TypeMethod and Description
pack-priv static String
decodeUtf8(ByteBuffer buffer, int index, int size)

Decodes the given UTF-8 portion of the ByteBuffer into a String.

pack-priv static String
decodeUtf8(byte[] bytes, int index, int size)

Decodes the given UTF-8 encoded byte array slice into a String.

pack-priv static int
encode(CharSequence in, byte[] out, int offset, int length)

pack-priv static int
encodedLength(CharSequence sequence)

Returns the number of bytes in the UTF-8-encoded form of sequence.

private static int
encodedLengthGeneral(CharSequence sequence, int start)

pack-priv static void
encodeUtf8(CharSequence
the source string to be encoded
in, ByteBuffer
the target buffer to receive the encoded string.
out)

Encodes the given characters to the target ByteBuffer using UTF-8 encoding.

private static int

Returns:

the number of ASCII characters found. The stopping position will be at or before the first non-ASCII byte.
estimateConsecutiveAscii(ByteBuffer
the buffer to be scanned for ASCII chars
buffer, int
the starting index of the scan
index, int
the limit within buffer for the scan
limit)

Counts (approximately) the number of consecutive ASCII characters in the given buffer.

private static int
incompleteStateFor(int byte1)

private static int
incompleteStateFor(int byte1, int byte2)

private static int
incompleteStateFor(int byte1, int byte2, int byte3)

private static int
incompleteStateFor(byte[] bytes, int index, int limit)

private static int
incompleteStateFor(final ByteBuffer buffer, final int byte1, final int index, final int remaining)

public static boolean
isValidUtf8(byte[] bytes)

Returns true if the given byte array is a well-formed UTF-8 byte sequence.

public static boolean
isValidUtf8(byte[] bytes, int index, int limit)

Returns true if the given byte array slice is a well-formed UTF-8 byte sequence.

pack-priv static boolean
isValidUtf8(ByteBuffer
the buffer to check.
buffer)

Determines if the given ByteBuffer is a valid UTF-8 string.

public static int

Returns:

MALFORMED if the partial byte sequence is definitely not well-formed, COMPLETE if it is well-formed (no additional input needed), or if the byte sequence is "incomplete", i.e. apparently terminated in the middle of a character, an opaque integer "state" value containing enough information to decode the character when passed to a subsequent invocation of a partial decoding method.
partialIsValidUtf8(int
either Utf8#COMPLETE (if this is the initial decoding operation) or the value returned from a call to a partial decoding method for the previous bytes
state, byte[] bytes, int index, int limit)

Tells whether the given byte array slice is a well-formed, malformed, or incomplete UTF-8 byte sequence.

pack-priv static int
partialIsValidUtf8(int state, ByteBuffer
the buffer to check.
buffer, int index, int limit)

Determines if the given ByteBuffer is a partially valid UTF-8 string.

Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait