public final Class SmileConstants

In addition we can use a marker to allow simple framing; splitting of physical data (like file) into distinct logical sections like JSON documents. 0xFF makes sense here since it is also used as end marker for Web Sockets.

Indicator bit that indicates whether encoded content may contain raw (unquoted) binary values. If no header available, can be assumed to be 0 (false). If header exists, and bit value is 1, parser can not assume that specific byte values always have default meaning (specifically, content end marker 0xFF and header signature can be contained in binary values)

Note that this bit being true does not automatically mean that such raw binary content indeed exists; just that it may exist. This because header is written before any binary data may be written.

Indicator bit that indicates whether encoded content may have Shared names (back references to recently encoded field names). If no header available, must be processed as if this was set to true. If (and only if) header exists, and value is 0, can parser omit storing of seen names, as it is guaranteed that no back references exist.

Indicator bit that indicates whether encoded content may have shared String values (back references to recently encoded 'short' String values, where short is defined as 64 bytes or less). If no header available, can be assumed to be 0 (false). If header exists, and bit value is 1, parsers has to store up to 1024 most recently seen distinct short String values.

First byte of data header (0x3A)

Second byte of data header (0x29)

Third byte of data header

Fourth byte of data header; contains version nibble, may have flags

Current version consists of four zero bits (nibble)

We need a byte marker to denote end of variable-length Strings. Although null byte is commonly used, let's try to avoid using it since it can't be embedded in Web Sockets content (similarly, 0xFF can't). There are multiple candidates for bytes UTF-8 can not have; 0xFC is chosen to allow reasonable ordering (highest values meaning most significant framing function; 0xFF being end-of-content and so on)

Longest back reference we use for field names is 10 bits; no point in keeping much more around

Also: whereas we can refer to names of any length, we will only consider text values that are considered "tiny" or "short" (ones encoded with length prefix); this value thereby has to be maximum length of Strings that can be encoded as such.

Longest back reference we use for short shared String values is 10 bits, so up to (1 << 10) values to keep track of.

Regardless of ASCII/non-ASCII aspect, maximum byte length for any short name is then 64 bytes.

Maximum byte length for short ASCII names is 64.

Maximum byte length for short non-ASCII names is slightly less due to having to reserve bytes 0xF8 and above (but we get one more as values 0 and 1 are not valid)

Encoding has special "short" forms for value Strings that can be represented by 64 bytes of UTF-8 or less.

And to make encoding logic tight and simple, we can always require that output buffer has this amount of space available before encoding possibly short String (3 bytes since longest UTF-8 encoded Java char is 3 bytes). Two extra bytes need to be reserved as well; first for token indicator, and second for terminating null byte (in case it's not a short String after all)

Additionally we can combine UTF-8 decoding info into similar data table. Values indicate "byte length - 1"; meaning -1 is used for invalid bytes, 0 for single-byte codes, 1 for 2-byte codes and 2 for 3-byte codes.

Let's use same code for empty key as for empty String value

Type (for misc, other) used for "safe" (encoded by only using 7 LSB, giving 8/7 expansion ratio). This is usually done to ensure that certain bytes are never included in encoded data (like 0xFF) Note: 2 LSB are reserved for future use; must be zeroes for now

Raw binary data marker is specifically chosen as separate from other types, since it can have significant impact on framing (or rather fast scanning based on structure and framing markers).

Numeric subtype (2 LSB) for indicating 32-bit IEEE single precision floating point number.

Numeric subtype (2 LSB) indicating 64-bit IEEE double precision floating point number.

Numeric subtype (2 LSB) for indicating java.math.BigDecimal type.

Numeric subtype (2 LSB) indicating 32-bit integer (int)

Numeric subtype (2 LSB) indicating 32-bit integer (long)

Numeric subtype (2 LSB) for indicating java.math.BigInteger type.

Type (for misc, other) used for variable length UTF-8 encoded text, when it is known to only contain ASCII chars.

Type (for misc, other) used for variable length UTF-8 encoded text, when it is NOT known to only contain ASCII chars (which means it MAY have multi-byte characters) Note: 2 LSB are reserved for future use; must be zeroes for now