Top Description Fields Constructors Methods
groovy.util

public Class CharsetToolkit

extends Object
Class Inheritance
Imports
java.io.BufferedReader, .File, .FileInputStream, .FileNotFoundException, .IOException, .InputStream, .InputStreamReader, .LineNumberReader, java.nio.charset.Charset, .StandardCharsets, java.util.Collection
Utility class to guess the encoding of a given text file.

Unicode files encoded in UTF-16 (low or big endian) or UTF-8 files with a Byte Order Marker are correctly discovered. For UTF-8 files with no BOM, if the buffer is wide enough, the charset should also be discovered.

A byte buffer of 4KB is used to be able to guess the encoding.

Usage: 

CharsetToolkit toolkit = new CharsetToolkit(file);

// guess the encoding
Charset guessedCharset = toolkit.getCharset();

// create a reader with the correct charset
BufferedReader reader = toolkit.getReader();

// read the file content
String line;
while ((line = br.readLine())!= null)
{
    System.out.println(line);
}

Field Summary

Modifier and TypeField and Description
private final byte[]
buffer

private Charset
charset

private Charset
defaultCharset

private static final byte[]
EMPTY_BYTE_ARRAY

private static final Charset[]
EMPTY_CHARSET_ARRAY

private boolean
enforce8Bit

private final File
file

Constructor Summary

AccessConstructor and Description
public
CharsetToolkit(File
of which we want to know the encoding.
file)

Constructor of the CharsetToolkit utility class.

Method Summary

Modifier and TypeMethod and Description
public static Charset[]

Returns:

an array of Charsets.
getAvailableCharsets()

Retrieves all the available Charsets on the platform, among which the default charset.

public Charset
getCharset()

public Charset
getDefaultCharset()

Retrieves the default Charset

public static Charset

Returns:

the default Charset.
getDefaultSystemCharset()

Retrieve the default charset of the system.

public boolean

Returns:

a boolean representing the flag of use of US-ASCII.
getEnforce8Bit()

Gets the enforce8Bit flag, in case we do not want to ever get a US-ASCII encoding.

public BufferedReader

Returns:

a BufferedReader
getReader()

Gets a BufferedReader (indeed a LineNumberReader) from the File specified in the constructor of CharsetToolkit using the charset discovered or the default charset if an 8-bit Charset is encountered.

private Charset

Returns:

the Charset recognized.
guessEncoding()

Guess the encoding of the provided buffer.

public boolean

Returns:

true if the buffer has a BOM for UTF-16 Big Endian.
hasUTF16BEBom()

Has a Byte Order Marker for UTF-16 Big Endian (utf-16 and ucs-2).

public boolean

Returns:

true if the buffer has a BOM for UTF-16 Low Endian.
hasUTF16LEBom()

Has a Byte Order Marker for UTF-16 Low Endian (ucs-2le, ucs-4le, and ucs-16le).

public boolean

Returns:

true if the buffer has a BOM for UTF8.
hasUTF8Bom()

Has a Byte Order Marker for UTF-8 (Used by Microsoft's Notepad and other editors).

private static boolean

Returns:

true if it's a continuation char.
isContinuationChar(byte
a byte.
b)

If the byte has the form 10xxxxx, then it's a continuation byte of a multiple byte character;

private static boolean

Returns:

true if it's the first byte of a five-bytes sequence.
isFiveBytesSequence(byte
a byte.
b)

If the byte has the form 11110xx, then it's the first byte of a five-bytes sequence character.

private static boolean

Returns:

true if it's the first byte of a four-bytes sequence.
isFourBytesSequence(byte
a byte.
b)

If the byte has the form 11110xx, then it's the first byte of a four-bytes sequence character.

private static boolean

Returns:

true if it's the first byte of a six-bytes sequence.
isSixBytesSequence(byte
a byte.
b)

If the byte has the form 1110xxx, then it's the first byte of a six-bytes sequence character.

private static boolean

Returns:

true if it's the first byte of a three-bytes sequence.
isThreeBytesSequence(byte
a byte.
b)

If the byte has the form 1110xxx, then it's the first byte of a three-bytes sequence character.

private static boolean

Returns:

true if it's the first byte of a two-bytes sequence.
isTwoBytesSequence(byte
a byte.
b)

If the byte has the form 110xxxx, then it's the first byte of a two-bytes sequence character.

public void
setDefaultCharset(Charset
the default Charset to be returned if an 8-bit Charset is encountered.
defaultCharset)

Defines the default Charset used in case the buffer represents an 8-bit Charset.

public void
setEnforce8Bit(boolean
a boolean specifying the use or not of US-ASCII.
enforce)

If US-ASCII is recognized, enforce to return the default encoding, rather than US-ASCII.

Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Field Detail

bufferback to summary
private final byte[] buffer
charsetback to summary
private Charset charset
defaultCharsetback to summary
private Charset defaultCharset
EMPTY_BYTE_ARRAYback to summary
private static final byte[] EMPTY_BYTE_ARRAY
EMPTY_CHARSET_ARRAYback to summary
private static final Charset[] EMPTY_CHARSET_ARRAY
enforce8Bitback to summary
private boolean enforce8Bit
fileback to summary
private final File file

Constructor Detail

CharsetToolkitback to summary
public CharsetToolkit(File file) throws IOException

Constructor of the CharsetToolkit utility class.

Parameters
file:File

of which we want to know the encoding.

Method Detail

getAvailableCharsetsback to summary
public static Charset[] getAvailableCharsets()

Retrieves all the available Charsets on the platform, among which the default charset.

Returns:Charset[]

an array of Charsets.

getCharsetback to summary
public Charset getCharset()
getDefaultCharsetback to summary
public Charset getDefaultCharset()

Retrieves the default Charset

getDefaultSystemCharsetback to summary
public static Charset getDefaultSystemCharset()

Retrieve the default charset of the system.

Returns:Charset

the default Charset.

getEnforce8Bitback to summary
public boolean getEnforce8Bit()

Gets the enforce8Bit flag, in case we do not want to ever get a US-ASCII encoding.

Returns:boolean

a boolean representing the flag of use of US-ASCII.

getReaderback to summary
public BufferedReader getReader() throws FileNotFoundException

Gets a BufferedReader (indeed a LineNumberReader) from the File specified in the constructor of CharsetToolkit using the charset discovered or the default charset if an 8-bit Charset is encountered.

Returns:BufferedReader

a BufferedReader

Exceptions
FileNotFoundException:
if the file is not found.
guessEncodingback to summary
private Charset guessEncoding()

Guess the encoding of the provided buffer. If Byte Order Markers are encountered at the beginning of the buffer, we immediately return the charset implied by this BOM. Otherwise, the file would not be a human readable text file.

If there is no BOM, this method tries to discern whether the file is UTF-8 or not. If it is not UTF-8, we assume the encoding is the default system encoding (of course, it might be any 8-bit charset, but usually, an 8-bit charset is the default one).

It is possible to discern UTF-8 thanks to the pattern of characters with a multi-byte sequence. 

UCS-4 range (hex.)        UTF-8 octet sequence (binary)
0000 0000-0000 007F       0xxxxxxx
0000 0080-0000 07FF       110xxxxx 10xxxxxx
0000 0800-0000 FFFF       1110xxxx 10xxxxxx 10xxxxxx
0001 0000-001F FFFF       11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
0020 0000-03FF FFFF       111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
0400 0000-7FFF FFFF       1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
With UTF-8, 0xFE and 0xFF never appear.
Returns:Charset

the Charset recognized.

hasUTF16BEBomback to summary
public boolean hasUTF16BEBom()

Has a Byte Order Marker for UTF-16 Big Endian (utf-16 and ucs-2).

Returns:boolean

true if the buffer has a BOM for UTF-16 Big Endian.

hasUTF16LEBomback to summary
public boolean hasUTF16LEBom()

Has a Byte Order Marker for UTF-16 Low Endian (ucs-2le, ucs-4le, and ucs-16le).

Returns:boolean

true if the buffer has a BOM for UTF-16 Low Endian.

hasUTF8Bomback to summary
public boolean hasUTF8Bom()

Has a Byte Order Marker for UTF-8 (Used by Microsoft's Notepad and other editors).

Returns:boolean

true if the buffer has a BOM for UTF8.

isContinuationCharback to summary
private static boolean isContinuationChar(byte b)

If the byte has the form 10xxxxx, then it's a continuation byte of a multiple byte character;

Parameters
b:byte

a byte.

Returns:boolean

true if it's a continuation char.

isFiveBytesSequenceback to summary
private static boolean isFiveBytesSequence(byte b)

If the byte has the form 11110xx, then it's the first byte of a five-bytes sequence character.

Parameters
b:byte

a byte.

Returns:boolean

true if it's the first byte of a five-bytes sequence.

isFourBytesSequenceback to summary
private static boolean isFourBytesSequence(byte b)

If the byte has the form 11110xx, then it's the first byte of a four-bytes sequence character.

Parameters
b:byte

a byte.

Returns:boolean

true if it's the first byte of a four-bytes sequence.

isSixBytesSequenceback to summary
private static boolean isSixBytesSequence(byte b)

If the byte has the form 1110xxx, then it's the first byte of a six-bytes sequence character.

Parameters
b:byte

a byte.

Returns:boolean

true if it's the first byte of a six-bytes sequence.

isThreeBytesSequenceback to summary
private static boolean isThreeBytesSequence(byte b)

If the byte has the form 1110xxx, then it's the first byte of a three-bytes sequence character.

Parameters
b:byte

a byte.

Returns:boolean

true if it's the first byte of a three-bytes sequence.

isTwoBytesSequenceback to summary
private static boolean isTwoBytesSequence(byte b)

If the byte has the form 110xxxx, then it's the first byte of a two-bytes sequence character.

Parameters
b:byte

a byte.

Returns:boolean

true if it's the first byte of a two-bytes sequence.

setDefaultCharsetback to summary
public void setDefaultCharset(Charset defaultCharset)

Defines the default Charset used in case the buffer represents an 8-bit Charset.

Parameters
defaultCharset:Charset

the default Charset to be returned if an 8-bit Charset is encountered.

setEnforce8Bitback to summary
public void setEnforce8Bit(boolean enforce)

If US-ASCII is recognized, enforce to return the default encoding, rather than US-ASCII. It might be a file without any special character in the range 128-255, but that may be or become a file encoded with the default charset rather than US-ASCII.

Parameters
enforce:boolean

a boolean specifying the use or not of US-ASCII.