normalize transforms Unicode text into an equivalent composed or
decomposed form, allowing for easier sorting and searching of text.
normalize supports the standard normalization forms described in
Unicode Standard Annex #15 — Unicode Normalization Forms.
Characters with accents or other adornments can be encoded in
several different ways in Unicode. For example, take the character A-acute.
In Unicode, this can be encoded as a single character (the
"composed" form):
00C1 LATIN CAPITAL LETTER A WITH ACUTE
or as two separate characters (the "decomposed" form):
0041 LATIN CAPITAL LETTER A
0301 COMBINING ACUTE ACCENT
To a user of your program, however, both of these sequences should be
treated as the same "user-level" character "A with acute accent". When you
are searching or comparing text, you must ensure that these two sequences are
treated equivalently. In addition, you must handle characters with more than
one accent. Sometimes the order of a character's combining accents is
significant, while in other cases accent sequences in different orders are
really equivalent.
Similarly, the string "ffi" can be encoded as three separate letters:
0066 LATIN SMALL LETTER F
0066 LATIN SMALL LETTER F
0069 LATIN SMALL LETTER I
or as the single character
FB03 LATIN SMALL LIGATURE FFI
The ffi ligature is not a distinct semantic character, and strictly speaking
it shouldn't be in Unicode at all, but it was included for compatibility
with existing character sets that already provided it. The Unicode standard
identifies such characters by giving them "compatibility" decompositions
into the corresponding semantic characters. When sorting and searching, you
will often want to use these mappings.
normalize helps solve these problems by transforming text into
the canonical composed and decomposed forms as shown in the first example
above. In addition, you can have it perform compatibility decompositions so
that you can treat compatibility characters the same as their equivalents.
Finally, normalize rearranges accents into the proper canonical
order, so that you do not have to worry about accent rearrangement on your
own.
Form FCD, "Fast C or D", is also designed for collation.
It allows to work on strings that are not necessarily normalized
with an algorithm (like in collation) that works under "canonical closure",
i.e., it treats precomposed characters and their decomposed equivalents the
same.
It is not a normalization form because it does not provide for uniqueness of
representation. Multiple strings may be canonically equivalent (their NFDs
are identical) and may all conform to FCD without being identical themselves.
The form is defined such that the "raw decomposition", the recursive
canonical decomposition of each character, results in a string that is
canonically ordered. This means that precomposed characters are allowed for
as long as their decompositions do not need canonical reordering.
Its advantage for a process like collation is that all NFD and most NFC texts
- and many unnormalized texts - already conform to FCD and do not need to be
normalized (NFD) for such a process. The FCD quick check will return YES for
most strings in practice.
normalize(FCD) may be implemented with NFD.
For more details on FCD see Unicode Technical Note #5 (Canonical Equivalence in Applications):
http://www.unicode.org/notes/tn5/#FCD
ICU collation performs either NFD or FCD normalization automatically if
normalization is turned on for the collator object. Beyond collation and
string search, normalized strings may be useful for string equivalence
comparisons, transliteration/transcription, unique representations, etc.
The W3C generally recommends to exchange texts in NFC.
Note also that most legacy character encodings use only precomposed forms and
often do not encode any combining marks by themselves. For conversion to such
character encodings the Unicode text needs to be normalized to NFC.
For more usage examples, see the Unicode Standard Annex.Note
The Normalizer class also provides API for iterative normalization. While the setIndex() and getIndex() refer to indices in the underlying Unicode input text, the next() and previous() methods iterate through characters in the normalized output. This means that there is not necessarily a one-to-one correspondence between characters returned by next() and previous() and the indices passed to and returned from setIndex() and getIndex(). It is for this reason that Normalizer does not implement the CharacterIterator interface.
| Modifier and Type | Class and Description |
|---|---|
| public abstract static class | NormalizerBase.
Constants for normalization modes. |
| private static class | |
| private static class | |
| private static class | |
| private static class | |
| private static class | |
| private static class | |
| private static class | |
| private static class | |
| private static class | |
| private static class | |
| private static class | |
| private static class | |
| private static class | |
| private static class | |
| private static class |
| Modifier and Type | Field and Description |
|---|---|
| private StringBuilder | |
| private int | |
| private int | |
| public static final int | DONE
Constant indicating that the end of the iteration has been reached. |
| private NormalizerBase. | |
| private int | |
| public static final NormalizerBase. | NFC
Canonical decomposition followed by canonical composition. |
| public static final NormalizerBase. | NFD
Canonical decomposition. |
| public static final NormalizerBase. | |
| public static final NormalizerBase. | NFKD
Compatibility decomposition. |
| public static final NormalizerBase. | NONE
No decomposition/composition. |
| private Normalizer2 | |
| private int | |
| private UCharacterIterator | |
| public static final int | UNICODE_3_2
Options bit set value to select Unicode 3.2 normalization (except NormalizationCorrections). |
| public static final int | |
| public static final int |
| Access | Constructor and Description |
|---|---|
| public | NormalizerBase(String
The string to be normalized. The normalization
will start at the beginning of the string. str, NormalizerBase.The normalization mode. mode,Any optional features to be enabled.
Currently the only available option is opt)UNICODE_3_2.
If you want the default behavior corresponding to one of the
standard Unicode Normalization Forms, use 0 for this argument.Creates a new |
| public | |
| public | NormalizerBase(CharacterIterator
The input text to be normalized. The normalization
will start at the beginning of the string. iter, NormalizerBase.The normalization mode. mode,Any optional features to be enabled.
Currently the only available option is opt)UNICODE_3_2.
If you want the default behavior corresponding to one of the
standard Unicode Normalization Forms, use 0 for this argument.Creates a new |
| public |
| Modifier and Type | Method and Description |
|---|---|
| private void | |
| public NormalizerBase | |
| public int | |
| public int | |
| public int | Returns: The codepoint as an int
Deprecated
ICU 2.2. Use startIndex() instead.
Retrieve the index of the start of the input text. |
| public int | Returns: The codepoint as an int
Deprecated
ICU 2.2. Use endIndex() instead.
Retrieve the index of the end of the input text. |
| public int | Returns: The current iteration positionRetrieve the current iteration position in the input text that is being normalized. |
| public NormalizerBase. | |
| public static boolean | isNormalized(String
the input string to be checked to see if it is
normalized str, NormalizerBase.the normalization mode mode,Options for use with exclusion set and tailored Normalization
The only option that is currently recognized is UNICODE_3_2 options)Test if a string is in a given normalization form. |
| public static boolean | |
| public static boolean | |
| public int | Returns: The codepoint as an intReturn the next character in the normalized text and advance the iteration position by one. |
| private boolean | |
| public static String | Returns: String the normalized stringthe input string to be normalized. str, NormalizerBase.the normalization mode mode,the optional features to be enabled. options)Normalizes a |
| public static String | |
| public static String | |
| public int | Returns: The codepoint as an intReturn the previous character in the normalized text and decrement the iteration position by one. |
| private boolean | |
| public void | |
| public int | Returns: the first normalized character that is the result of iterating forward starting at the given index.the desired index in the input text. index)Set the iteration position in the input text that is being normalized and return the first normalized character at that position. |
| public void | setIndexOnly(int
the desired index in the input text. index)Set the iteration position in the input text that is being normalized, without any immediate normalization. |
| public void | setMode(NormalizerBase.
the new mode for this newModeNormalizerBase.
The supported modes are:
Set the normalization mode for this object. |
| public void | |
| public void | setText(CharacterIterator
The new string to be normalized. newText)Set the input text over which this |
| private static NormalizerBase. |
| buffer | back to summary |
|---|---|
| private StringBuilder buffer | |
| bufferPos | back to summary |
|---|---|
| private int bufferPos | |
| currentIndex | back to summary |
|---|---|
| private int currentIndex | |
| DONE | back to summary |
|---|---|
| public static final int DONE Constant indicating that the end of the iteration has been reached.
This is guaranteed to have the same value as
| |
| mode | back to summary |
|---|---|
| private NormalizerBase. | |
| nextIndex | back to summary |
|---|---|
| private int nextIndex | |
| NFC | back to summary |
|---|---|
| public static final NormalizerBase. Canonical decomposition followed by canonical composition.
| |
| NFD | back to summary |
|---|---|
| public static final NormalizerBase. Canonical decomposition.
| |
| NFKC | back to summary |
|---|---|
| public static final NormalizerBase. | |
| NFKD | back to summary |
|---|---|
| public static final NormalizerBase. Compatibility decomposition.
| |
| NONE | back to summary |
|---|---|
| public static final NormalizerBase. No decomposition/composition.
| |
| norm2 | back to summary |
|---|---|
| private Normalizer2 norm2 | |
| options | back to summary |
|---|---|
| private int options | |
| text | back to summary |
|---|---|
| private UCharacterIterator text | |
| UNICODE_3_2 | back to summary |
|---|---|
| public static final int UNICODE_3_2 Options bit set value to select Unicode 3.2 normalization (except NormalizationCorrections). At most one Unicode version can be selected at a time.
| |
| UNICODE_3_2_0_ORIGINAL | back to summary |
|---|---|
| public static final int UNICODE_3_2_0_ORIGINAL | |
| UNICODE_LATEST | back to summary |
|---|---|
| public static final int UNICODE_LATEST | |
| NormalizerBase | back to summary |
|---|---|
| public NormalizerBase(String str, NormalizerBase. Creates a new
The
| |
| NormalizerBase | back to summary |
|---|---|
| public NormalizerBase(String str, NormalizerBase. | |
| NormalizerBase | back to summary |
|---|---|
| public NormalizerBase(CharacterIterator iter, NormalizerBase. Creates a new
| |
| NormalizerBase | back to summary |
|---|---|
| public NormalizerBase(CharacterIterator iter, NormalizerBase. | |
| clearBuffer | back to summary |
|---|---|
| private void clearBuffer() | |
| clone | back to summary |
|---|---|
| public NormalizerBase clone() Overrides java. Clones this
| |
| current | back to summary |
|---|---|
| public int current() Return the current character in the normalized text.
| |
| endIndex | back to summary |
|---|---|
| public int endIndex() Retrieve the index of the end of the input text. This is the end index
of the
| |
| getBeginIndex | back to summary |
|---|---|
| public int getBeginIndex()
Deprecated ICU 2.2. Use startIndex() instead. Retrieve the index of the start of the input text. This is the begin
index of the
| |
| getEndIndex | back to summary |
|---|---|
| public int getEndIndex()
Deprecated ICU 2.2. Use endIndex() instead. Retrieve the index of the end of the input text. This is the end index
of the
| |
| getIndex | back to summary |
|---|---|
| public int getIndex() Retrieve the current iteration position in the input text that is being normalized. This method is useful in applications such as searching, where you need to be able to determine the position in the input text that corresponds to a given normalized output character. Note This method sets the position in the input, while
| |
| getMode | back to summary |
|---|---|
| public NormalizerBase. Return the basic operation performed by this | |
| isNormalized | back to summary |
|---|---|
| public static boolean isNormalized(String str, NormalizerBase. Test if a string is in a given normalization form. This is semantically equivalent to source.equals(normalize(source, mode)). Unlike quickCheck(), this function returns a definitive result, never a "maybe". For NFD, NFKD, and FCD, both functions work exactly the same. For NFC and NFKC where quickCheck may return "maybe", this function will perform further tests to arrive at a true/false result.
| |
| isNormalized | back to summary |
|---|---|
| public static boolean isNormalized(String str, Normalizer. | |
| isNormalized | back to summary |
|---|---|
| public static boolean isNormalized(String str, Normalizer. | |
| next | back to summary |
|---|---|
| public int next() Return the next character in the normalized text and advance
the iteration position by one. If the end
of the text has already been reached,
| |
| nextNormalize | back to summary |
|---|---|
| private boolean nextNormalize() | |
| normalize | back to summary |
|---|---|
| public static String normalize(String str, NormalizerBase. Normalizes a
The
| |
| normalize | back to summary |
|---|---|
| public static String normalize(String str, Normalizer. | |
| normalize | back to summary |
|---|---|
| public static String normalize(String str, Normalizer. | |
| previous | back to summary |
|---|---|
| public int previous() Return the previous character in the normalized text and decrement
the iteration position by one. If the beginning
of the text has already been reached,
| |
| previousNormalize | back to summary |
|---|---|
| private boolean previousNormalize() | |
| reset | back to summary |
|---|---|
| public void reset() Reset the index to the beginning of the text. This is equivalent to setIndexOnly(startIndex)).
| |
| setIndex | back to summary |
|---|---|
| public int setIndex(int index) Set the iteration position in the input text that is being normalized and return the first normalized character at that position. Note This method sets the position in the input text,
while
| |
| setIndexOnly | back to summary |
|---|---|
| public void setIndexOnly(int index) Set the iteration position in the input text that is being normalized, without any immediate normalization. After setIndexOnly(), getIndex() will return the same index that is specified here.
| |
| setMode | back to summary |
|---|---|
| public void setMode(NormalizerBase. Set the normalization mode for this object. Note If the normalization mode is changed while iterating
over a string, calls to
| |
| setText | back to summary |
|---|---|
| public void setText(String newText) Set the input text over which this | |
| setText | back to summary |
|---|---|
| public void setText(CharacterIterator newText) Set the input text over which this
| |
| toMode | back to summary |
|---|---|
| private static NormalizerBase. | |
The Mode class is not intended for public subclassing. Only the Mode constants provided by the Normalizer class should be used, and any fields or methods should not be called or overridden by users.
| Access | Constructor and Description |
|---|---|
| protected |
| Modifier and Type | Method and Description |
|---|---|
| protected abstract Normalizer2 |
| Mode | back to summary |
|---|---|
| protected Mode()
Deprecated This API is ICU internal only. Sole constructor
| |
| getNormalizer2 | back to summary |
|---|---|
| protected abstract Normalizer2 getNormalizer2(int options)
Deprecated This API is ICU internal only.
| |
| Modifier and Type | Field and Description |
|---|---|
| private final Normalizer2 |
| Access | Constructor and Description |
|---|---|
| private |
| normalizer2 | back to summary |
|---|---|
| private final Normalizer2 normalizer2 | |
| ModeImpl | back to summary |
|---|---|
| private ModeImpl(Normalizer2 n2) | |
| Modifier and Type | Field and Description |
|---|---|
| private static final NormalizerBase. |
| Access | Constructor and Description |
|---|---|
| private |
| INSTANCE | back to summary |
|---|---|
| private static final NormalizerBase. | |
| NFC32ModeImpl | back to summary |
|---|---|
| private NFC32ModeImpl() | |
| Access | Constructor and Description |
|---|---|
| private |
| Modifier and Type | Method and Description |
|---|---|
| protected Normalizer2 | getNormalizer2(int options)
Implements abstract jdk.
|
| NFCMode | back to summary |
|---|---|
| private NFCMode() | |
| getNormalizer2 | back to summary |
|---|---|
| protected Normalizer2 getNormalizer2(int options) Implements abstract jdk. | |
| Modifier and Type | Field and Description |
|---|---|
| private static final NormalizerBase. |
| Access | Constructor and Description |
|---|---|
| private |
| INSTANCE | back to summary |
|---|---|
| private static final NormalizerBase. | |
| NFCModeImpl | back to summary |
|---|---|
| private NFCModeImpl() | |
| Modifier and Type | Field and Description |
|---|---|
| private static final NormalizerBase. |
| Access | Constructor and Description |
|---|---|
| private |
| INSTANCE | back to summary |
|---|---|
| private static final NormalizerBase. | |
| NFD32ModeImpl | back to summary |
|---|---|
| private NFD32ModeImpl() | |
| Access | Constructor and Description |
|---|---|
| private |
| Modifier and Type | Method and Description |
|---|---|
| protected Normalizer2 | getNormalizer2(int options)
Implements abstract jdk.
|
| NFDMode | back to summary |
|---|---|
| private NFDMode() | |
| getNormalizer2 | back to summary |
|---|---|
| protected Normalizer2 getNormalizer2(int options) Implements abstract jdk. | |
| Modifier and Type | Field and Description |
|---|---|
| private static final NormalizerBase. |
| Access | Constructor and Description |
|---|---|
| private |
| INSTANCE | back to summary |
|---|---|
| private static final NormalizerBase. | |
| NFDModeImpl | back to summary |
|---|---|
| private NFDModeImpl() | |
| Modifier and Type | Field and Description |
|---|---|
| private static final NormalizerBase. |
| Access | Constructor and Description |
|---|---|
| private |
| INSTANCE | back to summary |
|---|---|
| private static final NormalizerBase. | |
| NFKC32ModeImpl | back to summary |
|---|---|
| private NFKC32ModeImpl() | |
| Access | Constructor and Description |
|---|---|
| private |
| Modifier and Type | Method and Description |
|---|---|
| protected Normalizer2 | getNormalizer2(int options)
Implements abstract jdk.
|
| NFKCMode | back to summary |
|---|---|
| private NFKCMode() | |
| getNormalizer2 | back to summary |
|---|---|
| protected Normalizer2 getNormalizer2(int options) Implements abstract jdk. | |
| Modifier and Type | Field and Description |
|---|---|
| private static final NormalizerBase. |
| Access | Constructor and Description |
|---|---|
| private |
| INSTANCE | back to summary |
|---|---|
| private static final NormalizerBase. | |
| NFKCModeImpl | back to summary |
|---|---|
| private NFKCModeImpl() | |
| Modifier and Type | Field and Description |
|---|---|
| private static final NormalizerBase. |
| Access | Constructor and Description |
|---|---|
| private |
| INSTANCE | back to summary |
|---|---|
| private static final NormalizerBase. | |
| NFKD32ModeImpl | back to summary |
|---|---|
| private NFKD32ModeImpl() | |
| Access | Constructor and Description |
|---|---|
| private |
| Modifier and Type | Method and Description |
|---|---|
| protected Normalizer2 | getNormalizer2(int options)
Implements abstract jdk.
|
| NFKDMode | back to summary |
|---|---|
| private NFKDMode() | |
| getNormalizer2 | back to summary |
|---|---|
| protected Normalizer2 getNormalizer2(int options) Implements abstract jdk. | |
| Modifier and Type | Field and Description |
|---|---|
| private static final NormalizerBase. |
| Access | Constructor and Description |
|---|---|
| private |
| INSTANCE | back to summary |
|---|---|
| private static final NormalizerBase. | |
| NFKDModeImpl | back to summary |
|---|---|
| private NFKDModeImpl() | |
| Access | Constructor and Description |
|---|---|
| private |
| Modifier and Type | Method and Description |
|---|---|
| protected Normalizer2 | getNormalizer2(int options)
Implements abstract jdk.
|
| NONEMode | back to summary |
|---|---|
| private NONEMode() | |
| getNormalizer2 | back to summary |
|---|---|
| protected Normalizer2 getNormalizer2(int options) Implements abstract jdk. | |
| Modifier and Type | Field and Description |
|---|---|
| private static final UnicodeSet |
| Access | Constructor and Description |
|---|---|
| private |
| INSTANCE | back to summary |
|---|---|
| private static final UnicodeSet INSTANCE | |
| Unicode32 | back to summary |
|---|---|
| private Unicode32() | |