Modifier and Type	Class and Description
pack-priv class	LSTMBreakEngine.CodePointsVectorizer
public static enum	LSTMBreakEngine.EmbeddingType
pack-priv class	LSTMBreakEngine.GraphemeClusterVectorizer
public static enum	LSTMBreakEngine.LSTMClass
public static class	LSTMBreakEngine.LSTMData
pack-priv abstract class	LSTMBreakEngine.Vectorizer

Modifier and Type	Field and Description
private final LSTMBreakEngine.LSTMData	fData
private int	fScript
private final LSTMBreakEngine.Vectorizer	fVectorizer
private static final byte	MIN_WORD
private static final byte	MIN_WORD_SPAN

Access	Constructor and Description
public	LSTMBreakEngine(int script, UnicodeSet set, LSTMBreakEngine.LSTMData data)

Method Summary

Modifier and Type	Method and Description
private static void	addDotProductTo(final float[] a, final float[][] b, float[] result)
private static void	addHadamardProductTo(final float[] a, final float[] b, float[] result)
private static void	addTo(final float[] a, float[] result)
private float[]	compute(final float[][] W, final float[][] U, final float[] B, final float[] x, float[] h, float[] c)
public static LSTMBreakEngine	create(int script, LSTMBreakEngine.LSTMData data)
public static LSTMBreakEngine.LSTMData	createData(UResourceBundle bundle)
public static LSTMBreakEngine.LSTMData	createData(int script)
private static String	defaultLSTM(int script)
public int	divideUpDictionaryRange(CharacterIterator A UText representing the text fIter, int The start of the range of dictionary characters rangeStart, int The end of the range of dictionary characters rangeEnd, DictionaryBreakEngine.DequeI Output of break positions. Positions are pushed. Pre-existing contents of the output stack are unaltered. foundBreaks, boolean isPhraseBreaking) Implements abstract com.ibm.icu.impl.breakiter.DictionaryBreakEngine.divideUpDictionaryRange. Divide up a range of known dictionary characters handled by this break engine.
private static void	hadamardProductTo(final float[] a, float[] result)
public boolean	handles(int A Unicode codepoint value c) Overrides com.ibm.icu.impl.breakiter.DictionaryBreakEngine.handles. Implements com.ibm.icu.impl.breakiter.LanguageBreakEngine.handles.
public int	hashCode() Overrides java.lang.Object.hashCode. Returns a hash code value for the object.
private static float[]	make1DArray(int[] data, int start, int d1)
private static float[][]	make2DArray(int[] data, int start, int d1, int d2)
private LSTMBreakEngine.Vectorizer	makeVectorizer(LSTMBreakEngine.LSTMData data)
private static int	maxIndex(float[] data)
private static void	sigmoid(float[] result, int start, int length)
private static void	tanh(float[] result, int start, int length)

Inherited from com.ibm.icu.impl.breakiter.DictionaryBreakEngine:: findBreaks setCharacters

Field Detail

fData	back to summary
private final LSTMBreakEngine.LSTMData fData

fScript	back to summary
private int fScript

fVectorizer	back to summary
private final LSTMBreakEngine.Vectorizer fVectorizer

MIN_WORD	back to summary
private static final byte MIN_WORD

MIN_WORD_SPAN	back to summary
private static final byte MIN_WORD_SPAN

Constructor Detail

LSTMBreakEngine	back to summary
public LSTMBreakEngine(int script, UnicodeSet set, LSTMBreakEngine.LSTMData data)

Method Detail

addDotProductTo	back to summary
private static void addDotProductTo(final float[] a, final float[][] b, float[] result)

addHadamardProductTo	back to summary
private static void addHadamardProductTo(final float[] a, final float[] b, float[] result)

addTo	back to summary
private static void addTo(final float[] a, float[] result)

compute	back to summary
private float[] compute(final float[][] W, final float[][] U, final float[] B, final float[] x, float[] h, float[] c)

create	back to summary
public static LSTMBreakEngine create(int script, LSTMBreakEngine.LSTMData data)

createData	back to summary
public static LSTMBreakEngine.LSTMData createData(UResourceBundle bundle)

createData	back to summary
public static LSTMBreakEngine.LSTMData createData(int script)

defaultLSTM	back to summary
private static String defaultLSTM(int script)

divideUpDictionaryRange	back to summary
public int divideUpDictionaryRange(CharacterIterator fIter, int rangeStart, int rangeEnd, DictionaryBreakEngine.DequeI foundBreaks, boolean isPhraseBreaking) Implements abstract com.ibm.icu.impl.breakiter.DictionaryBreakEngine.divideUpDictionaryRange. Doc from com.ibm.icu.impl.breakiter.DictionaryBreakEngine.divideUpDictionaryRange. Divide up a range of known dictionary characters handled by this break engine. Parameters fIter:CharacterIterator A UText representing the text rangeStart:int The start of the range of dictionary characters rangeEnd:int The end of the range of dictionary characters foundBreaks:DictionaryBreakEngine.DequeI Output of break positions. Positions are pushed. Pre-existing contents of the output stack are unaltered. Returns:int The number of breaks found Annotations @Override

divideUpDictionaryRange

back to summary

public int divideUpDictionaryRange(CharacterIterator fIter, int rangeStart, int rangeEnd, DictionaryBreakEngine.DequeI foundBreaks, boolean isPhraseBreaking)

Implements abstract com.ibm.icu.impl.breakiter.DictionaryBreakEngine.divideUpDictionaryRange.

Doc from com.ibm.icu.impl.breakiter.DictionaryBreakEngine.divideUpDictionaryRange.

Divide up a range of known dictionary characters handled by this break engine.

Parameters

fIter:CharacterIterator: A UText representing the text
rangeStart:int: The start of the range of dictionary characters
rangeEnd:int: The end of the range of dictionary characters
foundBreaks:DictionaryBreakEngine.DequeI: Output of break positions. Positions are pushed. Pre-existing contents of the output stack are unaltered.

Returns:int

The number of breaks found

Annotations

@Override

hadamardProductTo	back to summary
private static void hadamardProductTo(final float[] a, float[] result)

handles	back to summary
public boolean handles(int c) Overrides com.ibm.icu.impl.breakiter.DictionaryBreakEngine.handles. Implements com.ibm.icu.impl.breakiter.LanguageBreakEngine.handles. Parameters c:int Doc from com.ibm.icu.impl.breakiter.LanguageBreakEngine.handles. A Unicode codepoint value Returns:boolean Doc from com.ibm.icu.impl.breakiter.LanguageBreakEngine.handles. true if the engine can handle this character, false otherwise Annotations @Override

handles

back to summary

public boolean handles(int c)

Overrides com.ibm.icu.impl.breakiter.DictionaryBreakEngine.handles.

Implements com.ibm.icu.impl.breakiter.LanguageBreakEngine.handles.

Parameters

c:int

Doc from com.ibm.icu.impl.breakiter.LanguageBreakEngine.handles.

A Unicode codepoint value

Returns:boolean

Doc from com.ibm.icu.impl.breakiter.LanguageBreakEngine.handles.

true if the engine can handle this character, false otherwise

Annotations

@Override

hashCode back to summary

hashCode	back to summary
public int hashCode() Overrides java.lang.Object.hashCode. Doc from java.lang.Object.hashCode. Returns a hash code value for the object. This method is supported for the benefit of hash tables such as those provided by `java.util.HashMap`. The general contract of `hashCode` is: Whenever it is invoked on the same object more than once during an execution of a Java application, the `hashCode` method must consistently return the same integer, provided no information used in `equals` comparisons on the object is modified. This integer need not remain consistent from one execution of an application to another execution of the same application. If two objects are equal according to the `equals` method, then calling the `hashCode` method on each of the two objects must produce the same integer result. It is not required that if two objects are unequal according to the `equals` method, then calling the `hashCode` method on each of the two objects must produce distinct integer results. However, the programmer should be aware that producing distinct integer results for unequal objects may improve the performance of hash tables. Returns:int a hash code value for this object. Annotations @Override

public int hashCode()

Overrides java.lang.Object.hashCode.

Doc from java.lang.Object.hashCode.

Returns a hash code value for the object. This method is supported for the benefit of hash tables such as those provided by java.util.HashMap.

The general contract of hashCode is:

Whenever it is invoked on the same object more than once during an execution of a Java application, the hashCode method must consistently return the same integer, provided no information used in equals comparisons on the object is modified. This integer need not remain consistent from one execution of an application to another execution of the same application.
If two objects are equal according to the equals method, then calling the hashCode method on each of the two objects must produce the same integer result.
It is not required that if two objects are unequal according to the equals method, then calling the hashCode method on each of the two objects must produce distinct integer results. However, the programmer should be aware that producing distinct integer results for unequal objects may improve the performance of hash tables.

Returns:int

a hash code value for this object.

Annotations

@Override

make1DArray	back to summary
private static float[] make1DArray(int[] data, int start, int d1)

make2DArray	back to summary
private static float[][] make2DArray(int[] data, int start, int d1, int d2)

makeVectorizer	back to summary
private LSTMBreakEngine.Vectorizer makeVectorizer(LSTMBreakEngine.LSTMData data)

maxIndex	back to summary
private static int maxIndex(float[] data)

sigmoid	back to summary
private static void sigmoid(float[] result, int start, int length)

tanh	back to summary
private static void tanh(float[] result, int start, int length)

com.ibm.icu.impl.breakiter back to summary

pack-priv Class LSTMBreakEngine.CodePointsVectorizer

Class Inheritance

extends Vectorizer

java.lang.Object
com.ibm.icu.impl.breakiter.LSTMBreakEngine.Vectorizer
com.ibm.icu.impl.breakiter.LSTMBreakEngine.CodePointsVectorizer

Constructor Summary

Access	Constructor and Description
public	CodePointsVectorizer(Map<String, Integer> dict)

Method Summary

Modifier and Type	Method and Description
public void	vectorize(CharacterIterator fIter, int rangeStart, int rangeEnd, List<Integer> offsets, List<Integer> indicies) Implements abstract com.ibm.icu.impl.breakiter.LSTMBreakEngine.Vectorizer.vectorize.

Inherited from com.ibm.icu.impl.breakiter.LSTMBreakEngine.Vectorizer:: getIndex

Constructor Detail

CodePointsVectorizer	back to summary
public CodePointsVectorizer(Map<String, Integer> dict)

Method Detail

vectorize	back to summary
public void vectorize(CharacterIterator fIter, int rangeStart, int rangeEnd, List<Integer> offsets, List<Integer> indicies) Implements abstract com.ibm.icu.impl.breakiter.LSTMBreakEngine.Vectorizer.vectorize.

com.ibm.icu.impl.breakiter back to summary

public final Enum LSTMBreakEngine.EmbeddingType

Class Inheritance

extends Enum<LSTMBreakEngine.EmbeddingType>

java.lang.Object
java.lang.Enum
com.ibm.icu.impl.breakiter.LSTMBreakEngine.EmbeddingType

Field Summary

Modifier and Type	Field and Description
public static final LSTMBreakEngine.EmbeddingType	CODE_POINTS
public static final LSTMBreakEngine.EmbeddingType	GRAPHEME_CLUSTER
public static final LSTMBreakEngine.EmbeddingType	UNKNOWN

Constructor Summary

Access	Constructor and Description
private	EmbeddingType()

Method Summary

Modifier and Type	Method and Description
public static LSTMBreakEngine.EmbeddingType	valueOf(String name)
public static LSTMBreakEngine.EmbeddingType[]	values()

Inherited from java.lang.Enum:: clone compareTo describeConstable equals finalize getDeclaringClass hashCode name ordinal toString valueOf

Field Detail

CODE_POINTS	back to summary
public static final LSTMBreakEngine.EmbeddingType CODE_POINTS

GRAPHEME_CLUSTER	back to summary
public static final LSTMBreakEngine.EmbeddingType GRAPHEME_CLUSTER

UNKNOWN	back to summary
public static final LSTMBreakEngine.EmbeddingType UNKNOWN

Constructor Detail

EmbeddingType	back to summary
private EmbeddingType()

Method Detail

valueOf	back to summary
public static LSTMBreakEngine.EmbeddingType valueOf(String name)

values	back to summary
public static LSTMBreakEngine.EmbeddingType[] values()

com.ibm.icu.impl.breakiter back to summary

pack-priv Class LSTMBreakEngine.GraphemeClusterVectorizer

Class Inheritance

extends Vectorizer

java.lang.Object
com.ibm.icu.impl.breakiter.LSTMBreakEngine.Vectorizer
com.ibm.icu.impl.breakiter.LSTMBreakEngine.GraphemeClusterVectorizer

Constructor Summary

Access	Constructor and Description
public	GraphemeClusterVectorizer(Map<String, Integer> dict)

Method Summary

Modifier and Type	Method and Description
private String	substring(CharacterIterator text, int startPos, int endPos)
public void	vectorize(CharacterIterator text, int startPos, int endPos, List<Integer> offsets, List<Integer> indicies) Implements abstract com.ibm.icu.impl.breakiter.LSTMBreakEngine.Vectorizer.vectorize.

Inherited from com.ibm.icu.impl.breakiter.LSTMBreakEngine.Vectorizer:: getIndex

Constructor Detail

GraphemeClusterVectorizer	back to summary
public GraphemeClusterVectorizer(Map<String, Integer> dict)

Method Detail

substring	back to summary
private String substring(CharacterIterator text, int startPos, int endPos)

vectorize	back to summary
public void vectorize(CharacterIterator text, int startPos, int endPos, List<Integer> offsets, List<Integer> indicies) Implements abstract com.ibm.icu.impl.breakiter.LSTMBreakEngine.Vectorizer.vectorize.

com.ibm.icu.impl.breakiter back to summary

public final Enum LSTMBreakEngine.LSTMClass

Class Inheritance

extends Enum<LSTMBreakEngine.LSTMClass>

java.lang.Object
java.lang.Enum
com.ibm.icu.impl.breakiter.LSTMBreakEngine.LSTMClass

Field Summary

Modifier and Type	Field and Description
public static final LSTMBreakEngine.LSTMClass	BEGIN
public static final LSTMBreakEngine.LSTMClass	END
public static final LSTMBreakEngine.LSTMClass	INSIDE
public static final LSTMBreakEngine.LSTMClass	SINGLE

Constructor Summary

Access	Constructor and Description
private	LSTMClass()

Method Summary

Modifier and Type	Method and Description
public static LSTMBreakEngine.LSTMClass	valueOf(String name)
public static LSTMBreakEngine.LSTMClass[]	values()

Inherited from java.lang.Enum:: clone compareTo describeConstable equals finalize getDeclaringClass hashCode name ordinal toString valueOf

Field Detail

BEGIN	back to summary
public static final LSTMBreakEngine.LSTMClass BEGIN

END	back to summary
public static final LSTMBreakEngine.LSTMClass END

INSIDE	back to summary
public static final LSTMBreakEngine.LSTMClass INSIDE

SINGLE	back to summary
public static final LSTMBreakEngine.LSTMClass SINGLE

Constructor Detail

LSTMClass	back to summary
private LSTMClass()

Method Detail

valueOf	back to summary
public static LSTMBreakEngine.LSTMClass valueOf(String name)

values	back to summary
public static LSTMBreakEngine.LSTMClass[] values()

com.ibm.icu.impl.breakiter back to summary

public Class LSTMBreakEngine.LSTMData

Class Inheritance

extends Object

java.lang.Object
com.ibm.icu.impl.breakiter.LSTMBreakEngine.LSTMData

ICU Status: Internal ICU component.

Field Summary

Modifier and Type	Field and Description
public float[]	fBackwardB
public float[][]	fBackwardU
public float[][]	fBackwardW
public Map<String, Integer>	fDict
public float[][]	fEmbedding
public float[]	fForwardB
public float[][]	fForwardU
public float[][]	fForwardW
public String	fName
public float[]	fOutputB
public float[][]	fOutputW
public LSTMBreakEngine.EmbeddingType	fType

Constructor Summary

Access	Constructor and Description
private	LSTMData()
public	LSTMData(UResourceBundle rb)

Method Summary

Inherited from java.lang.Object:: clone equals finalize getClass hashCode notify notifyAll toString wait wait wait

Field Detail

fBackwardB	back to summary
public float[] fBackwardB

fBackwardU	back to summary
public float[][] fBackwardU

fBackwardW	back to summary
public float[][] fBackwardW

fDict	back to summary
public Map<String, Integer> fDict

fEmbedding	back to summary
public float[][] fEmbedding

fForwardB	back to summary
public float[] fForwardB

fForwardU	back to summary
public float[][] fForwardU

fForwardW	back to summary
public float[][] fForwardW

fName	back to summary
public String fName

fOutputB	back to summary
public float[] fOutputB

fOutputW	back to summary
public float[][] fOutputW

fType	back to summary
public LSTMBreakEngine.EmbeddingType fType

Constructor Detail

LSTMData	back to summary
private LSTMData()

LSTMData	back to summary
public LSTMData(UResourceBundle rb)

com.ibm.icu.impl.breakiter back to summary

pack-priv abstract Class LSTMBreakEngine.Vectorizer

Class Inheritance

extends Object

java.lang.Object
com.ibm.icu.impl.breakiter.LSTMBreakEngine.Vectorizer

Known Direct Subclasses

com.ibm.icu.impl.breakiter.LSTMBreakEngine.CodePointsVectorizer, com.ibm.icu.impl.breakiter.LSTMBreakEngine.GraphemeClusterVectorizer

Field Summary

Modifier and Type	Field and Description
private Map<String, Integer>	fDict

Constructor Summary

Access	Constructor and Description
public	Vectorizer(Map<String, Integer> dict)

Method Summary

Modifier and Type	Method and Description
protected int	getIndex(String token)
public abstract void	vectorize(CharacterIterator fIter, int rangeStart, int rangeEnd, List<Integer> offsets, List<Integer> indicies)

Inherited from java.lang.Object:: clone equals finalize getClass hashCode notify notifyAll toString wait wait wait

Field Detail

fDict	back to summary
private Map<String, Integer> fDict

Constructor Detail

Vectorizer	back to summary
public Vectorizer(Map<String, Integer> dict)

Method Detail

getIndex	back to summary
protected int getIndex(String token)

vectorize	back to summary
public abstract void vectorize(CharacterIterator fIter, int rangeStart, int rangeEnd, List<Integer> offsets, List<Integer> indicies)

public Class LSTMBreakEngine

Nested and Inner Type Summary

Field Summary

Constructor Summary

Method Summary

Field Detail

Constructor Detail

Method Detail

pack-priv Class LSTMBreakEngine.CodePointsVectorizer

Constructor Summary

Method Summary

Constructor Detail

Method Detail

public final Enum LSTMBreakEngine.EmbeddingType

Field Summary

Constructor Summary

Method Summary

Field Detail

Constructor Detail

Method Detail

pack-priv Class LSTMBreakEngine.GraphemeClusterVectorizer

Constructor Summary

Method Summary

Constructor Detail

Method Detail

public final Enum LSTMBreakEngine.LSTMClass

Field Summary

Constructor Summary

Method Summary

Field Detail

Constructor Detail

Method Detail

public Class LSTMBreakEngine.LSTMData

Field Summary

Constructor Summary

Method Summary

Field Detail

Constructor Detail

pack-priv abstract Class LSTMBreakEngine.Vectorizer

Field Summary

Constructor Summary

Method Summary

Field Detail

Constructor Detail

Method Detail