Top Description Fields Constructors Methods
info.debatty.java.stringsimilarity

public abstract Class ShingleBased

extends Object
Class Inheritance
Known Direct Subclasses
info.debatty.java.stringsimilarity.SorensenDice, info.debatty.java.stringsimilarity.Cosine, info.debatty.java.stringsimilarity.Jaccard, info.debatty.java.stringsimilarity.QGram
Annotations
@Immutable
Imports
net.jcip.annotations.Immutable, java.util.Collections, .HashMap, .Map, java.util.regex.Pattern

Abstract class for string similarities that rely on set operations (like cosine similarity or jaccard index). k-shingling is the operation of transforming a string (or text document) into a set of n-grams, which can be used to measure the similarity between two strings or documents. Generally speaking, a k-gram is any sequence of k tokens. We use here the definition from Leskovec, Rajaraman & Ullman (2014), "Mining of Massive Datasets", Cambridge University Press: Multiple subsequent spaces are replaced by a single space, and a k-gram is a sequence of k characters. Default value of k is 3. A good rule of thumb is to imagine that there are only 20 characters and estimate the number of k-shingles as 20^k. For small documents like e-mails, k = 5 is a recommended value. For large documents, such as research articles, k = 9 is considered a safe choice.
Author
Thibault Debatty

Field Summary

Modifier and TypeField and Description
private static final int
private final int
k

private static final Pattern
SPACE_REG

Pattern for finding multiple following spaces.

Constructor Summary

AccessConstructor and Description
public
ShingleBased(final int k)

pack-priv

Method Summary

Modifier and TypeMethod and Description
public final int

Returns:

The length of k-shingles.
getK
()

Return k, the length of k-shingles (aka n-grams).

public final Map<String, Integer>

Returns:

the profile of this string, as an unmodifiable Map
getProfile
(final String string)

Compute and return the profile of s, as defined by Ukkonen "Approximate string-matching with q-grams and maximal matches".

Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Field Detail

DEFAULT_Kback to summary
private static final int DEFAULT_K
kback to summary
private final int k
SPACE_REGback to summary
private static final Pattern SPACE_REG

Pattern for finding multiple following spaces.

Constructor Detail

ShingleBasedback to summary
public ShingleBased(final int k)
Exceptions
IllegalArgumentException:
if k is <= 0
ShingleBasedback to summary
pack-priv ShingleBased()

Method Detail

getKback to summary
public final int getK()

Return k, the length of k-shingles (aka n-grams).

Returns:int

The length of k-shingles.

getProfileback to summary
public final Map<String, Integer> getProfile(final String string)

Compute and return the profile of s, as defined by Ukkonen "Approximate string-matching with q-grams and maximal matches". https://www.cs.helsinki.fi/u/ukkonen/TCS92.pdf The profile is the number of occurrences of k-shingles, and is used to compute q-gram similarity, Jaccard index, etc. Pay attention: the memory requirement of the profile can be up to k * size of the string

Returns:Map<String, Integer>

the profile of this string, as an unmodifiable Map