Top Description Fields Constructors Methods
org.jabref.logic.database

public Class DuplicateCheck

extends Object
Class Inheritance
Imports
java.util.Collection, .HashMap, .HashSet, .Locale, .Map, .Objects, .Optional, .Set, java.util.stream.Collectors, org.jabref.logic.util.OS, org.jabref.logic.util.strings.StringSimilarity, org.jabref.model.database.BibDatabase, .BibDatabaseMode, org.jabref.model.entry.AuthorList, .BibEntry, .BibEntryType, .BibEntryTypesManager, org.jabref.model.entry.field.BibField, .Field, .FieldProperty, .OrFields, .StandardField, org.jabref.model.entry.identifier.ISBN, org.jabref.model.entry.types.StandardEntryType, org.jabref.model.strings.StringUtil, com.google.common.collect.Sets, org.slf4j.Logger, .LoggerFactory
This class contains utility method for duplicate checking of entries.

Field Summary

Modifier and TypeField and Description
private static final double
DOUBT_RANGE

private static final double
DUPLICATE_THRESHOLD

private static final int
EMPTY_IN_BOTH

private static final int
EMPTY_IN_ONE

private static final int
EMPTY_IN_TWO

private final BibEntryTypesManager
entryTypesManager

private static final int
EQUAL

private static final Map<Field, Double>
FIELD_WEIGHTS

private static final Logger
LOGGER

private static final int
NOT_EQUAL

private static final double
REQUIRED_WEIGHT

Constructor Summary

AccessConstructor and Description
public
DuplicateCheck(BibEntryTypesManager entryTypesManager)

Method Summary

Modifier and TypeMethod and Description
private static int
compareAuthorField(final String stringOne, final String stringTwo)

private static int
compareChapterField(final String stringOne, final String stringTwo)

public static double
compareEntriesStrictly(BibEntry one, BibEntry two)

private static int
compareField(final String stringOne, final String stringTwo)

private static double[]
compareFieldSet(final Collection<Field> fields, final BibEntry one, final BibEntry two)

private static int
compareJournalField(final String stringOne, final String stringTwo)

We do not attempt to harmonize abbreviation state of the journal names, but we remove periods from the names in case they are abbreviated with and without dots:

private static boolean
compareOptionalFields(final BibEntryType type, final BibEntry one, final BibEntry two, final double[] req)

private static int
comparePagesField(final String stringOne, final String stringTwo)

Pages can be given with a variety of delimiters, "-", "--", " - ", " -- ".

private static double[]
compareRequiredFields(final BibEntryType type, final BibEntry one, final BibEntry two)

private static int
compareSingleField(final Field field, final BibEntry one, final BibEntry two)

public Optional<BibEntry>

Returns:

The first duplicate entry found. Empty Optional if no duplicates are found.
containsDuplicate(final BibDatabase
The database to search.
database, final BibEntry
The entry of which we are looking for duplicates.
entry, final BibDatabaseMode bibDatabaseMode)

Goes through all entries in the given database, and if at least one of them is a duplicate of the given entry, as per Util.isDuplicate(BibEntry, BibEntry), the duplicate is returned.

public static double

Returns:

a value in the interval [0, 1] indicating the degree of match.
correlateByWords(final String
The first string
s1, final String
The second string
s2)

Compare two strings on the basis of word-by-word correlation analysis.

private static boolean
haveDifferentChaptersOrPagesOfTheSameBook(final BibEntry one, final BibEntry two)

private static boolean
haveDifferentEditions(final BibEntry one, final BibEntry two)

private static boolean
haveDifferentEntryType(final BibEntry one, final BibEntry two)

private static boolean
haveSameIdentifier(final BibEntry one, final BibEntry two)

public boolean
isDuplicate(final BibEntry one, final BibEntry two, final BibDatabaseMode bibDatabaseMode)

Checks if the two entries represent the same publication.

private static boolean
isFarFromThreshold(double value)

private static boolean
isSingleFieldEqual(BibEntry one, BibEntry two, Field field)

private static double
similarity(final String first, final String second)

Calculates the similarity (a number within 0 and 1) between two strings.

Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAlltoStringwaitwaitwait

Field Detail

DOUBT_RANGEback to summary
private static final double DOUBT_RANGE
DUPLICATE_THRESHOLDback to summary
private static final double DUPLICATE_THRESHOLD
EMPTY_IN_BOTHback to summary
private static final int EMPTY_IN_BOTH
EMPTY_IN_ONEback to summary
private static final int EMPTY_IN_ONE
EMPTY_IN_TWOback to summary
private static final int EMPTY_IN_TWO
entryTypesManagerback to summary
private final BibEntryTypesManager entryTypesManager
EQUALback to summary
private static final int EQUAL
FIELD_WEIGHTSback to summary
private static final Map<Field, Double> FIELD_WEIGHTS
LOGGERback to summary
private static final Logger LOGGER
NOT_EQUALback to summary
private static final int NOT_EQUAL
REQUIRED_WEIGHTback to summary
private static final double REQUIRED_WEIGHT

Constructor Detail

DuplicateCheckback to summary
public DuplicateCheck(BibEntryTypesManager entryTypesManager)

Method Detail

compareAuthorFieldback to summary
private static int compareAuthorField(final String stringOne, final String stringTwo)
compareChapterFieldback to summary
private static int compareChapterField(final String stringOne, final String stringTwo)
compareEntriesStrictlyback to summary
public static double compareEntriesStrictly(BibEntry one, BibEntry two)
compareFieldback to summary
private static int compareField(final String stringOne, final String stringTwo)
compareFieldSetback to summary
private static double[] compareFieldSet(final Collection<Field> fields, final BibEntry one, final BibEntry two)
compareJournalFieldback to summary
private static int compareJournalField(final String stringOne, final String stringTwo)

We do not attempt to harmonize abbreviation state of the journal names, but we remove periods from the names in case they are abbreviated with and without dots:

compareOptionalFieldsback to summary
private static boolean compareOptionalFields(final BibEntryType type, final BibEntry one, final BibEntry two, final double[] req)
comparePagesFieldback to summary
private static int comparePagesField(final String stringOne, final String stringTwo)

Pages can be given with a variety of delimiters, "-", "--", " - ", " -- ". We do a replace to harmonize these to a simple "-" After this, a simple test for equality should be enough

compareRequiredFieldsback to summary
private static double[] compareRequiredFields(final BibEntryType type, final BibEntry one, final BibEntry two)
compareSingleFieldback to summary
private static int compareSingleField(final Field field, final BibEntry one, final BibEntry two)
containsDuplicateback to summary
public Optional<BibEntry> containsDuplicate(final BibDatabase database, final BibEntry entry, final BibDatabaseMode bibDatabaseMode)

Goes through all entries in the given database, and if at least one of them is a duplicate of the given entry, as per Util.isDuplicate(BibEntry, BibEntry), the duplicate is returned. The search is terminated when the first duplicate is found.

Parameters
database:BibDatabase

The database to search.

entry:BibEntry

The entry of which we are looking for duplicates.

Returns:Optional<BibEntry>

The first duplicate entry found. Empty Optional if no duplicates are found.

correlateByWordsback to summary
public static double correlateByWords(final String s1, final String s2)

Compare two strings on the basis of word-by-word correlation analysis.

Parameters
s1:String

The first string

s2:String

The second string

Returns:double

a value in the interval [0, 1] indicating the degree of match.

haveDifferentChaptersOrPagesOfTheSameBookback to summary
private static boolean haveDifferentChaptersOrPagesOfTheSameBook(final BibEntry one, final BibEntry two)
haveDifferentEditionsback to summary
private static boolean haveDifferentEditions(final BibEntry one, final BibEntry two)
haveDifferentEntryTypeback to summary
private static boolean haveDifferentEntryType(final BibEntry one, final BibEntry two)
haveSameIdentifierback to summary
private static boolean haveSameIdentifier(final BibEntry one, final BibEntry two)
isDuplicateback to summary
public boolean isDuplicate(final BibEntry one, final BibEntry two, final BibDatabaseMode bibDatabaseMode)

Checks if the two entries represent the same publication.

isFarFromThresholdback to summary
private static boolean isFarFromThreshold(double value)
isSingleFieldEqualback to summary
private static boolean isSingleFieldEqual(BibEntry one, BibEntry two, Field field)
similarityback to summary
private static double similarity(final String first, final String second)

Calculates the similarity (a number within 0 and 1) between two strings. http://stackoverflow.com/questions/955110/similarity-string-comparison-in-java