JabRef 6.0-beta.1 · org.jabref.logic.crawler.StudyRepository

Modifier and Type	Field and Description
private final BibEntryTypesManager	bibEntryTypesManager
public static final int	DEFAULT_RESULT_LIMIT
private final FileUpdateMonitor	fileUpdateMonitor
private final SlrGitHandler	gitHandler
private static final Logger	LOGGER
private static final Pattern	MATCH_COLON
private static final Pattern	MATCH_ILLEGAL_CHARACTERS
private final CliPreferences	preferences
private static final String	REMOTE
private final Path	repositoryPath
private static final String	SEARCH_BRANCH
private final Study	study
public static final String	STUDY_DEFINITION_FILE_NAME
public static final String	STUDY_LOCK_FILE_NAME
private final Path	studyDefinitionFile
private static final String	WORK_BRANCH

Access	Constructor and Description
public	StudyRepository(Path Where the repository root is located. pathToRepository, SlrGitHandler The git handler that manages any interaction with the remote repository gitHandler, CliPreferences preferences, FileUpdateMonitor fileUpdateMonitor, BibEntryTypesManager bibEntryTypesManager) Creates a study repository.

Method Summary

Modifier and Type	Method and Description
private void	addFetcherGroupToMetaData(BibDatabaseContext context, String fetcherName) Adds a fetcher group node to the database context metadata.
private Study	buildStudyLock() Builds the lock representation of the study.
private String	computeIDForQuery(String query) Helper to compute the query id for folder name creation.
private void	createBibFile(Path file)
private void	createFetcherResultFile(String query, SearchBasedFetcher searchBasedFetcher)
private void	createFolder(Path folder)
private void	createQueryResultFile(String query)
private void	createQueryResultFolder(String The query the folder is created for query) Creates a folder using the query and its corresponding query id. This folder name is unique for each query, as long as the query id in the study definition is unique for each query.
private void	createStudyResultFile()
private void	generateCiteKeys(BibDatabaseContext existingEntries, BibDatabase targetEntries)
private List<SearchBasedFetcher>	getActiveFetchers() Returns the fetchers of all enabled catalogs, catalogs without a matching fetcher are not returned
public List<StudyCatalog>	Returns: List of BibEntries of type Library getActiveLibraryEntries() Extracts all active fetchers from the library entries.
public BibDatabaseContext	getFetcherResultEntries(String query, String fetcherName) Returns entries stored in the repository for a certain query and fetcher
private Path	getPathToFetcherResultFile(String query, String fetcherName)
private Path	getPathToQueryDirectory(String query)
private Path	getPathToQueryResultFile(String query)
private Path	getPathToStudyResultFile()
public BibDatabaseContext	getQueryResultEntries(String query) Returns the merged entries stored in the repository for a certain query
public Map<String, Integer>	getResultLimitsPerCatalog() returns the effective result limit per catalog: its own override, else the study wide default, else `DEFAULT_RESULT_LIMIT`, keys match catalog names case-insensitively
public List<String>	Returns: List of all queries as Strings. getSearchQueryStrings() Returns all query strings of the study definition
public Study	getStudy()
public List<StudyQuery>	getStudyQueries() Returns all query definitions of the study, preserving catalog-specific overrides.
public BibDatabaseContext	getStudyResultEntries() Returns the merged entries stored in the repository for all queries
private Study	Returns: Returns the BibEntries parsed from the study definition file. parseStudyFile() The study definition file contains all the definitions of a study.
public void	persist(List<QueryResult> crawlResults) Persists the result locally and remotely by following the steps: Precondition: Currently checking out work branch Update the work and search branch Persist the results on the search branch Manually patch the diff of the search branch onto the work branch (as the merging will not work in certain cases without a conflict as it is context sensitive. But for this use case we do not need it to be context sensitive. So we can just prepend the patch without checking the "context" lines. Update the remote tracking branches of the work and search branch
private void	persistResults(List<QueryResult> The results that shall be persisted. crawlResults) Persists the crawling results in the local file based repository.
private void	setUpRepositoryStructureForQueriesAndFetchers() Create for each query a folder, and for each fetcher a bib file in the query folder to store its results.
private void	tagEntriesWithFetcherGroup(Collection<BibEntry> entries, String fetcherName) Tags each entry's groups field with the fetcher name, preserving any existing groups.
private StudyQuery	toLockQuery(StudyQuery query, List<String> catalogNames)
private String	Returns: a unique folder name for any query. trimNameAndAddID(String that is trimmed and combined with its query id query) Returns a string that can be used as a folder name.
private void	updateRemoteSearchAndWorkBranch() Update the remote tracking branches of the work and search branches The currently checked out branch is not changed if the method is executed successfully
private void	updateWorkAndSearchBranch() Updates the local work and search branches with changes from their tracking remote branches The currently checked out branch is not changed if the method is executed successfully
private void	writeResultToFile(Path pathToFile, BibDatabaseContext context)
private void	writeStudyLockFile() Writes the study lock file, it records the exact query sent to each catalog, so a crawl can be reproduced.

Inherited from java.lang.Object:: clone equals finalize getClass hashCode notify notifyAll toString wait wait wait

bibEntryTypesManager	back to summary
private final BibEntryTypesManager bibEntryTypesManager

DEFAULT_RESULT_LIMIT	back to summary
public static final int DEFAULT_RESULT_LIMIT

fileUpdateMonitor	back to summary
private final FileUpdateMonitor fileUpdateMonitor

gitHandler	back to summary
private final SlrGitHandler gitHandler

LOGGER	back to summary
private static final Logger LOGGER

MATCH_COLON	back to summary
private static final Pattern MATCH_COLON

MATCH_ILLEGAL_CHARACTERS	back to summary
private static final Pattern MATCH_ILLEGAL_CHARACTERS

preferences	back to summary
private final CliPreferences preferences

REMOTE	back to summary
private static final String REMOTE

repositoryPath	back to summary
private final Path repositoryPath

SEARCH_BRANCH	back to summary
private static final String SEARCH_BRANCH

study	back to summary
private final Study study

STUDY_DEFINITION_FILE_NAME	back to summary
public static final String STUDY_DEFINITION_FILE_NAME

STUDY_LOCK_FILE_NAME	back to summary
public static final String STUDY_LOCK_FILE_NAME

studyDefinitionFile	back to summary
private final Path studyDefinitionFile

WORK_BRANCH	back to summary
private static final String WORK_BRANCH

StudyRepository	back to summary
public StudyRepository(Path pathToRepository, SlrGitHandler gitHandler, CliPreferences preferences, FileUpdateMonitor fileUpdateMonitor, BibEntryTypesManager bibEntryTypesManager) throws IOException Creates a study repository. Parameters pathToRepository:Path Where the repository root is located. gitHandler:SlrGitHandler The git handler that manages any interaction with the remote repository Exceptions IOException: Thrown if the given repository does not exist, or the study definition file does not exist IllegalArgumentException: If the repository root directory does not exist, or the root directory does not contain the study definition file.

Method Detail

addFetcherGroupToMetaData	back to summary
private void addFetcherGroupToMetaData(BibDatabaseContext context, String fetcherName) Adds a fetcher group node to the database context metadata. If the group already exists, does nothing.

buildStudyLock	back to summary
private Study buildStudyLock() Builds the lock representation of the study. For each query, the catalog-specific map lists every catalog that is queried with its effective query: the catalog-specific override if one is defined for it, otherwise the query string itself. Disabled catalogs and catalogs without a matching fetcher are not listed, as no query is sent to them.

buildStudyLock

back to summary

private Study buildStudyLock()

Builds the lock representation of the study.

For each query, the catalog-specific map lists every catalog that is queried with its effective query: the catalog-specific override if one is defined for it, otherwise the query string itself. Disabled catalogs and catalogs without a matching fetcher are not listed, as no query is sent to them.

computeIDForQuery	back to summary
private String computeIDForQuery(String query) Helper to compute the query id for folder name creation.

createBibFile	back to summary
private void createBibFile(Path file)

createFetcherResultFile	back to summary
private void createFetcherResultFile(String query, SearchBasedFetcher searchBasedFetcher)

createFolder	back to summary
private void createFolder(Path folder) throws IOException

createQueryResultFile	back to summary
private void createQueryResultFile(String query)

createQueryResultFolder	back to summary
private void createQueryResultFolder(String query) throws IOException Creates a folder using the query and its corresponding query id. This folder name is unique for each query, as long as the query id in the study definition is unique for each query. Parameters query:String The query the folder is created for

createQueryResultFolder

back to summary

private void createQueryResultFolder(String query) throws IOException

Creates a folder using the query and its corresponding query id. This folder name is unique for each query, as long as the query id in the study definition is unique for each query.

Parameters

query:String: The query the folder is created for

createStudyResultFile	back to summary
private void createStudyResultFile()

generateCiteKeys	back to summary
private void generateCiteKeys(BibDatabaseContext existingEntries, BibDatabase targetEntries)

getActiveFetchers	back to summary
private List<SearchBasedFetcher> getActiveFetchers() Returns the fetchers of all enabled catalogs, catalogs without a matching fetcher are not returned

getActiveLibraryEntries	back to summary
public List<StudyCatalog> getActiveLibraryEntries() throws IllegalArgumentException Extracts all active fetchers from the library entries. Returns:List<StudyCatalog> List of BibEntries of type Library Exceptions IllegalArgumentException: If a transformation from Library entry to LibraryDefinition fails

getActiveLibraryEntries

back to summary

public List<StudyCatalog> getActiveLibraryEntries() throws IllegalArgumentException

Extracts all active fetchers from the library entries.

Returns:List<StudyCatalog>

List of BibEntries of type Library

Exceptions

IllegalArgumentException:: If a transformation from Library entry to LibraryDefinition fails

getFetcherResultEntries	back to summary
public BibDatabaseContext getFetcherResultEntries(String query, String fetcherName) throws IOException Returns entries stored in the repository for a certain query and fetcher

getPathToFetcherResultFile	back to summary
private Path getPathToFetcherResultFile(String query, String fetcherName)

getPathToQueryDirectory	back to summary
private Path getPathToQueryDirectory(String query)

getPathToQueryResultFile	back to summary
private Path getPathToQueryResultFile(String query)

getPathToStudyResultFile	back to summary
private Path getPathToStudyResultFile()

getQueryResultEntries	back to summary
public BibDatabaseContext getQueryResultEntries(String query) throws IOException Returns the merged entries stored in the repository for a certain query

getResultLimitsPerCatalog	back to summary
public Map<String, Integer> getResultLimitsPerCatalog() returns the effective result limit per catalog: its own override, else the study wide default, else `DEFAULT_RESULT_LIMIT`, keys match catalog names case-insensitively

getSearchQueryStrings	back to summary
public List<String> getSearchQueryStrings() Returns all query strings of the study definition Returns:List<String> List of all queries as Strings.

getSearchQueryStrings

back to summary

public List<String> getSearchQueryStrings()

Returns all query strings of the study definition

Returns:List<String>: List of all queries as Strings.

getStudy	back to summary
public Study getStudy()

getStudyQueries	back to summary
public List<StudyQuery> getStudyQueries() Returns all query definitions of the study, preserving catalog-specific overrides.

getStudyResultEntries	back to summary
public BibDatabaseContext getStudyResultEntries() throws IOException Returns the merged entries stored in the repository for all queries

parseStudyFile	back to summary
private Study parseStudyFile() throws IOException The study definition file contains all the definitions of a study. This method extracts this study from the yaml study definition file Returns:Study Returns the BibEntries parsed from the study definition file. Exceptions IOException: Problem opening the input stream.

parseStudyFile

back to summary

private Study parseStudyFile() throws IOException

The study definition file contains all the definitions of a study. This method extracts this study from the yaml study definition file

Returns:Study

Returns the BibEntries parsed from the study definition file.

Exceptions

IOException:: Problem opening the input stream.

persist	back to summary
public void persist(List<QueryResult> crawlResults) throws IOException, GitAPIException, SaveException, JabRefException Persists the result locally and remotely by following the steps: Precondition: Currently checking out work branch Update the work and search branch Persist the results on the search branch Manually patch the diff of the search branch onto the work branch (as the merging will not work in certain cases without a conflict as it is context sensitive. But for this use case we do not need it to be context sensitive. So we can just prepend the patch without checking the "context" lines. Update the remote tracking branches of the work and search branch

persist

back to summary

public void persist(List<QueryResult> crawlResults) throws IOException, GitAPIException, SaveException, JabRefException

Persists the result locally and remotely by following the steps: Precondition: Currently checking out work branch

Update the work and search branch
Persist the results on the search branch
Manually patch the diff of the search branch onto the work branch (as the merging will not work in certain cases without a conflict as it is context sensitive. But for this use case we do not need it to be context sensitive. So we can just prepend the patch without checking the "context" lines.
Update the remote tracking branches of the work and search branch

persistResults	back to summary
private void persistResults(List<QueryResult> crawlResults) throws IOException, GitAPIException, SaveException Persists the crawling results in the local file based repository. Parameters crawlResults:List<QueryResult> The results that shall be persisted.

persistResults

back to summary

private void persistResults(List<QueryResult> crawlResults) throws IOException, GitAPIException, SaveException

Persists the crawling results in the local file based repository.

Parameters

crawlResults:List<QueryResult>: The results that shall be persisted.

setUpRepositoryStructureForQueriesAndFetchers	back to summary
private void setUpRepositoryStructureForQueriesAndFetchers() throws IOException Create for each query a folder, and for each fetcher a bib file in the query folder to store its results.

tagEntriesWithFetcherGroup	back to summary
private void tagEntriesWithFetcherGroup(Collection<BibEntry> entries, String fetcherName) Tags each entry's groups field with the fetcher name, preserving any existing groups.

toLockQuery	back to summary
private StudyQuery toLockQuery(StudyQuery query, List<String> catalogNames)

trimNameAndAddID back to summary

trimNameAndAddID	back to summary
private String trimNameAndAddID(String query) Returns a string that can be used as a folder name. This removes all characters from the query that are illegal for directory names. Structure: ID-trimmed query Examples: Input: '(title: test-title AND abstract: Test)' as a query entry with id 12345678 Output: '12345678 - title= test-title AND abstract= Test' Input: 'abstract: Test*' as a query entry with id 87654321 Output: '87654321 - abstract= Test' Input: '"test driven"' as a query entry with id 12348765 Output: '12348765 - test driven' Note that this method might be similar to `org.jabref.logic.util.io.FileUtil#getValidFileName(String)` or `org.jabref.logic.util.io.FileNameCleaner#cleanFileName(String)` Parameters query:String that is trimmed and combined with its query id Returns:String a unique folder name for any query.

private String trimNameAndAddID(String query)

Returns a string that can be used as a folder name. This removes all characters from the query that are illegal for directory names. Structure: ID-trimmed query

Examples: Input: '(title: test-title AND abstract: Test)' as a query entry with id 12345678 Output: '12345678 - title= test-title AND abstract= Test'

Input: 'abstract: Test*' as a query entry with id 87654321 Output: '87654321 - abstract= Test'

Input: '"test driven"' as a query entry with id 12348765 Output: '12348765 - test driven'

Note that this method might be similar to org.jabref.logic.util.io.FileUtil#getValidFileName(String) or org.jabref.logic.util.io.FileNameCleaner#cleanFileName(String)

Parameters

query:String: that is trimmed and combined with its query id

Returns:String

a unique folder name for any query.

updateRemoteSearchAndWorkBranch	back to summary
private void updateRemoteSearchAndWorkBranch() throws IOException, GitAPIException, JabRefException Update the remote tracking branches of the work and search branches The currently checked out branch is not changed if the method is executed successfully

updateWorkAndSearchBranch	back to summary
private void updateWorkAndSearchBranch() throws IOException, GitAPIException Updates the local work and search branches with changes from their tracking remote branches The currently checked out branch is not changed if the method is executed successfully

writeResultToFile	back to summary
private void writeResultToFile(Path pathToFile, BibDatabaseContext context) throws SaveException

writeStudyLockFile

back to summary

private void writeStudyLockFile() throws IOException

Writes the study lock file, it records the exact query sent to each catalog, so a crawl can be reproduced.

The content depends only on the study definition, crawling the same unchanged study again produces identical content.

public Class StudyRepository

Field Summary

Constructor Summary

Method Summary

Field Detail

Constructor Detail

Method Detail