Top Description Fields Constructors Methods

org.jabref.logic.ai.embedding

public Class EmbeddingModelMetadataService

Class Inheritance

extends Object

java.lang.Object
org.jabref.logic.ai.embedding.EmbeddingModelMetadataService

Annotations

@NullMarked

Imports

java.io.IOException, java.net.URI, .URISyntaxException, java.util.Collection, .List, .Locale, .Map, .Optional, .OptionalInt, .OptionalLong, .Set, java.util.concurrent.ConcurrentHashMap, org.jabref.logic.ai.preferences.AiPreferences, org.jabref.logic.importer.FetcherException, org.jabref.logic.net.URLDownload, org.jabref.logic.util.strings.StringUtil, ai.djl.repository.Artifact, ai.djl.repository.MRL, ai.djl.repository.zoo.ModelLoader, ai.djl.repository.zoo.ModelZoo, com.google.gson.JsonArray, .JsonElement, .JsonObject, .JsonParseException, .JsonParser, org.jspecify.annotations.NullMarked, org.slf4j.Logger, .LoggerFactory

Service for discovering available AI embedding models and querying their download size and maximum snippet (token / sequence) limit dynamically at runtime from DJL and Hugging Face.

Field Summary

Modifier and Type	Field and Description
private final AiPreferences	aiPreferences
private static final Set<String>	CONFIG_AND_TOKENIZER_FILES
public static final String	DJL_PYTORCH_GROUP_ID
private static final Logger	LOGGER
private final Map<String, EmbeddingModelMetadata>	metadataCache

Constructor Summary

Access	Constructor and Description
public	EmbeddingModelMetadataService(AiPreferences aiPreferences)
public	EmbeddingModelMetadataService()

Method Summary

Modifier and Type	Method and Description
private OptionalInt	extractMaxTokensFromArtifact(ai.djl.repository.Artifact artifact)
private OptionalLong	extractSizeFromArtifact(ai.djl.repository.Artifact artifact)
private OptionalInt	fetchIntFieldFromJson(String url, String fieldName)
private OptionalInt	fetchMaxTokensFromHuggingFace(String modelName)
private OptionalLong	fetchSizeFromHuggingFace(String modelName)
public List<String>	getAvailableModels() Returns the list of available embedding models discovered from the DJL HuggingFace Model Zoo.
public Optional<EmbeddingModelMetadata>	getMetadata(String modelName) Queries metadata (download size in bytes and maximum snippet length in tokens) for the given embedding model from DJL ModelZoo, falling back to the Hugging Face Hub REST API.
private boolean	isAiEnabled()
private boolean	isTokenizerOrConfigFile(String path)

Inherited from java.lang.Object:: clone equals finalize getClass hashCode notify notifyAll toString wait wait wait

Field Detail

aiPreferences	back to summary
private final AiPreferences aiPreferences

CONFIG_AND_TOKENIZER_FILES	back to summary
private static final Set<String> CONFIG_AND_TOKENIZER_FILES

DJL_PYTORCH_GROUP_ID	back to summary
public static final String DJL_PYTORCH_GROUP_ID

LOGGER	back to summary
private static final Logger LOGGER

metadataCache	back to summary
private final Map<String, EmbeddingModelMetadata> metadataCache

Constructor Detail

EmbeddingModelMetadataService	back to summary
public EmbeddingModelMetadataService(AiPreferences aiPreferences)

EmbeddingModelMetadataService	back to summary
public EmbeddingModelMetadataService()

Method Detail

extractMaxTokensFromArtifact	back to summary
private OptionalInt extractMaxTokensFromArtifact(ai.djl.repository.Artifact artifact)

extractSizeFromArtifact	back to summary
private OptionalLong extractSizeFromArtifact(ai.djl.repository.Artifact artifact)

fetchIntFieldFromJson	back to summary
private OptionalInt fetchIntFieldFromJson(String url, String fieldName)

fetchMaxTokensFromHuggingFace	back to summary
private OptionalInt fetchMaxTokensFromHuggingFace(String modelName)

fetchSizeFromHuggingFace	back to summary
private OptionalLong fetchSizeFromHuggingFace(String modelName)

getAvailableModels	back to summary
public List<String> getAvailableModels() Returns the list of available embedding models discovered from the DJL HuggingFace Model Zoo.

getMetadata	back to summary
public Optional<EmbeddingModelMetadata> getMetadata(String modelName) Queries metadata (download size in bytes and maximum snippet length in tokens) for the given embedding model from DJL ModelZoo, falling back to the Hugging Face Hub REST API.

isAiEnabled	back to summary
private boolean isAiEnabled()

isTokenizerOrConfigFile	back to summary
private boolean isTokenizerOrConfigFile(String path)