Verified Owner
Top Description Fields Constructors Methods
org.jabref.logic.net

public Class URLDownload

extends Object
Class Inheritance
Imports
java.io.BufferedInputStream, .BufferedReader, .DataOutputStream, .IOException, .InputStream, .InputStreamReader, .Reader, .StringWriter, .Writer, java.net.CookieHandler, .CookieManager, .CookiePolicy, .HttpCookie, .HttpURLConnection, .MalformedURLException, .URISyntaxException, .URL, .URLConnection, java.nio.charset.Charset, .StandardCharsets, java.nio.file.Files, .Path, .StandardCopyOption, java.security.NoSuchAlgorithmException, java.time.Duration, java.util.HashMap, .List, .Map, .Map.Entry, .Optional, javax.net.ssl.HttpsURLConnection, .SSLContext, org.jabref.logic.importer.FetcherClientException, .FetcherException, .FetcherServerException, .ImporterPreferences, org.jabref.logic.importer.fetcher.UnpaywallFetcher, org.jabref.logic.util.URLUtil, org.jabref.logic.util.io.FileUtil, org.jabref.logic.util.strings.StringUtil, org.jabref.model.http.SimpleHttpResponse, kong.unirest.core.HttpResponse, kong.unirest.core.Unirest, kong.unirest.core.UnirestException, org.jspecify.annotations.NonNull, .Nullable, org.slf4j.Logger, .LoggerFactory

Example

java URLDownload dl = new URLDownload(URL); String content = dl.asString(ENCODING); dl.toFile(Path); // available in FILE String contentType = dl.getMimeType();

Almost every call to a public method creates a new HTTP connection (except for asString, which uses an already opened connection).

Nothing is cached.

Implentation note: This relies on https://kong.github.io/unirest-java/configuration/ setting followRedirects to true and enabling cookie management.

Field Summary

Modifier and TypeField and Description
private Duration
connectTimeout

private static final Duration
DEFAULT_CONNECT_TIMEOUT

private @Nullable ImporterPreferences
importerPreferences

private static final Logger
LOGGER

private static final int
MAX_RETRIES

private final Map<String, String>
parameters

private String
postData

private final URL
source

private @Nullable SSLContext
sslContext

public static final String
USER_AGENT

Constructor Summary

AccessConstructor and Description
public
URLDownload(String
the URL to download from
source)

public
URLDownload(URL
The URL to download.
source)

public
URLDownload(@NonNull ImporterPreferences importerPreferences, @NonNull URL citationsUrl)

Method Summary

Modifier and TypeMethod and Description
public void
addHeader(String key, String value)

public ProgressInputStream
asInputStream()

Uses the web resource as source and creates a monitored input stream.

public ProgressInputStream
asInputStream(HttpURLConnection urlConnection)

Uses the web resource as source and creates a monitored input stream.

public String

Returns:

the downloaded string
asString()

Downloads the web resource to a String.

private static String

Returns:

the downloaded string
asString(Charset
the desired String encoding
encoding, URLConnection
an existing connection
connection)

Downloads the web resource to a String.

public boolean

Returns:

the status code of the response
canBeReached()

Check the connection by using the HEAD request.

private static void
copy(InputStream in, Writer out, Charset encoding)

public Duration
getConnectTimeout()

public List<HttpCookie>
getCookieFromUrl()

public Optional<String>
getMimeType()

public URL
getSource()

private URLConnection
getUrlConnection()

public boolean
isMimeType(String type)

public boolean
isPdf()

public URLConnection

Returns:

an open connection
openConnection()

Open a connection to this object's URL (with specified settings).

public void
setConnectTimeout(Duration connectTimeout)

public void
setPostData(String postData)

public void
toFile(Path
the destination file path.
destination)

Downloads the web resource to a file.

public String
toString()

Overrides java.lang.Object.toString.

Returns a string representation of the object.

public Path

Returns:

the path of the temporary file.
toTemporaryFile()

Downloads the web resource to a temporary file.

Inherited from java.lang.Object:
cloneequalsfinalizegetClasshashCodenotifynotifyAllwaitwaitwait

Field Detail

connectTimeoutback to summary
private Duration connectTimeout
DEFAULT_CONNECT_TIMEOUTback to summary
private static final Duration DEFAULT_CONNECT_TIMEOUT
importerPreferencesback to summary
private @Nullable ImporterPreferences importerPreferences
LOGGERback to summary
private static final Logger LOGGER
MAX_RETRIESback to summary
private static final int MAX_RETRIES
parametersback to summary
private final Map<String, String> parameters
postDataback to summary
private String postData
sourceback to summary
private final URL source
sslContextback to summary
private @Nullable SSLContext sslContext
USER_AGENTback to summary
public static final String USER_AGENT

Constructor Detail

URLDownloadback to summary
public URLDownload(String source) throws MalformedURLException
Parameters
source:String

the URL to download from

Exceptions
MalformedURLException:
if no protocol is specified in the source, or an unknown protocol is found
URLDownloadback to summary
public URLDownload(URL source)
Parameters
source:URL

The URL to download.

URLDownloadback to summary
public URLDownload(@NonNull ImporterPreferences importerPreferences, @NonNull URL citationsUrl)

Method Detail

addHeaderback to summary
public void addHeader(String key, String value)
asInputStreamback to summary
public ProgressInputStream asInputStream() throws FetcherException

Uses the web resource as source and creates a monitored input stream.

asInputStreamback to summary
public ProgressInputStream asInputStream(HttpURLConnection urlConnection) throws FetcherException

Uses the web resource as source and creates a monitored input stream.

Exposing the urlConnection is required for dynamic API limiting of CrossRef

asStringback to summary
public String asString() throws FetcherException

Downloads the web resource to a String. Uses UTF-8 as encoding.

Returns:String

the downloaded string

asStringback to summary
private static String asString(Charset encoding, URLConnection connection) throws FetcherException

Downloads the web resource to a String.

Parameters
encoding:Charset

the desired String encoding

connection:URLConnection

an existing connection

Returns:String

the downloaded string

canBeReachedback to summary
public boolean canBeReached() throws kong.unirest.core.UnirestException

Check the connection by using the HEAD request. UnirestException can be thrown for invalid request.

Returns:boolean

the status code of the response

copyback to summary
private static void copy(InputStream in, Writer out, Charset encoding) throws IOException
getConnectTimeoutback to summary
public Duration getConnectTimeout()
getCookieFromUrlback to summary
public List<HttpCookie> getCookieFromUrl() throws FetcherException
getMimeTypeback to summary
public Optional<String> getMimeType()
getSourceback to summary
public URL getSource()
getUrlConnectionback to summary
private URLConnection getUrlConnection() throws IOException
isMimeTypeback to summary
public boolean isMimeType(String type)
isPdfback to summary
public boolean isPdf()
openConnectionback to summary
public URLConnection openConnection() throws FetcherException

Open a connection to this object's URL (with specified settings).

If accessing an HTTP URL, remember to close the resulting connection after usage.

Returns:URLConnection

an open connection

setConnectTimeoutback to summary
public void setConnectTimeout(Duration connectTimeout)
setPostDataback to summary
public void setPostData(String postData)
toFileback to summary
public void toFile(Path destination) throws FetcherException

Downloads the web resource to a file.

Parameters
destination:Path

the destination file path.

toStringback to summary
public String toString()

Overrides java.lang.Object.toString.

Doc from java.lang.Object.toString.

Returns a string representation of the object. Satisfying this method's contract implies a non-null result must be returned.

Annotations
@Override
toTemporaryFileback to summary
public Path toTemporaryFile() throws FetcherException

Downloads the web resource to a temporary file.

Returns:Path

the path of the temporary file.