HTTP get list

Extractor, allows the user to download a list of files from the web.

Number of threads used for download
This component supports parallel download using multiple threads. This property sets the thread pool size.
Force to follow redirects
The component will follow HTTP 3XX redirects
Skip on errors
When enabled and one of the files in the list fails to download, the component skips it and downloads the rest of the list instead of failing

Characteristics

ID
e-httpgetfiles
Type
extractor
Inputs
RDF single graph
Outputs
Files
Look in pipeline
HTTP get list
Sample pipeline
available

The HTTP get list component downloads a list of files given on the input, potentially following redirects, and stores them under given file names. When using runtime configuration, custom HTTP headers can be set for the individual files. This can be used e.g. for HTTP Content Negotiation.

Runtime configuration

Below you can see sample runtime configuration for the component. You can also import a sample pipeline. Note that RDF blank nodes are not allowed in runtime configurations. 

@prefix httpList: <http://plugins.linkedpipes.com/ontology/e-httpGetFiles#> .

<http://localhost/resource/configuration> a httpList:Configuration ;
    httpList:reference <http://localhost/resource/ref/1> ;
    httpList:reference <http://localhost/resource/ref/2> .

<http://localhost/resource/ref/1> a httpList:Reference ;
    httpList:fileUri "https://raw.githubusercontent.com/linkedpipes/etl/master/LICENSE" ;
    httpList:fileName "LICENSE" ;
    httpList:header <http://localhost/resource/ref/1/header/1> .

<http://localhost/resource/ref/1/header/1> a httpList:Header ;
    httpList:key "Accept" ;
    httpList:value "text/turtle" .

<http://localhost/resource/ref/2> a httpList:Reference ;
    httpList:fileUri "https://raw.githubusercontent.com/linkedpipes/etl/master/README.md" ;
    httpList:fileName "README.md" .