HTTP Request

Special purpose component, issues a set of arbitrary HTTP requests according to configuration.

Number of threads used for execution
Number of threads used for execution of the individual tasks. Network operations tend to be slow, so it is advantageous to process them in parallel.
Threads per group
For cases where there are multiple tasks aimed e.g. at the same host, this sets the number of threads to be used for a single group (e.g. a single host) as specified in the input tasks.
Skip on error
When an HTTP or another error is encountered in one of the tasks, this switch controls whether the execution of the whole pipeline should fail, or whether this is expected and therefore the execution should continue.
Follow redirects
Controls whether the component should follow HTTP 30x redirects.
Encode input URL
Percent encodes input URLs before using them as URIs in HTTP. Useful for cases where the target web server does not handle IRIs well of the input URLs may contain invalid characters.

Characteristics

ID
x-httprequest
Type
other
Inputs
Configuration - runtime configuration
Input - possible files to be sent e.g. using HTTP POST
Tasks - definition of tasks to be executed
Outputs
Output - possibly downloaded files
Report - report of results from individual tasks, including HTTP response headers
Look in pipeline
HTTP Request
Sample pipeline
available

The HTTP Request component issues a set of HTTP request. This has a wide range of use cases from checking whether a resource exists using HTTP HEAD to accessing a URL with explicitly defined HTTP headers.

Tasks configuration

Below you can see sample tasks configuration for the component. You can also import a sample pipeline. The sample executes two HTTP HEAD requests and one HTTP POST request, using which it sends a configuration.jsonld file from Input, and uses HTTP Basic Authentication headers. 

@prefix req: <http://plugins.linkedpipes.com/ontology/x-httpRequest#> .

<http://creativecommons.org/licenses/by-sa/3.0/cz/> a req:Task ;
    req:group "creativecommons.org" ;
    req:header <http://creativecommons.org/licenses/by-sa/3.0/cz//ua-header> ;
    req:method "HEAD" ;
    req:outputHeaders true ;
    req:url <http://creativecommons.org/licenses/by-sa/3.0/cz/> .

<http://creativecommons.org/licenses/by-sa/3.0/cz//ua-header> a req:Header ;
    req:name "User-Agent" ;
    req:value "LinkedPipes-ETL/0.0.0" .

<http://creativecommons.org/licenses/by-sa/4.0/deed.cs> a req:Task ;
    req:group "creativecommons.org" ;
    req:header <http://creativecommons.org/licenses/by-sa/4.0/deed.cs/ua-header> ;
    req:method "HEAD" ;
    req:outputHeaders true ;
    req:url <http://creativecommons.org/licenses/by-sa/4.0/deed.cs> .

<http://creativecommons.org/licenses/by-sa/4.0/deed.cs/ua-header> a req:Header ;
    req:name "User-Agent" ;
    req:value "LinkedPipes-ETL/0.0.0" .

<http://localhost/query3> a req:Task ;
    req:content <http://localhost/query3/content> ;
    req:header <http://localhost/query3/auth> ;
    req:method "POST" ;
    req:url "https://demo.etl.linkedpipes.com/resources/executions?pipeline=https://demo.etl.linkedpipes.com/resources/pipelines/created-1527156775493" .

<http://localhost/query3/auth> a req:Header ;
    req:name "Authorization" ;
    req:value "Basic bHQmtMSWx2aDYxTX==" .

<http://localhost/query3/content> a req:Content ;
    req:fileName "configuration.jsonld" ;
    req:fileReference "configuration.jsonld" ;
    req:name "configuration" .