Component templates
Sooner or later, a need for sharing configuration of components among individual pipelines arises. A typical example can be multiple pipelines loading data to a single destination. When that destination changes, without templates, one needs to go through individual pipelines and rewrite all loader instances manually. Well, in the case of LP-ETL, this operation can also be done by a text editor by replacing all occurences in the RDF files in which the pipelines are stored. Nevertheless, a proper support for this use case (and many others) comes in a form of component templates.
Creating a component template
To create a template, click on the add button of the selected component in the pipeline edit view.
A "Create template from:" dialog will appear, with configuration pre-filled according to the component instance from which the template is being created. The configuration has the same items as does the configuration of the component itself. If you want the newly created template to have a different configuration stored, you can change it now. Next, click on the "Create template" button.
Once the template is created, take a look at the original component instance. Its configuration will be disabled (grayed out). This is due to the default setting in the "Inheritance" tab, where all configuration items are set to inherit their values from the template. In case you want to inherit only some of the items and change others in the component instances, this is where you can disable the inheritance for each such item.
Managing existing templates
To see existing templates, click on the "Templates" top-level menu item. A searchable list of component templates will appear. Each item in the list represents a component template. On the second row of the item are the ancestor templates in a template hierarchy, which we will explain later. In our case, the name of the original component is displayed there.
In the "Configuration" tab of a template, its configuration can be changed, possibly affecting all component instances of the template. In the "Usage" tab of a template, its usage in pipelines can be seen. Since there can be a hierarchy of templates, the name of the template actually used in the pipeline (which can be a descendant of the currently displayed template) is displayed on the second row of the usage item. By clicking on the usage item, we get to the pipeline edit mode.
On the "Delete" tab of the template, the template can be deleted. If the template is being used, a warning will be displayed. Please note that deleting a template being used is possible, but discouraged as it will break all descendant templates and all pipelines in which it or a descendant template was used. The recommended way of deleting a template is to delete all affected pipelines and templates first.
Template hierarchies
In more complex projects, it may be necessary to use a multi-level configuration inheritance scheme. This is possible as you can create a template from an instance of a template already used in a pipeline in the same way as described above. The resulting template hierarchy will be displayed in the "Hierarchy" tab of the component instance. A similar view with ancestor and descendant templates overview can be seen for templates in their "Hierarchy" tab.
Configuration inheritance types
In the "Template" tab of a template, inheritance types can be controlled for each configuration item. The "Inherit from template" option is already familiar from the component instance. It says that this value will not be set in this template and will be inherited from the ancestor (root) template. When enabled, the corresponding configuration item will be disabled (grayed out) and its value ignored.
The "Overwrite instance" option allows us to forbid changing this value in descendant templates and component instances. When enabled, these configuration items will be hidden in the descendants and the value set in this template (or inherited from an ancestor template) will be used.
Importing pipelines with templates
Pipelines using templates can be exported to and imported from other LP-ETL instances. Pipeline export files (or the pipeline exports obtained by dereferencing the pipeline IRI) contain all templates used in that pipeline. The "Update existing templates" switch determines whether templates already present in the pipeline being imported should overwrite or reuse the templates existing in the LP-ETL instance.
When you have, e.g. a testing and a production instance, and you update your templates in the testing instance and want to update the production instance by importing pipeline from the testing envrionment, you probably want to rewrite the pre-existing templates.
On the other hand, when you have two independent instances, e.g. of two developers, which load data to different triple stores, and this loading configuration is stored in a template, you may want to reuse the existing templates when importing a pipeline. This causes the incoming pipeline to load to the correct triple store, if it uses the same loader templates.