Tags: tutorial, advanced-user, crawler

Crawler Tutorials

Note

This new documentation page has not yet been fully reviewed and may be incomplete.

This chapter covers tutorials on various topics related to the crawler, the main tool for automatic data insertion into LinkAhead. Its task is to automatically synchronize data found on file systems or in other sources of data with the semantic data model of LinkAhead. More specifically, data that is contained in a hierarchical structure is converted to a data structure that is consistent with a predefined semantic data model. The hierarchical structure can be for example a file tree. However it can be also something different like the contents of a JSON file or a file tree with JSON files.

These tutorials can be grouped into three subtopics as follows:

• Tutorials introducing to the basic functionality, see Basic Crawler Functionality

• Tutorials covering standard cases for data integration, like crawling CSV files can be found in the section: Standard Crawler Applications

• Tutorials discussing more advanced and more specific applications can be found in the subtopic: Specific Applications

Basic Crawler Functionality

• Generic procedure of setting up a crawler workflow, see: Setting up a crawler workflow

• Tutorial about crawler macros: Crawler macro tutorial

• Tutorial on how to create custom converters: Custom converter tutorial

• CFood tutorial

• Tutorial on transformation functions

• Standard converters

• Further converters

Standard Crawler Applications

• Crawling CSV files

• Running the Crawler from Python

• Crawling parameter files

Specific Applications

• Crawler for the SCIFOLDER folder structure

• Setting up a crawler for integrating data from the electronic lab notebook elabFTW