---
last_review: "2025-01-01"
last_reviewer: "-"
documented_code: []
---

```{tags} tutorial, advanced-user, crawler
```

# Crawler Tutorials

:::{note}
This new documentation page has not yet been fully reviewed and may be incomplete.
:::
% TODO: Discuss whether it may make sense to subdivide the toctree. The Basic Functionality /
% TODO: Standard Crawler Applications / Specific Applications division would in my opinion make more
% TODO: sense in explanation, but we could have Introduction / Custom converters and transformers /
% TODO: Specific Applications or something along those lines, which does not separate concepts and
% TODO: quite as strictly. We could also keep Introduction files top-level, and have eLabFTW in
% TODO: Specific Applications.

```{toctree}
:caption: 'Integrations'
:hidden:
:maxdepth: 3

Running the Crawler from Python <helloworld>
Crawling parameter files <parameter_file>
CFood tutorial <cfood>
Crawling a single CSV file <single_file>
Setting up a crawler workflow <workflow>

Standard converters <standard_converters>
Further converters <further_converters>
Custom converter tutorial <custom_converters>
Tutorial on transformation functions <transform_functions>
Crawler macro tutorial <macros>

Crawling a SCIFOLDER folder structure <sci_folder>
The eLabFTW Crawler <elab_crawler/index>
```

This chapter covers tutorials on various topics related to the {term}`crawler <Crawler>`,
the main tool for automatic data insertion into LinkAhead.
Its task is to automatically synchronize data found on file systems or in other
sources of data with the semantic data model of LinkAhead.
More specifically, data that is contained in a hierarchical structure is converted to a data
structure that is consistent with a predefined semantic data model.
The hierarchical structure can be for example a file tree. However it can be
also something different like the contents of a JSON file or a file tree with JSON files.

These tutorials can be grouped into three subtopics as follows:

- Tutorials introducing to the basic functionality, see
  [Basic Crawler Functionality](#basic-crawler-functionality)
- Tutorials covering standard cases for data integration, like crawling CSV files can be found in
  the section: [Standard Crawler Applications](#standard-crawler-applications)
- Tutorials discussing more advanced and more specific applications can be found in the subtopic:
  [Specific Applications](#specific-applications)

## Basic Crawler Functionality

- Generic procedure of setting up a crawler workflow, see: [Setting up a crawler workflow](workflow)
- Tutorial about crawler macros: [Crawler macro tutorial](./macros)
- Tutorial on how to create custom converters: [Custom converter tutorial](custom_converters)
- [CFood tutorial](./cfood)
- Tutorial on [transformation functions](transform_functions)
- [Standard converters](standard_converters)
- [Further converters](further_converters)

## Standard Crawler Applications

- [Crawling CSV files](single_file)
- [Running the Crawler from Python](helloworld)
- [Crawling parameter files](parameter_file)

## Specific Applications

- Crawler for the SCIFOLDER [folder structure](sci_folder)
- Setting up a crawler for integrating data from the
  [electronic lab notebook elabFTW](elab_crawler/index)