---
last_review: "2025-01-01"
last_reviewer: "-"
documented_code: []
---

```{tags} tutorial, crawler, advanced-user, administrator
```

# Crawler Macro Tutorial

:::{note}
This page has been migrated from the old documentation, and has not yet been fully revised.
There might be inconsistencies or errors when using with current LinkAhead versions.
:::
% TODO: Issue: https://gitlab.indiscale.com/caosdb/src/linkahead-docs/-/issues/82
% TODO: Archive documentation if for old crawler, Split into tutorial & explanation

## Introduction

Macros highly facilitate the writing of complex [CFoods](./cfood). Consider the following common
example:

(example-files)=

```yaml
ExperimentalData:
  type: Directory
  match: ExperimentalData
  subtree:
    README:
      type: SimpleFile
      match: ^README.md$
      records:
        ReadmeFile:
          parents:
          - MarkdownFile
          role: File
          path: $README
          file: $README
```

This example just inserts a file called `README.md` contained in Folder `ExpreimentalData/` into
LinkAhead, assigns the {term}`parent <Parent>` ({term}`RecordType`) `MarkdownFile` and allows for
later referencing this {term}`entity <Entity>` within the cfood. As file objects are created in the
cfood specification using the `records` section with the special role `File`, defining and using
many files can become very cumbersome and make the cfood file difficult to read.

The same version using cfood macros could be defined as follows:

(example-files-2)=

```yaml
---
metadata:
  macros:
  - !defmacro
    name: MarkdownFile
    params:
      name: null
      filename: null
    definition:
      ${name}_filename:
        type: SimpleFile
        match: $filename
        records:
          $name:
            parents:
            - MarkdownFile
            role: File
            path: ${name}_filename
            file: ${name}_filename
---
ExperimentalData:
  type: Directory
  match: ExperimentalData
  subtree: !macro
    MarkdownFile:
    - name: README
      filename: ^README.md$
```

The "MarkdownFile" key and its value will be replaced by everything that is
given below "definition" in the Macro.

The expanded version of `ExperimentalData` will look like:

(example-files-2-expanded)=

```yaml
ExperimentalData:
  match: ExperimentalData
  subtree:
    README_filename:
      match: ^README.md$
      records:
        README:
          file: README_filename
          parents:
          - MarkdownFile
          path: README_filename
          role: File
      type: SimpleFile
  type: Directory
```

This {ref}`example<example-files-2>` can also be found in the macro unit tests (see
{func}`unittests.test_macros.test_documentation_example_2`).

### Mixing macros and plain definitions

You can also mix macros and plain definitions. Whenever a name cannot be resolved to a macro, a
plain yaml node definition is used as a fallback:

```yaml
---
metadata:
  macros:
  - !defmacro
    name: MarkdownFile
# ... Definition here ...
---
ExperimentalData:
  type: Directory
  match: ExperimentalData
  subtree: !macro
    MarkdownFile:
    - name: README
      filename: ^README.md$
    OtherContent:  # There is no macro named "OtherContent", so this is parsed as normal content.
      type: SimpleFile
      match: .*txt
      records:
        # ... Normal content ...
```

## Complex example

Let's try something more complex: what happens to multiple `$`? This example demonstrates the use
of `macro` variable substitutions to generate `crawler` variable substitutions:

- `$$` will be converted into `$`.
- `$$$nodename` will retain a single `$` and substitute `$nodename` during macro expansion.
- So in the cfood, if `nodename: value`, the string `$$$nodename` will be converted to
  `$value`.

(example-1)=

```yaml
macros:
- !defmacro
  name: SimulationDatasetFile
  params:
    match: null
    recordtype: null
    nodename: null
  definition:
    $nodename:
      match: $match
      type: SimpleFile
      records:
        File:
          parents:
          - $recordtype
          role: File
          path: $$$nodename
          file: $$$nodename
        Simulation:
          $recordtype: +$File
```

The expanded version of the {ref}`example above<example-1>` (with `nodename: Dataset`) can be seen
{ref}`here<example-1-expanded>`:

(example-1-expanded)=

```yaml
SimulationData:
  match: SimulationData
  subtree:
    Dataset:
      match: .*
      records:
        File:
          file: $Dataset
          parents:
          - DatasetFile
          path: $Dataset
          role: File
        Simulation:
          DatasetFile: +$File
      type: SimpleFile
  type: Directory
```

This example can also be found in the macro unit tests (see
{func}`unittests.test_macros.test_documentation_example_1`).

## Using macros multiple times

To use the same macro multiple times in the same yaml node, lists can be used:

(example-multiple)=

```yaml
---
metadata:
  macros:
    - !defmacro
      name: test_twice
      params:
        macro_name: default_name
        a: 4
      definition:
        $macro_name:
          something:
            a: $a
---
extroot: !macro
  test_twice:
  - macro_name: once  # <- This is the first replacement of the macro
  - macro_name: twice # <- This is the second one, with different arguments
    a: 5
  - {}                # <- This is the third one, just using default arguments
```

This {ref}`example<example-multiple>` is taken from the macro unit tests (see
{func}`unittests.test_macros.test_use_macro_twice`).

The example will be expanded to:

(example-multiple-expanded)=

```yaml
extroot:
  default_name:
    something:
      a: '4'
  once:
    something:
      a: '4'
  twice:
    something:
      a: '5'
```

Note, that you need to make sure that subsequent macro calls do not
use the same top level key. Because later versions would overwrite previous
ones. Here we used `$macro_name` to prevent that.

## Limitations

Currently it is not possible to use the same macro twice in the same yaml node, if it occurs in
different positions. Consider:

(example-multiple-limitation)=

```yaml
---
metadata:
  macros:
    - !defmacro
      name: test_twice
      params:
        macro_name: default_name
        a: 4
      definition:
        $macro_name:
          something:
            a: $a
---
extroot: !macro
  test_twice:
  - macro_name: once  # <- This is the first replacement of the macro

  Other_node:
    type: test

  test_twice:  # This is NOT possible as each key
               # can only appear once in a yaml node.
  - macro_name: twice # <- This is the second one, with different arguments
    a: 5
  - {}                # <- This is the third one, just using default arguments
```

This should not be a real limitation however, as the order of nodes does not matter for the
{term}`crawler <Crawler>`.

## Using macros within macro definitions

It is possible to use other macros in macro definitions. Again, examples can be found in
the macro unit tests (see e.g. {func}`unittests.test_macros.test_macros_in_macros`):

(example-macros-in-macros)=

```yaml
---
metadata:
  crawler-version: 0.3.1
  macros:
    - !defmacro
      name: one_macro
      params:
        a: 25
      definition:
        macro_sub_$a:
          b: $a
          another_param: 3
    - !defmacro
      name: test_macrodef
      params: {}
      definition:
        macro_top: !macro
          one_macro:
          - a: 17
          - {}
          - a: 98
          not_macro:
            a: 26
---
extroot: !macro
    test_macrodef:
```

% TODO: to be continued