---
last_review: "2025-01-01"
last_reviewer: "-"
documented_code: [ ]
---
```{tags} tutorial, administrator, developer, server-side-scripting
```
# Server-Side Scripting
:::{note}
This page has been migrated from the old documentation, and has not yet been fully revised.
There might be inconsistencies or errors when using with current LinkAhead versions.
:::
% TODO: Issue: https://gitlab.indiscale.com/caosdb/src/linkahead-docs/-/issues/86
% TODO: Merge with other serverside scripting files, then write Tutorial, How-To, Explanation,
% TODO: Reference from there
## Introduction
Server-Side Scripting is a LinkAhead feature which allows to install executables in the LinkAhead
Server and trigger the execution via the Server-Side Scripting API.
Small computation task, like some visualization, might be easily implemented in Python or some other
language, but cumbersome to integrate into the server. Furthermore, the LinkAhead server should stay
a general tool without burden from specific projects. Also, one might want to trigger some
standardized processing task from the web interface for convenience. For these situations,
server side scripting can be used.
The basic idea is that a script or program (script in the following) can be called to run on the
server (or elsewhere in future) to do some calculations. This triggering of the script is done over
the API so it can be done with any client. Input arguments can be passed to the script and
STDOUT and STDERR are returned.
Each script is executed in a temporary home directory, which is automatically cleaned up. Scripts
can store files in a special folder and provide a link that allows to download these afterward.
This section contains hands-on, step-by-step instructions on:
* [The installation of server-side scripts](<#install-a-minimal-script>)
* How to trigger scripts
* [With a plain html form](<#simple-static-form>)
* [With the webui's `form_elements` module](<#using-the-form-elements-module>)
* [With the LinkAhead python library](<#using-a-python-script>)
* How to implement a server-side script:
* [As a minimal bash script](<#install-a-minimal-script>)
* [With the LinkAhead python library](<#using-the-linkahead-python-library>)
* [Using the Caosadvancedtools package package](<#using-the-convenience-methods-of-caosadvancedtools>)
* [For sending email](<#how-to-send-an-email>)
* [For creating a downloadable file](<#how-to-create-a-downloadable-file>)
* How to configure the permissions:
* [Who can trigger a particular script](<#permissions-to-call-a-script>)
* [What are the permissions of the script when the script interacts with the LinkAhead server.](<#permissions-of-the-auth-token-passed-to-the-script>)
Please also consider to look at the more technical [specification](./Server-side-scripting.md)
of the API.
## Recipes
### Install a Minimal Script
This is our minimal example script, `minimal.bash`:
```bash
#!/bin/bash
printf "$0"
printf " %s" "$@"
printf "\n"
```
1. Move it to your scripting directory. By default, this in the
`custom/caosdb-server/scripting/bin/` directory under your profile directory.
2. Add executable permissions at least for the user account of the server (user in the docker
container, see `user_group` in the `profile.yml`):
`chmod +x custom/caosdb-server/scripting/bin/minimal.bash`
:::{note}
The scripting directories can be configured via the`SERVER_SIDE_SCRIPTING_BIN_DIRS` option of the
server. See [Server Configuration](/explanation/server/configuration.md) for further information.
:::
In principle, the script can be invoked via the scripting API now after restarting the server. You
can trigger the script via a browser based HTML form or by using the LinkAhead python library. See
below for more on that.
If you face any issues try the Deep Dive section which explains how to test and debug the
installation and how exactly scripts are being called and parameter and files are being passed to
the script by the server.
### How to Trigger Scripts
Server side scripts are triggered by sending a POST to the `/scripting` resource using a
`multipart/form-data` or `application/x-www-form-urlencoded` content type. There are the following
arguments that can be provided via the requests fields.
- `call`: the name of the script to be called
- `-pN`: positional arguments (e.g. `-p0`, `-p1` etc.)
- `-ONAME`: named arguments (e.g. `-Otest`, `-Onumber` etc.)
The arguments will be passed on to the script. See
the [Writing Server-Side Scripts](<#writing-server-side-scripts>) section or take
the [Deep Dive](<#deep-dive-into-the-scripting-api>) for more information.
#### Simple, Static Form
The scripting API was designed to interact with HTML forms.
A simple HTML form like this
```html
Trigger Analysis
```
will trigger a call to the scripting API when submitted. The server will execute
`minimal.bash --auth-token=[...] --algorithm=fast analyze [...]` with the respective parameters the
user specified in the form.
To install a static web form like this one, just store this html file to
`custom/caosdb-server/caosdb-webui/src/ext/html/minimal.html`.
Then (re-)start the server. You can browse to
`https://localhost:10443/webinterface/html/minimal.html` where the webform will be served.
:::{note}
Usually you need to sign in first, otherwise you might not have the permissions to execute the
script. In this case, browse to `https://localhost:10443` first.
:::
:::{note}
While static forms are a straightforward way to trigger scripts via the browser they produce rather
ugly responses. See the section on the `form_elements` module for other means to create web forms
and present the script's output in a way that is more pleasing to the eye.
:::
(using-the-form-elements-module)=
#### Using the `form_elements` module.
The linkahead-webui contains the `form_elements`module. It provides functionality to dynamically
generate forms which interact with the scripting API and present the script's results to the user.
A configuration for a simple form triggering the `minimal.bash` script might look like this:
```javascript
const config = {
script: "minimal.bash",
fields: [
{ type: "integer", name: "number", label: "A Number", required: true },
{ type: "date", name: "date", label: "A Date", required: false },
{ type: "text", name: "comment", label: "A Comment", required: false },
],
};
```
The data from this kind of forms will be converted into a json file `form.json` and uploaded to the
scripting API. The script will be called as
`minimal.bash --auth-token=[...] ./upload_files/form.json`
This sure is an opinionated implementation as it only calls scripts in a very particular way and
uses a json file instead of passing the parameters directly to the script. However, forms generated
by the `form_elements` module may contain deeply nested fields which is not conveniently mapping to
the flat structure of optional and positional parameters.
:::{note}
This section is obviouly just a teaser. Find more information about forms for triggering server-side
scripts in the webui documentation.
:::
:::{note}
The `form_elements` module of the webui works particularly good for python scripts using the
convenience methods of the caosadvancedtools package. See next section.
:::
#### Using a Python Script
The linkahead-pylib library comes with convenient functionality for triggering server-side scripts.
This can be as easy as this:
```python
from linkahead.utils.server_side_scripting import run_server_side_script
with open("test_file.txt", "w") as f:
f.write("this is a test")
response = run_server_side_script("my_script.py",
"pos0",
"pos1",
option1="val1",
option2="val2",
files={"-Ofile": "test_file.txt"})
assert response.stderr is None
assert response.code == 0
assert response.call == ('my_script.py '
'--option1=val1 --option2=val2 --file=.upload_files/test_file.txt '
'pos0 pos1')
```
You can try this out using for example the `diagnostics.py` script (it is part of
the [LinkAhead server repository](https://gitlab.com/linkahead/linkahead-server/-/blob/main/scripting/bin/administration/diagnostics.py)
and is also available on ). The script returns information about the
server in JSON format. You can do for example the following:
```python
import json
from linkahead.utils.server_side_scripting import run_server_side_script
response = run_server_side_script('administration/diagnostics.py')
print("JSON content:")
print(json.loads(response.stdout))
print("stderr:")
print(response.stderr)
```
See [`linkahead.utils.server_side_scripting`](/reference/pylib/api/linkahead.utils.server_side_scripting.rst)
module for more information.
### Writing Server-Side Scripts
Up until now this document has been about *triggering* server-side scripts, mainly. This section is
about writing convenient scripts which are to be deployed and executed as server-side scripts.
#### Minimal Requirements
* Be executable. The server will spawn a process and execute the script as the server user. So the
script needs to be executable for the server user.
* Be located in the correct directory. The server needs to find the script in one of it's configure
`SERVER_SIDE_SCRIPTING_BIN_DIRS`.
#### How to Find Uploaded Files
The uploaded files are being stored in the `./upload_files` directory, relative to the working
directory of the server-side script.
The script may search in that directory for a particular file name pattern. However, if the HTTP
request to the scripting API (using `multipart/form-data` type) specified the name of the file field
the way described above (as optional or positional parameters) the files location is passed to the
script via the respective parameter. This is the recommended approach.
#### How to Create a Downloadable File
The runtime environment of the script has a special `SHARED_DIR` variable pointing to a directory
where the script may dump files. If the script wants to output links to the files for downloading it
can construct a relative link like this:
`/${CONTEXT_ROOT_OF_THE_SERVER}/Shared/${SHARED_DIR}/${FILE_NAME}`
:::{note}
The Caosadvancedtools package provides a convenience function for this under
`caosadvancedtools.serverside.helper.get_shared_filename`. See below for more information.
:::
#### How to Invoke Transactions
To insert, retrieve, update or delete entities from within a script, use a suitable client, e.g. the
python client.
:::{note}
The LinkAhead python client library is pre-installed in every LinkAhead instance and can be used by
server-side scripts right away.
:::
#### How to Send an Email
If LinkAhead is configured appropriately, a `sendmail` client can be accessed to send emails by a
script.
:::{note}
The caosadvancedtools package provides a convenience function for this under
`caosadvancedtools.serverside.helper.send_mail`. See [below](<#send-an-email>) for more information.
:::
#### Using the LinkAhead Python Library
The LinkAhead python client library is pre-installed such that you can use it for writing scripts
right away.
In order to configure the connection to the server you need to place a `.pylinkahead.ini` file at
`custom/caosdb-server/scripting/home/.pylinkahead.ini` with the following content (or similar):
```ini
# -- custom/caosdb-server/scripting/home/.pylinkahead.ini
[Connection]
url = https://localhost:10443
cacert = /opt/caosdb/cert/caosdb.cert.pem
#ssl_insecure = True
[Misc]
#optional:
#sendmail=/usr/sbin/sendmail
```
Use the hostname in the URL for which the certificate is created and the correct port. If the
certificate is signed by a well-known CA you may leave `cacert` blank since the certificate can then
be verified without the file.
:::{note}
The script is executed inside Docker. Thus, the URL should be publicly valid (i.e. the external
hostname and port) or at least valid inside the container (which is typically the one given above
with `localhost`). If you cannot make it work because of ssl errors try to add `ssl_insecure = True`
to the
:::
% TODO: To the what? Fix the note, probably got broken in one of the moves
Now you can use the Auth Token which is being passed to the script to configure the authentication
and use the library:
```python
import linkahead as la
from caosadvancedtools.serverside import helper
def main():
parser = helper.get_argument_parser()
args = parser.parse_args()
auth_token = args.auth_token
la.configure_connection(auth_token=auth_token)
la.execute_query("FIND Experiment")
...
```
:::{note}
We use Caosadvancedtools for parsing the parameters passed to the script. See the next section for
more information.
:::
#### Using the Convenience Methods of Caosadvancedtools
The python package `caosadvancedtools` comes with the module `caosadvancedtools.serverside` which
provides functionality for sending mails, making files available for download, rendering the output
as HTML, and much more.
:::{note}
See the [documentation of caosadvancedtools](/reference/pylib/advanced-tools/index) for
more information.
:::
##### Auth Token
Caosadvancedtools come with a special argument parser which parses the Auth Token right away:
```python
import linkahead as la
from caosadvancedtools.serverside import helper
def main():
parser = helper.get_argument_parser()
args = parser.parse_args()
auth_token = args.auth_token
la.configure_connection(auth_token=auth_token)
la.execute_query("FIND Experiment")
...
```
If for some reason you need information about the user who called the script (such as name, roles,
or realm), you will find that the `linkahead.get_connection().get_username()` method will not work
because the connection is configured with an auth token. However, you can still access the user
information using the `Info` resource and the `UserInfo` (see
[pylib API documentation](/reference/pylib/api/linkahead.common.models.rst) ) therein:
```python
import linkahead as la
inf = la.Info()
username = inf.user_info.name
```
##### Using the Logger for Output
Caosadvancedtools comes with a convenient function for configuring the native python logging for
html output. This helps to produce pretty output for users calling the script via the webinterace.
```python
...
import logging
def main():
# setup logging and reporting
configure_server_side_logging("my-logger")
...
logger = logging.getLogger("my-logger")
logger.info("The script terminated successfully.")
```
This renders as
```html
The script terminated successfully.
```
##### Send an Email
% TODO: Does this need to be reworked for the new email setup?
You can use Caosadvancedtools to send an email via the `sendmail` daemon. You need to configure
LinkAhead appropriately, and add `sendmail=/usr/sbin/sendmail` to the `[Misc]` section of the
`pylinkahead.ini` of the server-side scripting home directory.
Now you can do
```python
def notify_user(user_email, subject, message):
helper.send_mail(from_addr="admin@example.com", to=user_email, subject=subject, body=message)
```
##### Uploaded File
Caosadvancedtools integrates particularly well with the `form_elements` module of the webui.
To read the json file provided by the `form_elements` trigger request you can use the argument
parser as well:
```python
...
import json
def main():
parser = helper.get_argument_parser()
args = parser.parse_args()
# Read the input from the form (form.json)
with open(args.filename) as form_json:
form_data = json.load(form_json)
...
```
##### Create Files for Download
Caosadvancedtools has a helper method named `get_shared_filename`. It generates a pair of paths. The
first path is intended for generating the download links for the user. The second is intended for
opening and write the file:
```python
...
def write_output_file(result):
display_path, internal_path = helper.get_shared_filename("output.json")
with open(internal_path, 'w') as f:
json.dump(result, f)
# use the logger to inform the user where the can download the file
logger.info(f"Download the output.json.")
...
```
### Changing Permissions
Up until now, our examples only worked for users with the administration role. If anyone else tried
to trigger a script they most likely got a permission error. This is the case when a user lacks the
appropriate permissions to *call* a script.
#### Permissions to Call a Script
You can allow users to run a particular script by granting the role permission
`SCRIPTING:EXECUTE:?PATH?` where the path is the path of the script in the server's scripting
directory (`SERVER_SIDE_SCRIPTING_BIN_DIRS`).
:::{note}
The path can also specify a glob pattern using `*`. E.g.`SCRIPTING:EXECUTE:*` is the permission to
execute any script. `SCRIPTING:EXECUTE:my_scripts/*` is the permission to execute any script under
`./my_scripts/` in one of the servers scripting directories.
:::
:::{note}
Find more information about permissions in general and how to grant them in the server docs on
permissions.
:::
#### Permissions of the Auth Token passed to the Script
By default, the Auth Token passed to the script as the `--auth-token` optional parameter
authenticates the script against the server as the user who called the script in the first place.
However, there are use cases, where we want the script to have 1) more permissions than the caller
or use a different user. One example are scripts which are to be called by unprivileged users or the
anonymous user which need to update entities. This is a pattern to allow unprivileged users to
exactly one operation (like adding a new entry to a list) but nothing else.
In this case, you can configure the `authtoken.yaml` (server option: `AUTHTOKEN_CONFIG`) to issue a
more powerful Auth Token for the script. See the server's documentation for the `AUTHTOKEN_CONFIG`
for more information.
## Example: Putting it All Together
The sections show how to create a web form using `form_element` which triggers a server-side script.
The script will be executable for any user having the `team` role, and will act with
`administration` permissions.
### Create the HTML Form
It has one field which is a file upload with the name "csvfile" and one text field with the name
"myparameter".
The javascript snippet to create such a form using the webUI's `form_elements` module looks like
this:
```javascript
// -- trigger_sss_form.js
const form_config = {
script: "handle_csv.py",
fields: [{
type: "text",
name: "myparameter",
label: "My Parameter",
required: true,
help: "Define a parameter",
}, {
type: "file",
name: "csvfile",
label: "CSV Table",
required: true,
accept: "*.csv",
help: "Upload a CSV Table",
}, ],
};
// create the form
const form = form_elements.make_form(form_config);
// now inject it into the document
document.body.append(form);
```
This snippet needs to be installed under
`custom/caosdb-server/caosdb-webui/src/ext/js/trigger_sss_form.js`
### Create a Server-Side Script
This script accepts a form.json file containing `myparameter` and `csvfile` fields as generated by
the above form. It uses the Auth Token to communicate with the server, it uses the logger configured
by Caosadvancedtools to print the output nicely renderen as HTML, and it returns a download link to
the user for downloading a generated file.
```python
# -- handle_csv.py
import os
import logging
import linkahead as la
from caosadvancedtools.serverside import helper
from caosadvancedtools.serverside.logging import configure_server_side_logging
LOGGER_NAME = "my-logger"
logger = logging.getLogger(LOGGER_NAME)
def do_something(csv_file_path, my_parameter):
"""Do something with the uploaded csv file and the parameter, then return a dict."""
la.execute_query("FIND Something")
return {"data": "example"}
def main():
# setup logging and reporting
configure_server_side_logging(LOGGER_NAME)
parser = helper.get_argument_parser()
args = parser.parse_args()
# configure the connection using the auth_token.
la.configure_connection(auth_token=args.auth_token)
# Read the input from the form (form.json)
with open(args.filename) as form_json:
form_data = json.load(form_json)
# files are uploaded to this directory
upload_dir = os.path.dirname((args.filename))
# Read content of the uploaded form data
csv_file_path = os.path.join(upload_dir, form_data["csvfile"])
my_parameter = form_data["myparameter"]
result = do_something(csv_file_path, my_parameter)
# create a downloadable file and write the results into that file
display_path, internal_path = helper.get_shared_filename("output.json")
with open(internal_path, 'w') as f:
json.dump(result, f)
# use the logger to inform the user where the can download the file
logger.info(f"Download the output.json.")
if __name__ == "__main__":
main()
```
This file needs to be installed under `custom/caosdb-server/scripting/bin/handle_csv.py` as an
executable file.
### Configure the Server
You can add a configuration file at `custom/caosdb-server/conf/ext/server.conf.d/sss.conf`
Be sure to have the server configured with `SERVER_SIDE_SCRIPTING_BIN_DIRS=`./scripting/bin`. This
is the default, so probably you are just fine.
Add a configuration file at `custom/caosdb-server/conf/ext/authtoken.yaml` for the Auth Tokens
passed to the script:
```yaml
# -- custom/caosdb-server/conf/ext/authtoken.yaml
- purpose: SCRIPTING:EXECUTE:handle_csv.py
roles:
- administration
```
:::{note}
Find more examples for the configuration in the [authtoken.example.yaml](https://gitlab.indiscale.com/caosdb/src/caosdb-server/-/blob/dev/conf/core/authtoken.example.yaml)
:::
Put the file into `custom/caosdb-server/conf/authtoken.yaml` and configure the server with
```
# -- custom/caosdb-server/conf/ext/server.conf.d/sss.conf
AUTHTOKEN_CONFIG=`./conf/ext/authtoken.yaml`
```
## Deep Dive into the Scripting API
This section shows how parameters or files are being passed to the script and how you can test an
installation.
### Testing an Installation
Given that you installed the minimal bash script from the previous section, you can start the server
and test it with curl:
```shell
curl -F call=minimal.bash 'https://localhost:10443/scripting' -H 'Cookie: SessionToken=[...]'
```
:::{note}
The scripting API expects an HTTP POST request with content type `application/x-www-form-urlencoded`
or `multipart/form-data`. It has one mandatory parameter, which is `call`, and the value must be the
script's path below the scripting directory.
:::
:::{note}
You need a valid session token. Easiest way to get one: Log in with the browser, open the developer
tools and copy-paste it from "Storage" -> "Cookies" -> "SessionToken" (firefox).
:::
:::{note}
If curl fails with "curl: (60) SSL certificate problem: self-signed certificate" or a similar error
message, you must give curl the path to the public certificate or a signing certificate authority
using the `--cacert` or `--capath` options. If you want to ignore the issue because you only do this
for testing purposes you may use the `--insecure|-k` option.
:::
The server's response should look like this:
```xml
...
```
You can see the exit code `
```
You can see that the parameters have been passed to the script and the effective call was
`minimal.bash --option1= --option2=some-value Positional0 Positional1`.
:::{note}
The session token should never be leaked by the script. This script does it only for didactic
reasons. The `...` element prints the complete call but omits the session token.
:::
### Pass Files to the Script
You can upload files and pass the files to the script via the scripting API using the POST request
with the `multipart/form-data` content type (this is the standard way of uploading files via HTTP).
Let's assume you have a file `data.json` like this:
```json
{ "my_key": "my value" }
```
You can upload the file and call the script via
```shell
curl -F call=minimal.bash -F -Omy-file=@data.json 'https://localhost:10443/scripting' -H 'Cookie: SessionToken=[...]'
```
The server will store the file under `.upload_files/data.json` relative to the scripts working
directory.
```xml
...
...
```
You can see that the server passes the file's location to the script.
:::{note}
We used a key starting with `-O` here, resulting in an optional parameter. However, you can also use
a positional parameter by using the `-pN` keys for the file.
:::
We can now add another minimal script `print_file.bash` which prints the content of the uploaded
file and install it as shown above.
```bash
#!/bin/bash
cat $2
```
:::{note}
Note: Since the first parameter is *always* the server-generated `--auth-token` the `$2` refers to
the first parameter specified by the user.
:::
This script expects to be called with one positional parameter. So we call it via
```shell
curl -F call=print_file.bash -F -p0=@data.json 'https://localhost:10443/scripting' -H 'Cookie: SessionToken=[...]'
```
The server's response will look like this:
```xml
...
...
```
You can see that the script can indeed access the file and print the content.
### Using the Auth Token From Within the Script
Up until now, we ignored the `--auth-token` parameter. It can be used by the script to make calls to
the server. By default, the authentication token passed to script is just a session token for the
current session of the user account who is calling the script: If you call the script as "user1" the
script can execute any operation on the server as the same "user1" user.
Using curl inside a minimal script `user_info.bash` we can see this happening:
```bash
#!/bin/bash
# extract the session token from the --auth-token option.
SESSION_TOKEN="${1#--auth_token=}"
curl -o /dev/null -s -w "\nNo SessionToken: %{http_code}" --insecure 'https://localhost:10443/'
curl -o /dev/null -s -w "\nWith SessionToken: %{http_code}" --insecure -H "Cookie: SessionToken=${SESSION_TOKEN}" 'https://localhost:10443'
```
If we call this script via the scripting API without any further parameters
```shell
curl -k -F call=user_info.bash 'https://localhost:10443/scripting' -H 'Cookie: SessionToken=[...]'
```
the server's response shows a "401" (Unauthorized) response for the first request and a "200" (Ok)
response for the second request:
```xml
...
...
```
### A Pattern: The Server as a Client of Itself.
So what did we do in the last step?
We installed a script `user_info.bash`. If we call that script via the scripting API, the server
executes that script on our behalf. The script sends two request back to the server and the
scripting API response reaches us after both curl requests of the script terminated.
```
| USER: calls `user_info.bash`
| | SERVER: receives the scripting API request
| | SERVER: executes `user_info.bash`
| | | SCRIPT: curl sends request 1
| | | | SERVER receives the request
| | | * SERVER responds with 401
| | | SCRIPT: curl terminates 1st time
| | | SCRIPT: curl sends request 2
| | | | SERVER receives the request
| | | * SERVER responds with 200
| | | SCRIPT: curl terminates 2nd time
| | * SCRIPT: terminates
| * SERVER: sends response `