Tags: tutorial, administrator, developer, server-side-scripting

Server-Side Scripting#

Note

This page has been migrated from the old documentation, and has not yet been fully revised. There might be inconsistencies or errors when using with current LinkAhead versions.

Introduction#

Server-Side Scripting is a LinkAhead feature which allows to install executables in the LinkAhead Server and trigger the execution via the Server-Side Scripting API.

Small computation task, like some visualization, might be easily implemented in Python or some other language, but cumbersome to integrate into the server. Furthermore, the LinkAhead server should stay a general tool without burden from specific projects. Also, one might want to trigger some standardized processing task from the web interface for convenience. For these situations, server side scripting can be used.

The basic idea is that a script or program (script in the following) can be called to run on the server (or elsewhere in future) to do some calculations. This triggering of the script is done over the API so it can be done with any client. Input arguments can be passed to the script and STDOUT and STDERR are returned.

Each script is executed in a temporary home directory, which is automatically cleaned up. Scripts can store files in a special folder and provide a link that allows to download these afterward.

This section contains hands-on, step-by-step instructions on:

Please also consider to look at the more technical specification of the API.

Recipes#

Install a Minimal Script#

This is our minimal example script, minimal.bash:

#!/bin/bash
printf "$0"
printf " %s" "$@"
printf "\n"

  1. Move it to your scripting directory. By default, this in the custom/caosdb-server/scripting/bin/ directory under your profile directory.

  2. Add executable permissions at least for the user account of the server (user in the docker container, see user_group in the profile.yml): chmod +x custom/caosdb-server/scripting/bin/minimal.bash

Note

The scripting directories can be configured via theSERVER_SIDE_SCRIPTING_BIN_DIRS option of the server. See Server Configuration for further information.

In principle, the script can be invoked via the scripting API now after restarting the server. You can trigger the script via a browser based HTML form or by using the LinkAhead python library. See below for more on that.

If you face any issues try the Deep Dive section which explains how to test and debug the installation and how exactly scripts are being called and parameter and files are being passed to the script by the server.

How to Trigger Scripts#

Server side scripts are triggered by sending a POST to the /scripting resource using a multipart/form-data or application/x-www-form-urlencoded content type. There are the following arguments that can be provided via the requests fields.

  • call: the name of the script to be called

  • -pN: positional arguments (e.g. -p0, -p1 etc.)

  • -ONAME: named arguments (e.g. -Otest, -Onumber etc.)

The arguments will be passed on to the script. See the Writing Server-Side Scripts section or take the Deep Dive for more information.

Simple, Static Form#

The scripting API was designed to interact with HTML forms.

A simple HTML form like this

<h1>Trigger Analysis</h1>
<form action="/scripting" method="post">
<input type="hidden" name="call" value="minimal.bash"/>
<input type="hidden" name="-p0" value="analyze"/>
<label>Some parameter<input type="text" name="-p1"/></label>
<label>Algorithm<input type="text" name="-Oalgorithm" value="fast"/></label>
<input type="submit" value="Submit">
</form>

will trigger a call to the scripting API when submitted. The server will execute minimal.bash --auth-token=[...] --algorithm=fast analyze [...] with the respective parameters the user specified in the form.

To install a static web form like this one, just store this html file to custom/caosdb-server/caosdb-webui/src/ext/html/minimal.html.

Then (re-)start the server. You can browse to https://localhost:10443/webinterface/html/minimal.html where the webform will be served.

Note

Usually you need to sign in first, otherwise you might not have the permissions to execute the script. In this case, browse to https://localhost:10443 first.

Note

While static forms are a straightforward way to trigger scripts via the browser they produce rather ugly responses. See the section on the form_elements module for other means to create web forms and present the script’s output in a way that is more pleasing to the eye.

Using the form_elements module.#

The linkahead-webui contains the form_elementsmodule. It provides functionality to dynamically generate forms which interact with the scripting API and present the script’s results to the user.

A configuration for a simple form triggering the minimal.bash script might look like this:

const config = {
  script: "minimal.bash",
  fields: [
    { type: "integer", name: "number", label: "A Number", required: true },
    { type: "date", name: "date", label: "A Date", required: false },
    { type: "text", name: "comment", label: "A Comment", required: false },
  ],
};

The data from this kind of forms will be converted into a json file form.json and uploaded to the scripting API. The script will be called as minimal.bash --auth-token=[...] ./upload_files/form.json

This sure is an opinionated implementation as it only calls scripts in a very particular way and uses a json file instead of passing the parameters directly to the script. However, forms generated by the form_elements module may contain deeply nested fields which is not conveniently mapping to the flat structure of optional and positional parameters.

Note

This section is obviouly just a teaser. Find more information about forms for triggering server-side scripts in the webui documentation.

Note

The form_elements module of the webui works particularly good for python scripts using the convenience methods of the caosadvancedtools package. See next section.

Using a Python Script#

The linkahead-pylib library comes with convenient functionality for triggering server-side scripts. This can be as easy as this:

from linkahead.utils.server_side_scripting import run_server_side_script

with open("test_file.txt", "w") as f:
    f.write("this is a test")

response = run_server_side_script("my_script.py",
                                  "pos0",
                                  "pos1",
                                  option1="val1",
                                  option2="val2",
                                  files={"-Ofile": "test_file.txt"})
assert response.stderr is None
assert response.code == 0
assert response.call == ('my_script.py '
                         '--option1=val1 --option2=val2 --file=.upload_files/test_file.txt '
                         'pos0 pos1')

You can try this out using for example the diagnostics.py script (it is part of the LinkAhead server repository and is also available on https://demo.indiscale.com). The script returns information about the server in JSON format. You can do for example the following:

import json
from linkahead.utils.server_side_scripting import run_server_side_script

response = run_server_side_script('administration/diagnostics.py')
print("JSON content:")
print(json.loads(response.stdout))
print("stderr:")
print(response.stderr)

See linkahead.utils.server_side_scripting module for more information.

Writing Server-Side Scripts#

Up until now this document has been about triggering server-side scripts, mainly. This section is about writing convenient scripts which are to be deployed and executed as server-side scripts.

Minimal Requirements#

  • Be executable. The server will spawn a process and execute the script as the server user. So the script needs to be executable for the server user.

  • Be located in the correct directory. The server needs to find the script in one of it’s configure SERVER_SIDE_SCRIPTING_BIN_DIRS.

How to Find Uploaded Files#

The uploaded files are being stored in the ./upload_files directory, relative to the working directory of the server-side script.

The script may search in that directory for a particular file name pattern. However, if the HTTP request to the scripting API (using multipart/form-data type) specified the name of the file field the way described above (as optional or positional parameters) the files location is passed to the script via the respective parameter. This is the recommended approach.

How to Create a Downloadable File#

The runtime environment of the script has a special SHARED_DIR variable pointing to a directory where the script may dump files. If the script wants to output links to the files for downloading it can construct a relative link like this: /${CONTEXT_ROOT_OF_THE_SERVER}/Shared/${SHARED_DIR}/${FILE_NAME}

Note

The Caosadvancedtools package provides a convenience function for this under caosadvancedtools.serverside.helper.get_shared_filename. See below for more information.

How to Invoke Transactions#

To insert, retrieve, update or delete entities from within a script, use a suitable client, e.g. the python client.

Note

The LinkAhead python client library is pre-installed in every LinkAhead instance and can be used by server-side scripts right away.

How to Send an Email#

If LinkAhead is configured appropriately, a sendmail client can be accessed to send emails by a script.

Note

The caosadvancedtools package provides a convenience function for this under caosadvancedtools.serverside.helper.send_mail. See below for more information.

Using the LinkAhead Python Library#

The LinkAhead python client library is pre-installed such that you can use it for writing scripts right away.

In order to configure the connection to the server you need to place a .pylinkahead.ini file at custom/caosdb-server/scripting/home/.pylinkahead.ini with the following content (or similar):

# -- custom/caosdb-server/scripting/home/.pylinkahead.ini

[Connection]
url = https://localhost:10443
cacert = /opt/caosdb/cert/caosdb.cert.pem
#ssl_insecure = True

[Misc]
#optional:
#sendmail=/usr/sbin/sendmail

Use the hostname in the URL for which the certificate is created and the correct port. If the certificate is signed by a well-known CA you may leave cacert blank since the certificate can then be verified without the file.

Note

The script is executed inside Docker. Thus, the URL should be publicly valid (i.e. the external hostname and port) or at least valid inside the container (which is typically the one given above with localhost). If you cannot make it work because of ssl errors try to add ssl_insecure = True to the

Now you can use the Auth Token which is being passed to the script to configure the authentication and use the library:

import linkahead as la
from caosadvancedtools.serverside import helper

def main():
    parser = helper.get_argument_parser()
    args = parser.parse_args()
    auth_token = args.auth_token
    la.configure_connection(auth_token=auth_token)
    la.execute_query("FIND Experiment")
...

Note

We use Caosadvancedtools for parsing the parameters passed to the script. See the next section for more information.

Using the Convenience Methods of Caosadvancedtools#

The python package caosadvancedtools comes with the module caosadvancedtools.serverside which provides functionality for sending mails, making files available for download, rendering the output as HTML, and much more.

Note

See the documentation of caosadvancedtools for more information.

Auth Token#

Caosadvancedtools come with a special argument parser which parses the Auth Token right away:

import linkahead as la
from caosadvancedtools.serverside import helper

def main():
    parser = helper.get_argument_parser()
    args = parser.parse_args()
    auth_token = args.auth_token
    la.configure_connection(auth_token=auth_token)
    la.execute_query("FIND Experiment")
...

If for some reason you need information about the user who called the script (such as name, roles, or realm), you will find that the linkahead.get_connection().get_username() method will not work because the connection is configured with an auth token. However, you can still access the user information using the Info resource and the UserInfo (see pylib API documentation ) therein:

import linkahead as la

inf = la.Info()
username = inf.user_info.name
Using the Logger for Output#

Caosadvancedtools comes with a convenient function for configuring the native python logging for html output. This helps to produce pretty output for users calling the script via the webinterace.

...
import logging

def main():
    # setup logging and reporting
    configure_server_side_logging("my-logger")

    ...

    logger = logging.getLogger("my-logger")
    logger.info("The script terminated successfully.")

This renders as

<div class="alert alert-info alert-dismissible" role="alert">The script terminated successfully.</div>
Send an Email#

You can use Caosadvancedtools to send an email via the sendmail daemon. You need to configure LinkAhead appropriately, and add sendmail=/usr/sbin/sendmail to the [Misc] section of the pylinkahead.ini of the server-side scripting home directory.

Now you can do

def notify_user(user_email, subject, message):
    helper.send_mail(from_addr="admin@example.com", to=user_email, subject=subject, body=message)
Uploaded File#

Caosadvancedtools integrates particularly well with the form_elements module of the webui.

To read the json file provided by the form_elements trigger request you can use the argument parser as well:

...
import json

def main():
    parser = helper.get_argument_parser()
    args = parser.parse_args()

    # Read the input from the form (form.json)
    with open(args.filename) as form_json:
        form_data = json.load(form_json)
...
Create Files for Download#

Caosadvancedtools has a helper method named get_shared_filename. It generates a pair of paths. The first path is intended for generating the download links for the user. The second is intended for opening and write the file:

...
def write_output_file(result):
    display_path, internal_path = helper.get_shared_filename("output.json")
    with open(internal_path, 'w') as f:
        json.dump(result, f)

    # use the logger to inform the user where the can download the file
    logger.info(f"Download the <a href=/Shared/{display_path}>output.json</a>.")
...

Changing Permissions#

Up until now, our examples only worked for users with the administration role. If anyone else tried to trigger a script they most likely got a permission error. This is the case when a user lacks the appropriate permissions to call a script.

Permissions to Call a Script#

You can allow users to run a particular script by granting the role permission SCRIPTING:EXECUTE:?PATH? where the path is the path of the script in the server’s scripting directory (SERVER_SIDE_SCRIPTING_BIN_DIRS).

Note

The path can also specify a glob pattern using *. E.g.SCRIPTING:EXECUTE:* is the permission to execute any script. SCRIPTING:EXECUTE:my_scripts/* is the permission to execute any script under ./my_scripts/ in one of the servers scripting directories.

Note

Find more information about permissions in general and how to grant them in the server docs on permissions.

Permissions of the Auth Token passed to the Script#

By default, the Auth Token passed to the script as the --auth-token optional parameter authenticates the script against the server as the user who called the script in the first place.

However, there are use cases, where we want the script to have 1) more permissions than the caller or use a different user. One example are scripts which are to be called by unprivileged users or the anonymous user which need to update entities. This is a pattern to allow unprivileged users to exactly one operation (like adding a new entry to a list) but nothing else.

In this case, you can configure the authtoken.yaml (server option: AUTHTOKEN_CONFIG) to issue a more powerful Auth Token for the script. See the server’s documentation for the AUTHTOKEN_CONFIG for more information.

Example: Putting it All Together#

The sections show how to create a web form using form_element which triggers a server-side script. The script will be executable for any user having the team role, and will act with administration permissions.

Create the HTML Form#

It has one field which is a file upload with the name “csvfile” and one text field with the name “myparameter”.

The javascript snippet to create such a form using the webUI’s form_elements module looks like this:

// -- trigger_sss_form.js
const form_config = {
    script: "handle_csv.py",
    fields: [{
        type: "text",
        name: "myparameter",
        label: "My Parameter",
        required: true,
        help: "Define a parameter",
    }, {
        type: "file",
        name: "csvfile",
        label: "CSV Table",
        required: true,
        accept: "*.csv",
        help: "Upload a CSV Table",
    }, ],
};

// create the form
const form = form_elements.make_form(form_config);

// now inject it into the document
document.body.append(form);

This snippet needs to be installed under custom/caosdb-server/caosdb-webui/src/ext/js/trigger_sss_form.js

Create a Server-Side Script#

This script accepts a form.json file containing myparameter and csvfile fields as generated by the above form. It uses the Auth Token to communicate with the server, it uses the logger configured by Caosadvancedtools to print the output nicely renderen as HTML, and it returns a download link to the user for downloading a generated file.

# -- handle_csv.py
import os
import logging
import linkahead as la
from caosadvancedtools.serverside import helper
from caosadvancedtools.serverside.logging import configure_server_side_logging

LOGGER_NAME = "my-logger"
logger = logging.getLogger(LOGGER_NAME)

def do_something(csv_file_path, my_parameter):
    """Do something with the uploaded csv file and the parameter, then return a dict."""
    la.execute_query("FIND Something")

    return {"data": "example"}

def main():
    # setup logging and reporting
    configure_server_side_logging(LOGGER_NAME)

    parser = helper.get_argument_parser()
    args = parser.parse_args()

    # configure the connection using the auth_token.
    la.configure_connection(auth_token=args.auth_token)

    # Read the input from the form (form.json)
    with open(args.filename) as form_json:
        form_data = json.load(form_json)

    # files are uploaded to this directory
    upload_dir = os.path.dirname((args.filename))
    # Read content of the uploaded form data
    csv_file_path = os.path.join(upload_dir, form_data["csvfile"])
    my_parameter = form_data["myparameter"]

    result = do_something(csv_file_path, my_parameter)

    # create a downloadable file and write the results into that file
    display_path, internal_path = helper.get_shared_filename("output.json")
    with open(internal_path, 'w') as f:
        json.dump(result, f)

    # use the logger to inform the user where the can download the file
    logger.info(f"Download the <a href=/Shared/{display_path}>output.json</a>.")

if __name__ == "__main__":
    main()

This file needs to be installed under custom/caosdb-server/scripting/bin/handle_csv.py as an executable file.

Configure the Server#

You can add a configuration file at custom/caosdb-server/conf/ext/server.conf.d/sss.conf

Be sure to have the server configured with SERVER_SIDE_SCRIPTING_BIN_DIRS=./scripting/bin`. This is the default, so probably you are just fine.

Add a configuration file at custom/caosdb-server/conf/ext/authtoken.yaml for the Auth Tokens passed to the script:

# -- custom/caosdb-server/conf/ext/authtoken.yaml
- purpose: SCRIPTING:EXECUTE:handle_csv.py
  roles:
    - administration

Note

Find more examples for the configuration in the authtoken.example.yaml

Put the file into custom/caosdb-server/conf/authtoken.yaml and configure the server with

# -- custom/caosdb-server/conf/ext/server.conf.d/sss.conf
AUTHTOKEN_CONFIG=`./conf/ext/authtoken.yaml`

Deep Dive into the Scripting API#

This section shows how parameters or files are being passed to the script and how you can test an installation.

Testing an Installation#

Given that you installed the minimal bash script from the previous section, you can start the server and test it with curl:

curl -F call=minimal.bash 'https://localhost:10443/scripting' -H 'Cookie: SessionToken=[...]'

Note

The scripting API expects an HTTP POST request with content type application/x-www-form-urlencoded or multipart/form-data. It has one mandatory parameter, which is call, and the value must be the script’s path below the scripting directory.

Note

You need a valid session token. Easiest way to get one: Log in with the browser, open the developer tools and copy-paste it from “Storage” -> “Cookies” -> “SessionToken” (firefox).

Note

If curl fails with “curl: (60) SSL certificate problem: self-signed certificate” or a similar error message, you must give curl the path to the public certificate or a signing certificate authority using the --cacert or --capath options. If you want to ignore the issue because you only do this for testing purposes you may use the --insecure|-k option.

The server’s response should look like this:

<Response>
  ...
  <script code="0">
    <call>minimal.bash</call>
    <stdout>/opt/caosdb/git/caosdb-server/scripting/bin/minimal.bash --auth-token=["S","PAM","admin",...]</stdout>
    <stderr />
  </script>
</Response>

You can see the exit code <script code="0"> was 0 (no errors). The standard output and error output are there, and you can see that the server called the script with one additional --auth-token=["S", "PAM", "admin",...] option. This AuthToken can be used by the script for further communication with the server.

Note

The session token should never be leaked by the script. This script does it only for didactic reasons.

Note

You need to restart LinkAhead every time you install a new script or make changes to an existing one. If you need to add or change a script in a running system you must copy your script into the docker container to the correct directory manually.

Call a Script With Parameters#

You can add parameter to the scripting call by adding key-value pairs to your POST request. There are two types of parameters that will be processed:

  1. Keys matching -p[1-9][0-9]*, e.g. -p0, -p1,… These are positional parameters which are going to be passed to the script.

  2. Keys starting with -O. These are options which are going to be parsed to the script.

Using our minimal script and calling it via curl:

curl -F call=minimal.bash -F -p0=Positional0 -F -p1=Positional1 -F -Ooption1= -F -Ooption2=some-value 'https://localhost:10443/scripting' -H 'Cookie: SessionToken=[...]'

This server’s response should look like this:

<Response>
  ...
  <script code="0">
    <call>minimal.bash --option1= --option2=some-value Positional0 Positional1</call>
    <stdout>/opt/caosdb/git/caosdb-server/scripting/bin/minimal.bash --auth-token=["S","PAM","admin",...] --option1= --option2=some-value Positional0 Positional1</stdout>
    <stderr />
  </script>
</Response>

You can see that the parameters have been passed to the script and the effective call was minimal.bash --option1= --option2=some-value Positional0 Positional1.

Note

The session token should never be leaked by the script. This script does it only for didactic reasons. The <call>... element prints the complete call but omits the session token.

Pass Files to the Script#

You can upload files and pass the files to the script via the scripting API using the POST request with the multipart/form-data content type (this is the standard way of uploading files via HTTP).

Let’s assume you have a file data.json like this:

{ "my_key": "my value" }

You can upload the file and call the script via

curl -F call=minimal.bash -F -Omy-file=@data.json 'https://localhost:10443/scripting' -H 'Cookie: SessionToken=[...]'

The server will store the file under .upload_files/data.json relative to the scripts working directory.

...
<script code="0">
  <call>minimal.bash --my-file=.upload_files/data.json </call>
  ...
</script>
...

You can see that the server passes the file’s location to the script.

Note

We used a key starting with -O here, resulting in an optional parameter. However, you can also use a positional parameter by using the -pN keys for the file.

We can now add another minimal script print_file.bash which prints the content of the uploaded file and install it as shown above.

#!/bin/bash
cat $2

Note

Note: Since the first parameter is always the server-generated --auth-token the $2 refers to the first parameter specified by the user.

This script expects to be called with one positional parameter. So we call it via

curl -F call=print_file.bash -F -p0=@data.json 'https://localhost:10443/scripting' -H 'Cookie: SessionToken=[...]'

The server’s response will look like this:

...
<script code="0">
  <call>print_file.bash .upload_files/data.json</call>
  <stdout>{"my-key":"my-value"}</stdout>
  <stderr />
</script>
...

You can see that the script can indeed access the file and print the content.

Using the Auth Token From Within the Script#

Up until now, we ignored the --auth-token parameter. It can be used by the script to make calls to the server. By default, the authentication token passed to script is just a session token for the current session of the user account who is calling the script: If you call the script as “user1” the script can execute any operation on the server as the same “user1” user.

Using curl inside a minimal script user_info.bash we can see this happening:

#!/bin/bash

# extract the session token from the --auth-token option.
SESSION_TOKEN="${1#--auth_token=}"

curl -o /dev/null -s -w "\nNo SessionToken: %{http_code}" --insecure 'https://localhost:10443/'
curl -o /dev/null -s -w "\nWith SessionToken: %{http_code}" --insecure -H "Cookie: SessionToken=${SESSION_TOKEN}" 'https://localhost:10443'

If we call this script via the scripting API without any further parameters

curl -k -F call=user_info.bash 'https://localhost:10443/scripting' -H 'Cookie: SessionToken=[...]'

the server’s response shows a “401” (Unauthorized) response for the first request and a “200” (Ok) response for the second request:

...
<script code="0">
  <call>user_info.bash</call>
  <stdout>No SessionToken: 401
th SessionToken: 200</stdout>
  <stderr />
</script>
...

A Pattern: The Server as a Client of Itself.#

So what did we do in the last step?

We installed a script user_info.bash. If we call that script via the scripting API, the server executes that script on our behalf. The script sends two request back to the server and the scripting API response reaches us after both curl requests of the script terminated.

| USER: calls `user_info.bash`
|       | SERVER: receives the scripting API request
|       | SERVER: executes `user_info.bash`
|       |         | SCRIPT: curl sends request 1
|       |         |         | SERVER receives the request
|       |         |         * SERVER responds with 401
|       |         | SCRIPT: curl terminates 1st time
|       |         | SCRIPT: curl sends request 2
|       |         |         | SERVER receives the request
|       |         |         * SERVER responds with 200
|       |         | SCRIPT: curl terminates 2nd time
|       |         * SCRIPT: terminates
|       * SERVER: sends response `<script code="0"> ...`
* USER: receives the servers response.

So in the lines in the middle, the server is talking to itself. Actually, this is a very common pattern. This pattern can be used to encapsulate a complex series of transactions into one single command with parameters.