Warp 10 scheduler secrets

This article deals with the internal Warp 10 scheduler. If you do not have a job orchestrator like Prefect in your production environment, Warp 10 internal scheduler is the easiest way to launch WarpScript jobs at regular intervals. Discover the basics and some less known features.

Runners basics

By default, to run a WarpScript every hour, just place its mc2 file in /opt/warp10/warpscripts/myapp/3600000/myrunner.mc2. The first subdirectory myapp is here to isolate runners easily (choice is up to you, this is not linked to your token .app). The child directory is the scheduling period in milliseconds (3600000 for one hour). The scheduler configuration is grouped into /opt/warp10/etc/conf.d/10-runner.conf.

The Warp 10 scheduler will scan for new .mc2 files the /opt/warp10/warpscripts/ directory every runner.scanperiod (default config is one minute).

Every script is then added to an execution queue. You can set multiple threads to parallelize execution with the runner.nthreads config.

To test runners the easiest way, activate the debug extension to be able to use STDOUT function. Create your own configuration file, for example /opt/warp10/etc/conf.d/99-myconf.conf, and add:

warpscript.extension.debug= io.warp10.script.ext.debug.DebugWarpScriptExtension

Then, create your first runner, for example /opt/warp10/warpscripts/test/3000/myRunner.mc2, and paste this simple WarpScript:

'hello from Runner' STDOUT

Wait one minute, then tail Warp 10 logs. If you set a systemd unit, it is as simple as sudo journalctl -f -u warp10.service. You should see one line every 3 seconds:

april 04 21:43:33  warp10-standalone.sh[61464]: hello from Runner
april 04 21:43:36  warp10-standalone.sh[61464]: hello from Runner
april 04 21:43:39  warp10-standalone.sh[61464]: hello from Runner
april 04 21:43:42  warp10-standalone.sh[61464]: hello from Runner
april 04 21:43:45  warp10-standalone.sh[61464]: hello from Runner
april 04 21:43:48  warp10-standalone.sh[61464]: hello from Runner

What if the execution time is greater than the period?

The scheduler will schedule the next execution at the end of the script, based on the script start + periodicity. So, the script will run as soon as the previous execution is finished. It will saturate one scheduler thread. You can test that with this WarpScript:

'starting runner at ' NOW ISO8601 + STDOUT
NOW 'start' STORE
<% NOW $start - 4 s < %>
<%  %> // 4 seconds endless loop 
WHILE
'ending runner at ' NOW ISO8601 + STDOUT

You see in the logs that the next runner is scheduled just after the end of the previous execution.

april 04 21:46:58  warp10-standalone.sh[61464]: starting runner at 2022-04-04T19:46:58.212148Z
april 04 21:47:02  warp10-standalone.sh[61464]: ending runner at 2022-04-04T19:47:02.212360Z
april 04 21:47:02  warp10-standalone.sh[61464]: starting runner at 2022-04-04T19:47:02.215421Z
april 04 21:47:06  warp10-standalone.sh[61464]: ending runner at 2022-04-04T19:47:06.215551Z
april 04 21:47:06  warp10-standalone.sh[61464]: starting runner at 2022-04-04T19:47:06.220383Z


Hidden variables

Before the runner execution, the scheduler sets a few variables you can use in your code:

[
  $runner.scheduledat 
  $runner.path
  $runner.periodicity
] ->JSON STDOUT

$runner.path variable contains the relative script path as a string: "test/3000/myRunner.mc2".

$runner.periodicity variable will contain the period as a long in millisecond: 3000

$runner.scheduledat will contain the scheduled date in milliseconds. At the beginning of the script, if $runner.scheduledat is far from NOW, it means the scheduler threads are too busy to execute the script on time.

Hide sensitive content in your configuration

Runners provide the same mechanism as macros to avoid publishing secrets on your git repos: You can hide tokens or sensitive information inside Warp 10 configuration, with a namespace restriction based on the runner path. For example, add in your runner code:

// ${var:defaultvalue}
'myvariable ' ${myvariable:-1} TOSTRING + STDOUT
'anothervariable ' ${anothervariable:-1} TOSTRING + STDOUT

In the output, you should see -1 for both variables.

Now, in your /opt/warp10/etc/conf.d/99-myconf.conf, add the following line:

myvariable@/test = 42

Then restart Warp 10. Now, myvariable is available for every runner in the test parent directory. Note, you can restrict the definition to only one runner, omitting the period of the path:

myvariable@/test/myRunner = 42

Schedule at a fixed hour

Warp 10 will run every script at startup, then plan the next execution. You can override this behavior with runner.runatstartup configuration. If you want to fix the execution hour, you have to trick the system: schedule a runner every minute, and check if the current time is equal to the hour/minute/day you want.

To make this easy, SenX provides a "cron" like syntax decoder:

NOW '36 22 * * *' 'Europe/Paris' @senx/cron/check

This code will return true every day at 22h36 Paris time.

Nonce for runners

Imagine you designed a macro that must be called from a runner, and only from a runner, for security purposes. This macro may be on another instance. Runners can generate a nonce that can be securely decoded by the macro. To enable this function, add another secret AES key in your configuration:

runner.psk = hex:bbed7e59ef540c56a6e57b823d5896b30d9880bd84e45629d29193dafeab886f

Once Warp 10 is restarted, you can use the $runner.nonce variable in your runner code:

$runner.nonce  STDOUT
$runner.nonce RUNNERNONCE ISO8601 STDOUT

$runner.nonce contains system time when the runner is executed, salted with a random 64-bit number, wrapped with the runner.psk key.

It can be decoded by any instance that shares the same runner.psk configuration.

The function RUNNERNONCE allows you to extract the time contained in the nonce.

Coming soon…

In the next Warp 10 version (>1.2.10), the RUNNERIN function will allow to dynamically reschedule a runner from the runner itself. It will allow to speed up or slow down a runner from WarpScript.

To be able to use this function, you will need to generate a token with a capability named runner.reschedule.min.period set to 5000, for example, to be able to reschedule a runner 5s after it starts. Be careful not to saturate your execution thread(s)!

This function can also be used to synchronize a runner on the next hour's start. See RUNNERAT function documentation for some examples.

Take away

If you do not have a job scheduling system, or if you do not wish to install one, Warp 10's own internal scheduler provides a quick way to automate things.

Pierre Papin

Electronics engineer, fond of computer science, embedded solution developer.