Learn advanced features of Warp 10 scheduler: Runners are an easy way to automate WarpScript, without any new external component.
This article deals with the internal Warp 10 scheduler. If you do not have a job orchestrator like Prefect in your production environment, Warp 10 internal scheduler is the easiest way to launch WarpScript jobs at regular intervals. Discover the basics and some less known features.
| Since version 2.11.0 of Warp 10, a runner can decide when to reschedule itself. Dynamic scheduling is now really easy within WarpScript. Read this article to learn more. |
Runners basics
By default, to run a WarpScript every hour, just place its mc2 file in /opt/warp10/warpscripts/myapp/3600000/myrunner.mc2. The first subdirectory myapp is here to isolate runners easily (choice is up to you, this is not linked to your token .app). The child directory is the scheduling period in milliseconds (3600000 for one hour). The scheduler configuration is grouped into /opt/warp10/etc/conf.d/10-runner.conf.
The Warp 10 scheduler will scan for new .mc2 files the /opt/warp10/warpscripts/ directory every runner.scanperiod (default config is one minute).
Every script is then added to an execution queue. You can set multiple threads to parallelize execution with the runner.nthreads config.
To test runners the easiest way, activate the debug extension to be able to use STDOUT function. Create your own configuration file, for example /opt/warp10/etc/conf.d/99-myconf.conf, and add:
warpscript.extension.debug= io.warp10.script.ext.debug.DebugWarpScriptExtension
Then, create your first runner, for example /opt/warp10/warpscripts/test/3000/myRunner.mc2, and paste this simple WarpScript:
Wait one minute, then tail Warp 10 logs. If you set a systemd unit, it is as simple as sudo journalctl -f -u warp10.service. You should see one line every 3 seconds:
What if the execution time is greater than the period?
The scheduler will schedule the next execution at the end of the script, based on the script start + periodicity. So, the script will run as soon as the previous execution is finished. It will saturate one scheduler thread. You can test that with this WarpScript:
You see in the logs that the next runner is scheduled just after the end of the previous execution.
Hidden variables
Before the runner execution, the scheduler sets a few variables you can use in your code:
$runner.path variable contains the relative script path as a string: "test/3000/myRunner.mc2".
$runner.periodicity variable will contain the period as a long in millisecond: 3000
$runner.scheduledat will contain the scheduled date in milliseconds. At the beginning of the script, if $runner.scheduledat is far from NOW, it means the scheduler threads are too busy to execute the script on time.
Hide sensitive content in your configuration
Runners provide the same mechanism as macros to avoid publishing secrets on your git repos: You can hide tokens or sensitive information inside Warp 10 configuration, with a namespace restriction based on the runner path. For example, add in your runner code:
In the output, you should see -1 for both variables.
Now, in your /opt/warp10/etc/conf.d/99-myconf.conf, add the following line:
myvariable@/test = 42
Then restart Warp 10. Now, myvariable is available for every runner in the test parent directory. Note, you can restrict the definition to only one runner, omitting the period of the path:
myvariable@/test/myRunner = 42
Schedule at a fixed hour
Warp 10 will run every script at startup, then plan the next execution. You can override this behavior with runner.runatstartup configuration. If you want to fix the execution hour, you have to trick the system: schedule a runner every minute, and check if the current time is equal to the hour/minute/day you want.
To make this easy, SenX provides a "cron" like syntax decoder:
This code will return true every day at 22h36 Paris time.
Nonce for runners
Imagine you designed a macro that must be called from a runner, and only from a runner, for security purposes. This macro may be on another instance. Runners can generate a nonce that can be securely decoded by the macro. To enable this function, add another secret AES key in your configuration:
runner.psk = hex:bbed7e59ef540c56a6e57b823d5896b30d9880bd84e45629d29193dafeab886f
Once Warp 10 is restarted, you can use the $runner.nonce variable in your runner code:
$runner.nonce contains system time when the runner is executed, salted with a random 64-bit number, wrapped with the runner.psk key.
It can be decoded by any instance that shares the same runner.psk configuration.
The function RUNNERNONCE allows you to extract the time contained in the nonce.
Coming soon…
In the next Warp 10 version (>1.2.10), the RUNNERIN function will allow to dynamically reschedule a runner from the runner itself. It will allow to speed up or slow down a runner from WarpScript.
To be able to use this function, you will need to generate a token with a capability named runner.reschedule.min.period set to 5000, for example, to be able to reschedule a runner 5s after it starts. Be careful not to saturate your execution thread(s)!
This function can also be used to synchronize a runner on the next hour's start. See RUNNERAT function documentation for some examples.
Take away
If you do not have a job scheduling system, or if you do not wish to install one, Warp 10's own internal scheduler provides a quick way to automate things.
Read more
Les données dynamiques au cœur des Smart cities
Interacting with QuestDB in WarpScript through JDBC
Introducing the Warp 10 Kafka Plugin
Electronics engineer, fond of computer science, embedded solution developer.