Runners dynamic scheduling

Warp 10 2.11.0 (released in August 2022) comes with two new functions to reschedule runners from the runner code itself. It is very convenient to program a task at a fixed hour or to change the period up to events. These functions are protected by a capability.

runner.reschedule.min.period capability

As a sysadmin, you do understand the risk to let someone schedule tasks within a small period. If the task lasts longer than its period, it will be rescheduled right after, and monopolize a thread. So, dynamic rescheduling is constrained by a minimum period encrypted in a token you will give to WarpScript developers.

Since 2.11.0, you can use TOKENGEN without the need to activate the extension or define a secret.

First, save the following script as /opt/warp10/tokengen-runnercap.mc2:

// If you want to run this script on exec endpoint, you need to activate
// the tokengen extension and define a good secret
//   warpscript.extension.token = io.warp10.script.ext.token.TokenWarpScriptExtension
//   token.secret = strongsecret
{
  'id' 'TokenWithRunnerCap'
  'type' 'READ'
  'application' 'noapp'
  'owner'  UUID
  'producer' UUID
  'expiry' NOW 10 ADDYEARS
  'labels' { }   
  'attributes' {
    '.cap:runner.reschedule.min.period' '5000'
    '.cap:debug' '' // allow STDOUT or LOGMSG use
  }
} 
//'strongsecret' // only if used this script on exec endpoint, not needed with warp10-standalone.sh tokengen
TOKENGEN

Then, log in as warp10 user, and execute the following command:

/opt/warp10/bin/warp10-standalone.sh tokengen /opt/warp10/tokengen-runnercap.mc2

The output should contain your token. It is time to test it:

"9DzdR8qAoouAZM8F89ZoN3JGBSFXP6jDF_LIHP.DoHBsh0X_gLB8wI27Yon5sKjfBdwIhhJH4yBAGgQII6EQ6.IxcghWeKexDzDoceMTgqRvmqZMx2HPVvRISNkUVOCrE876KXj01hM623Em_d1_s."
CAPADD

"runner.reschedule.min.period" CAPGET
"debug" CAPGET

It should return 5000.

Create a standard 1-minute runner

For an easier understanding of runners' behavior, we will print debug messages to the Warp 10 process log output.

Since Warp 10 3.0, debug is built-in and capability protected. To use STDOUT, you just need a token with the debug capability set to an empty string.

If you still work with Warp 10 2.x, you need to follow these instructions. Enable the debug extension in Warp 10 by adding this line to your personal configurations (/opt/warp10/etc/conf.d/99-personalconfig.conf), and restart Warp 10:

warpscript.extension.debug = io.warp10.script.ext.debug.DebugWarpScriptExtension

Once done, you can save the following WarpScript to a file that will be scheduled every minute: /opt/warp10/warpscripts/test/60000/runnertest.mc2:

"9DzdR8qAoouAZM8F89ZoN3JGBSFXP6jDF_LIHP.DoHBsh0X_gLB8wI27Yon5sKjfBdwIhhJH4yBAGgQII6EQ6.IxcghWeKexDzDoceMTgqRvmqZMx2HPVvRISNkUVOCrE876KXj01hM623Em_d1_s."
CAPADD // token generated earlier, with debug capability

'hello from runner, time = ' 
NOW ISO8601 + 
", execution count = " + 
$runner.execution.count TOSTRING +  // this variable is only available in a runner execution
STDOUT // print to standard output

In another terminal, monitor the logs with tail -f /opt/warp10/logs/warp10.log. You should see your message every minute in the logs:

hello from runner, time = 2022-09-07T12:28:31.717447Z, execution count = 0
hello from runner, time = 2022-09-07T12:29:30.656690Z, execution count = 1
hello from runner, time = 2022-09-07T12:30:30.654385Z, execution count = 2

Basically, your runner is executed:

• When Warp 10 starts
• Every fixed period, defined by the script path (here 60000 ms)

So, it is not obvious for example to trigger a script at 1 o'clock AM every day… Except if you test the current time every minute, and STOP if it is not exactly 1:00, or if you wake up to restart Warp 10 at 1 o'clock.

The period is not perfectly exact too. The system will slowly drift, with a few milliseconds error at each reschedule.

Speed up or slow down scheduling

Imagine Warp 10 is monitoring a device. Every minute, a runner checks several parameters and sends an email if there is a problem.

Of course, the failure can last more than one minute… And you do not want to receive one email per minute for the same problem. RUNNERIN allow to override the period punctually:

// one minute runner
"9DzdR8qAoouAZM8F89ZoN3JGBSFXP6jDF_LIHP.DoHBsh0X_gLB8wI27Yon5sKjfBdwIhhJH4yBAGgQII6EQ6.IxcghWeKexDzDoceMTgqRvmqZMx2HPVvRISNkUVOCrE876KXj01hM623Em_d1_s."
CAPADD

<%
  // check the system behavior, returns true if failure
  RAND 0.8 > 
%>
<%
  // 'runnerB' @alert/sendAlertEmail
  NOW ISO8601 " " + $runner.path +
  ', problem detected, next check in 5 minutes' +  STDOUT
  5 m RUNNERIN
%> 
<% 
  NOW ISO8601 " " + $runner.path +
   ', no problem, keep the default period of ' + 
  $runner.periodicity TOSTRING + 
  STDOUT
%> IFTE

The output is:

2022-09-07T13:56:10.131460Z test/60000/dynamic.mc2, no problem, keep the default period of 60000
2022-09-07T13:57:09.123331Z test/60000/dynamic.mc2, no problem, keep the default period of 60000
2022-09-07T13:58:09.133805Z test/60000/dynamic.mc2, no problem, keep the default period of 60000
2022-09-07T13:59:09.143360Z test/60000/dynamic.mc2, no problem, keep the default period of 60000
2022-09-07T14:00:09.150462Z test/60000/dynamic.mc2, no problem, keep the default period of 60000
2022-09-07T14:01:09.150263Z test/60000/dynamic.mc2, no problem, keep the default period of 60000
2022-09-07T14:02:09.158071Z test/60000/dynamic.mc2, problem detected, next check in 5 minutes
2022-09-07T14:07:09.162971Z test/60000/dynamic.mc2, problem detected, next check in 5 minutes
2022-09-07T14:12:09.170850Z test/60000/dynamic.mc2, problem detected, next check in 5 minutes
2022-09-07T14:17:09.171089Z test/60000/dynamic.mc2, no problem, keep the default period of 60000
2022-09-07T14:18:09.177483Z test/60000/dynamic.mc2, no problem, keep the default period of 60000

By default, the period is one minute and is randomly extended to 5 minutes.

Execute a WarpScript once every Warp 10 restart

Runners will run as soon as the storage is ready. If you do want to log something at each Warp 10 restart, but never after, you can use MAXLONG RUNNERIN to completely disable a runner.

'Warp 10 restarted' STDOUT
MAXLONG RUNNERIN // disable the runner

Postpone next execution to a fixed hour

Since the latest Warp 10 release, the $runner.execution.count variable makes the task to detect the first execution very easy. So we can build a runner that detect the first execution, reschedule itself to 1:00AM, and keep running exactly at 1:00AM the next day.

"9DzdR8qAoouAZM8F89ZoN3JGBSFXP6jDF_LIHP.DoHBsh0X_gLB8wI27Yon5sKjfBdwIhhJH4yBAGgQII6EQ6.IxcghWeKexDzDoceMTgqRvmqZMx2HPVvRISNkUVOCrE876KXj01hM623Em_d1_s."
CAPADD

<%
  NOW 'Europe/Paris' ->TSELEMENTS 0 2 SUBLIST 'Europe/Paris' TSELEMENTS-> 'midnightToday' STORE
  $midnightToday 25 h +   // 1 AM next day
%> 'getNext1AM' STORE

$runner.execution.count 0 ==
<%
  @getNext1AM 'nextRun' STORE
  'runnerA first execution, reschedule to ' $nextRun ISO8601 + STDOUT
  MAXLONG RUNNERIN // override the period defined in path
  $nextRun RUNNERAT
%>
<%
  // 1 AM, do what you need to do...

  // and reschedule next 1 AM:
  @getNext1AM 'nextRun' STORE
  'runnerA done, reschedule to ' $nextRun ISO8601 + STDOUT
  MAXLONG RUNNERIN // override the period defined in path
  $nextRun RUNNERAT
%> IFTE

So far, the only way to do this was to use cron, or a systemd timer, to run a WarpScript with curl. But cron or other schedulers cannot reschedule up to execution result. RUNNERAT brings a native solution, with more flexibility, as you can choose to reschedule anytime.

Note that the previous example does not completely override the period defined in the script path… If the code fails before reaching RUNNERIN and RUNNERAT, the script will be rescheduled 1 minute later. If you want to make sure this does not happen, use TRY function, display the stack trace in the catch macro, and reschedule in the finally macro. This will mimic a cron or a systemd timer.

Priority

So, WarpScript runners' dynamic scheduling depends on:

• the initial period, fixed in the runner path
• RUNNERAT
• RUNNERIN

The Warp 10 standalone scheduler will apply the following rules:

• RUNNERIN overrides the initial period.
• If RUNNERAT is used, the script is rescheduled to the closest in time between RUNNERAT time and "script start + initial period" time.

That's why RUNNERAT alone can speed up a runner, but you need MAXLONG RUNNERIN to slow down a runner.

Conclusion

RUNNERAT and RUNNERIN make dynamic scheduling easy, you do not need to maintain complex code or use third-party schedulers. If you do not know everything about runners (basics, special variables, NONCE…), you can read this previous article.

Pierre Papin

Electronics engineer, fond of computer science, embedded solution developer.