Thinking in WarpScript

Learn how WarpScript Macros can make your code more concise, more robust and shareable.

This is another post in our Thinking in WarpScript series. It aims at demystifying macros, a very useful WarpScript construct to factor outcode and loosen the link between your components.

What are macros?

A WarpScript Macro is a collection of WarpScript statements which can be executed directly, or indirectly if passed as an argument to a function. Macros are created very easily by enclosing WarpScript code between <% and %>.

You can create macros inside a WarpScript code fragment and store them in variables to use later in your code. Or you can call macros deployed in your Warp 10 instance or remotely.

We encourage all Warp 10 users to share macros they have created. For this, we have created WarpFleet, a community space where macros can be described and made available.

Macro template

Macros can contain any amount of WarpScript code, from no statement to millions of lines of code. As macros are meant to be shared, it is a good idea to somehow document what each macro is doing. This could be done using comments in your code, but we have a better suggestion. We created the INFO function specifically for this purpose. And as part of the WarpScript best practices, we recommend you use it to document your macros.

This function accepts as a parameter a map which, as described in its documentation, should contain specific keys describing various aspects of your macro. When your macro is called, the INFO function behaves just like DROP. It means that it simply discards the map on the top of the stack. But if INFOMODE was called before your macro was called, the INFO function will stop the macro execution and leave the map on the stack, thus making it available to the caller.

We have built several tools which can interpret the map left by INFO. The whole WarpScript reference documentation is built from such info maps. The pages describing functions and macros in WarpFleet are also built from those maps.

So, we highly recommend you use the following template when creating macros. This template can be filled automatically when you are using VS Code with the WarpScript plugin by simply typing macro and hitting enter at the template suggestion.

{
  'name' ' '   // Description of what the macro is doing. 
  'desc'        // Name of your macro.
  <'
  // This is assumed to be formatted using MarkDown
  '>
  // Signatures of the macro. Each signature is a list containing
  // itself two lists, one for the expected input, and one for the
  // output created by the macro.
  // Each of the input and output list contains the stack levels  
  // starting with the top of the stack.
  // Each element of those lists is a STRING with the format name:type
  'sig' [ [ [   ] [   ] ] ] 
  // Description of each named parameter from the input and output lists of
  // the signatures.
  'params' {
    // Signature params description
  }
  'examples' [
    <'

    '>
  ]
} 'info' STORE

<%
  !$info INFO
  SAVE 'context' STORE
  <%
    // Code of the actual macro
      
  %>
  <% // catch any exception
    RETHROW
  %>
  <% // finally, restore the context
    $context RESTORE
  %> TRY
%>
'macro' STORE

// Unit tests

$macro

Context

WarpScript code can use variables, created using the STORE function and accessed using either LOAD or the $name syntax. When creating a macro, you may want to use variables too. But you don't know what variables the calling code has already used, so it might be difficult to avoid collisions by simply using cryptic names.

In order to simplify things, the macro template above uses the SAVE and RESTORE functions. These functions manipulate the context of the WarpScript execution environment which contains all variables, registers, and functions set by the code. A call to SAVE will take a photograph of the context at the time of the call and the call to RESTORE will, as its name implies, restore it. Therefore, between the calls to SAVE and RESTORE, variables and registers can be used freely, any changes will be reverted at the time RESTORE is called. In the template, the context is stored in a variable context, so this variable should not be reused until the RESTORE, otherwise, the context would be lost!

The template also uses the TRY function to ensure that the call to RESTORE does indeed occur.

Learn more about how to use variables in macros

Dealing with parameters

Some macros do their things without requiring any input, but others may need to have some parameters passed to them. As for function calls, parameters should be pushed onto the WarpScript stack prior to calling the macro.

Within the macro, the easiest and most understandable thing to do is to store those parameters in variables right after the call to SAVE. Assuming a macro accepts four parameters as input, the following code will store them in variables 'A', 'B', 'C', and 'D' with a single call to STORE.

[ 'A' 'B' 'C' 'D' ] STORE

In the rest of the macro code, those parameters can be accessed using $A, $B, $C or $D.

Understanding bindings

When creating a macro, the WarpScript execution engine builds a list of statements. Those statements can be data such as strings, numbers, or composite structures such as lists or maps, but also functions and other macros.

The statements added to the macro are those that exist at the time they are added. It means that, for example, if a function added to a macro is later redefined, the function which will be called when the macro is executed is the one that existed before that redefinition. This is what is called early binding.

<%
  NOW
%> 'macro' STORE
<% 42 %> 'NOW' DEF

// Will emit the current time because the original function NOW was added
// to the macro prior to its redefinition
@macro

In contrast, if you dereference a variable using $name in a macro, the macro will really contain 'name' LOAD which will be evaluated at the time the macro is executed, thus leading to the loading of the current value contained in the symbol name at that time. This is called late binding.

42 'a' STORE
<% $a %> 'macro' STORE
43 'a' STORE
@macro // will emit 43 as 'a' is loaded at macro execution time which is after the update of 'a' to 43

If you want to use late binding for function calls, or for that matter for any WarpScript code, you can use the EVAL function in your macro with a parameter which is a STRING containing the WarpScript code to run. This code will be executed when the macro is executed, and therefore will call the latest redefinition of the functions.

<%
  'NOW' EVAL
%> 'macro' STORE
<% 42 %> 'NOW' DEF
@macro // Will emit 42 as the function NOW used is the one redefined after the macro was created

If you want to use early binding for variable dereference, you must use the syntax !$name which will insert into the macro the value associated with variable 'name' at the time of the macro creation.

42 'a' STORE
<% !$a %> 'macro' STORE
43 'a' STORE
@macro // will emit 42 as 'a' is loaded at macro creation time which is before the update of 'a' to 43

Calling macros

Macros can be defined anywhere in WarpScript code and serve as parameters to some functions, they can also be stored in variables and called using the @ notation, a macro stored in variable foo can be called using @foo.

Macros can also be deployed in jar files in the class path, in a directory called a macro repository, or on remote servers accessed via the WarpFleet resolver. The calling syntax is similar, a macro call @foo/bar will first check for the existence of a variable $foo/bar, then will check if a file exists under foo/bar.mc2 in the directory defined as the macro repository, then if a resource foo/bar.mc2 exists in the class path and lastly if a resource foo/bar.mc2 exists as a subpath of the configured WarpFleet repositories.

Macro Config

With the possibility to distribute macros via the WarpFleet mechanism, it became necessary to be able to dynamically configure macros. For example, because a particular macro needs a Warp 10 token to read or store Geo Time Series.

Starting with Warp 10 2.0.3, two functions have been added to access configuration keys from within macros. Those functions are MACROCONFIG and MACROCONFIGDEFAULT. They can only be used from macros being loaded from the macro repository, the class path, or via the WarpFleet Resolver. They cannot be used from macros stored in WarpScript variables.

When a macro uses one of those functions, it can access keys suffixed with the macro path or any of its prefixes. This allows to define the needed information in the Warp 10 configuration file and have the macro access them. The suffix cannot be changed by the macro, so this provides a certain level of security and isolation. Nevertheless, when using the WarpFleet Resolver, care must be taken to only add trusted sources. Otherwise, rogue repositories could expose macros that could shadow legitimate ones (by using the same name) and possibly access sensitive information meant to be shared only with those legitimate macros.

Speeding up macros

Macro execution is rather fast, the WarpScript execution engine is able to process several 10s of millions of statements every second. Nevertheless, some operations are costlier than others, and among those are the variable accesses, whether for reading or for writing.

In order to improve performance when accessing variables, WarpScript has the notion of registers which offer a limited number of slots (typically 256) for storage, accessed via a numerical index. Instead of using variables, one can use registers, using POPRx and PUSHRx instead of STORE and LOAD. But using registers instead of variables reduces code readability. So WarpScript has two functions, VARS and ASREGS which can be used to attempt to perform the automatic transformation of variable uses into register accesses.

While those two functions usually do a pretty good job there are some cases where ASREGS cannot determine how to transform the variable accesses into register accesses. Among those are the following:

// List based STORE
// ASREGS cannot transform those because the list is not known at macro creation time
[ 'A' 'B' 'C' ] STORE
// Non explicit variable names
1 TOSTRING STORE 
// ASREGS sees the parameter to STORE as being a function call
$a STORE // ASREGS cannot determine what is in $a

These cases can usually be avoided if you know you want to use ASREGS on the macro being defined.

Compiling macros

Lastly, for über performance of your macros, SenX provides a commercial WarpScript extension which can compile macros into Java bytecode for an average performance gain of 75%. You can reach our sales department for more information.