Macro Value Encoder FTW

Release 2.5.0 of Warp 10 brought the possibility to define custom value encoders. Such encoders handle values starting with a colon, interpreting them, and outputting the actual value stored in the Geo Time Series.

Adding a ValueEncoder was done by writing a Java class and registering it in your Warp 10 instance via a plugin or extension, handy but not very flexible.

Release 2.6.0 builds upon the custom value encoder feature by providing a value encode. It can use a WarpScript macro to parse the input value, thus bringing the missing flexibility.

Enabling the Macro Value Encoder

You can activate the Macro Value Encoder by enabling a special WarpScript extension, which does nothing but registering the encoder.

Then you enable this extension by adding the following line to your Warp 10 configuration and restarting Warp 10:

warpscript.extension.macrovalueencoder = io.warp10.continuum.ingress.MacroValueEncoder$Extension

You can set two options in this extension. The first one will determine the prefix which should be used by the values that the Macro Value Encoder should handle. By default, this prefix is :m:, but it can be changed using the following configuration key:

macro.value.encoder.prefix

The values handled by the Macro Value Encoder are those of the form:

:PREFIX:MACRO:CONTENT

The macro @MACRO will be called with CONTENT as a STRING input. In order to limit the macros which can be used for parsing the content, you can define a prefix for the macros using the configuration below:

macro.value.encoder.macro.prefix = foo

With the above configuration, CONTENT would be parsed by the macro @foo/MACRO.

Writing a value encoder macro

As mentioned, the macro doing the actual parsing will be called with the CONTENT part of the value in the form of a STRING parameter.

The macro is expected to return a result of type LONG, DOUBLE, BOOLEAN, STRING or BYTES (byte array).

The WarpScript execution environment in which the macro will be executed is guaranteed to remain the same for all data points pushed in the same request to the /update endpoint. This means that you could use variables to pass information from one parsing call to the other.

The execution environment has no access to the Directory or Store backend, so functions FETCH and FIND will not work.

As the parsing macros are called for each value, try to make them as efficient as possible, use VARS and ASREGS for example.

Example

As an example, we will demonstrate how you can use the Macro Value Encoder feature to encode values as a Protobuf structure on the fly on update.

Warp 10 configuration, stored in /opt/warp10/etc/conf.d/99-myconfig.conf is:

#enables the protobuf extension
warpscript.extension.protobuf = io.warp10.ext.protobuf.ProtobufWarpScriptExtension

#enables the macro value encoder extension
warpscript.extension.macrovalueencoder = io.warp10.continuum.ingress.MacroValueEncoder$Extension
macro.value.encoder.macro.prefix = mve

macro.value.encoder.prefix is not defined, and defaults to "m". So, with this configuration, :m:demo: will refer to /opt/warp10/macros/mve/demo.mc2

The macro will use the PROTOC and ->PB functions provided by the Protocol Buffers extension. This extension is available on WarpFleet.

The Protobuf structure that our values will conform to is the one below:

message Value {  string s = 1;  int64 l = 2;  float d = 3;}

Note that we voluntarily keep the field names short, so the amount of data pushed in the /update request remains low.

The macro which will convert the values to the Protobuf structure will receive a STRING as input. It expects this STRING to be valid WarpScript code defining a map which may contain any of the s, l or d fields.

The macro will evaluate this STRING thus producing such a map and will then serialize this map into the above structure. It will return the serialized content, which will be used as the actual value of the data point.

The macro is below, it is available on the sandbox as @mve/demo:

<'
  syntax = 'proto3';
  message Value {
    string s = 1;
    int64 l = 2;
    float d = 3;
  }
'>

// Compile the proto definition
PROTOC 'proto' STORE

<%
  // Evaluate the input STRING
  EVAL
  
  // Serialize the map according as a 'Value' instance
  !$proto 'Value' ->PB
%>

The PARSE function can now call :m:demo, for a very efficient PROTOBUF encoding instead of json.

//
// Parse using MacroValueEncoder and macro @mve/demo
//
<'
  1000// test{} :m:demo:{ 's' 'string' }
  2000// test{} :m:demo:{ 'd' -1.25 }
  3000// test{} :m:demo:{ 'l' 128 }
  4000// test{} :m:demo:{ 's' 'hello' 'd' 42.0 'l' 100 }
'>
PARSE

You can experiment a full encoding and decoding code via this sandbox snapshot.

Takeaways

The MacroValueEncoder feature introduced in Warp 10 2.6.0 allows you to perform advanced parsing of Geo Time Series values at the time of insertion in the platform.

The possible use cases are endless, from correcting values on the fly to encoding complex structures as you read here using the protocol buffers integration.

Mathias Herberts

Co-Founder & Chief Technology Officer