Parse and transform values on the fly using WarpScript macros as they are pushed to your Warp 10 instance. Discover this feature via protobuf serialization.
Release 2.5.0 of Warp 10 brought the possibility to define custom value encoders. Such encoders handle values starting with a colon, interpreting them, and outputting the actual value stored in the Geo Time Series.
Adding a ValueEncoder
was done by writing a Java class and registering it in your Warp 10 instance via a plugin or extension, handy but not very flexible.
Release 2.6.0 builds upon the custom value encoder feature by providing a value encode. It can use a WarpScript macro to parse the input value, thus bringing the missing flexibility.
Enabling the Macro Value Encoder
You can activate the Macro Value Encoder by enabling a special WarpScript extension, which does nothing but registering the encoder.
Then you enable this extension by adding the following line to your Warp 10 configuration and restarting Warp 10:
warpscript.extension.macrovalueencoder = io.warp10.continuum.ingress.MacroValueEncoder$Extension
You can set two options in this extension. The first one will determine the prefix which should be used by the values that the Macro Value Encoder should handle. By default, this prefix is :m:
, but it can be changed using the following configuration key:
macro.value.encoder.prefix
The values handled by the Macro Value Encoder are those of the form:
:PREFIX:MACRO:CONTENT
The macro @MACRO
will be called with CONTENT
as a STRING
input. In order to limit the macros which can be used for parsing the content, you can define a prefix for the macros using the configuration below:
macro.value.encoder.macro.prefix = foo
With the above configuration, CONTENT would be parsed by the macro @foo/MACRO
.
Writing a value encoder macro
As mentioned, the macro doing the actual parsing will be called with the CONTENT
part of the value in the form of a STRING
parameter.
The macro is expected to return a result of type LONG
, DOUBLE
, BOOLEAN
, STRING
or BYTES
(byte array).
The WarpScript execution environment in which the macro will be executed is guaranteed to remain the same for all data points pushed in the same request to the /update
endpoint. This means that you could use variables to pass information from one parsing call to the other.
The execution environment has no access to the Directory
or Store
backend, so functions FETCH
and FIND
will not work.
As the parsing macros are called for each value, try to make them as efficient as possible, use VARS
and ASREGS
for example.
Example
As an example, we will demonstrate how you can use the Macro Value Encoder feature to encode values as a Protobuf structure on the fly on update.
Warp 10 configuration, stored in /opt/warp10/etc/conf.d/99-myconfig.conf
is:
#enables the protobuf extension
warpscript.extension.protobuf = io.warp10.ext.protobuf.ProtobufWarpScriptExtension
#enables the macro value encoder extension
warpscript.extension.macrovalueencoder = io.warp10.continuum.ingress.MacroValueEncoder$Extension
macro.value.encoder.macro.prefix = mve
macro.value.encoder.prefix
is not defined, and defaults to "m". So, with this configuration, :m:demo:
will refer to /opt/warp10/macros/mve/demo.mc2
The macro will use the PROTOC
and ->PB
functions provided by the Protocol Buffers extension. This extension is available on WarpFleet.
The Protobuf structure that our values will conform to is the one below:
message Value { string s = 1; int64 l = 2; float d = 3;}
Note that we voluntarily keep the field names short, so the amount of data pushed in the /update
request remains low.
The macro which will convert the values to the Protobuf structure will receive a STRING
as input. It expects this STRING
to be valid WarpScript code defining a map which may contain any of the s
, l
or d
fields.
The macro will evaluate this STRING
thus producing such a map and will then serialize this map into the above structure. It will return the serialized content, which will be used as the actual value of the data point.
The macro is below, it is available on the sandbox as @mve/demo
:
<'
syntax = 'proto3';
message Value {
string s = 1;
int64 l = 2;
float d = 3;
}
'>
// Compile the proto definition
PROTOC 'proto' STORE
<%
// Evaluate the input STRING
EVAL
// Serialize the map according as a 'Value' instance
!$proto 'Value' ->PB
%>
The PARSE
function can now call :m:demo, for a very efficient PROTOBUF encoding instead of json.
//
// Parse using MacroValueEncoder and macro @mve/demo
//
<'
1000// test{} :m:demo:{ 's' 'string' }
2000// test{} :m:demo:{ 'd' -1.25 }
3000// test{} :m:demo:{ 'l' 128 }
4000// test{} :m:demo:{ 's' 'hello' 'd' 42.0 'l' 100 }
'>
PARSE
You can experiment a full encoding and decoding code via this sandbox snapshot.
Takeaways
The MacroValueEncoder feature introduced in Warp 10 2.6.0 allows you to perform advanced parsing of Geo Time Series values at the time of insertion in the platform.
The possible use cases are endless, from correcting values on the fly to encoding complex structures as you read here using the protocol buffers integration.
Read more
Working with GEOSHAPEs
Upload Geo Time Series from a Warp 10 instance to another
UFO sightings datasets, the truth is out there... Part 2
Co-Founder & Chief Technology Officer