How-to: create MultiValue within WarpScript

Multivalue was created for easy storage of multiple values in a single Time Series. We already explained how to push values on the update endpoint, how to fetch and decode them. But we never explained how to create them from within WarpScript…

MultiValue Internals

MultiValue ingestion and decoding function is a way to simplify something advanced Warp 10 users already did long before Warp 10 2.0:

• GTS can store strings
• GTS can be wrapped in a string

So you can take a GTS, wrap it, and store the wrapped version as the data point of a GTS:

NEWGTS 'forecast' RENAME
NOW 1 h + NaN NaN NaN 'rainy' ADDVALUE
NOW 2 h + NaN NaN NaN 'cloudy' ADDVALUE
'forecastGTS' STORE

NEWGTS 'forecast evolution' RENAME
NOW -4.41445 48.44214 NaN  $forecastGTS WRAP ADDVALUE

You can generalize to heterogeneous types with the use of encoders rather than GTS, when you need it. That is exactly what Warp 10 does when you ingest multivalues!

1578408287040000/-4.4144541:48.4421411/120 engine{} [ 1/940.0 2/820 3/815 4/805 ]

1/940.0 2/820 3/815 4/805 is an encoder, where timestamp 1 is a DOUBLE value (940.0), timestamp 2 is a LONG value (820), timestamp 3 is a LONG (815), timestamp 4 is a LONG (805).

If you do not use indexes, then everyone has timestamp 0, and you save a few bits.

1578408287040000/-4.4144541:48.4421411/120 engine{} [ 940.0 820 ]

940.0 820 is an encoder, where timestamp 0 is a DOUBLE value (940.0), timestamp 0 is a LONG value (820). No, there is no typo here, this is perfectly legal to store several values with the same timestamp, as long as you do not try to store them in the Warp 10 storage engine.

Multivalue with PARSE

A simple way to create a new multivalue GTS is to use the PARSE function:

<'
  1578408287040000/-4.4144541:48.4421411/120 engine{} [ 1/940 2/820 3/815 4/805 ]
  =1578408287050000/-4.4144541:48.4421411/120 [ 1/910 2/823 3/811 4/805.5 5/24.7 6/48.1 7/41 ]
  =1578408287060000/-4.4144541:48.4421411/120 [ 1/905.5 2/821 3/809 4/808 ]
'>
PARSE

But building a string to parse it later is not performant for millions of data points.

Multivalue with encoders

I will create a WarpScript that does exactly the same, a parsed STRING:

<'
  1578408287040000/-4.4144541:48.4421411/120 engine{} [ 820 940 true ]
'>
PARSE 0 GET

// this is strictly equivalent:
NEWGTS 'engine' RENAME
1578408287040000 -4.4144541 48.4421411 120 NEWENCODER
0 NaN NaN NaN 820 ADDVALUE
0 NaN NaN NaN 940 ADDVALUE
0 NaN NaN NaN true ADDVALUE
WRAPMV ADDVALUE

In detail: NEWENCODER allows adding values from different types. Here I add a LONG, another LONG, and a boolean. Then WRAPMV compresses the encoder to a byte array in the most efficient way. Since Warp 10 2.0, byte array is accepted by ADDVALUE.

For indexed Multivalues, you can do the same way with non-zero timestamps:

<'
  1578408287040000/-4.4144541:48.4421411/120 engine{} [ 1/820 2/940 3/true  ]
'>
PARSE 0 GET

NEWGTS 'engine' RENAME
1578408287040000 -4.4144541 48.4421411 120  NEWENCODER
1 NaN NaN NaN 820 ADDVALUE
2 NaN NaN NaN 940 ADDVALUE
3 NaN NaN NaN true ADDVALUE
WRAPMV ADDVALUE

Takeaways

• You cannot add lists or maps directly to a GTS:

// this will never work
NEWGTS 'mvclass' RENAME
NOW NaN NaN NaN [ 820 940 true ] ADDVALUE

• Multivalues with an index is just a compressed ENCODER.
• Multivalues without an index is just a compressed ENCODER, where all data points have 0 timestamp.
• A GTS can store a byte array. The value of a multivalue GTS is a byte array, represented as iso8859-1 encoded strings.
• You can WRAPRAW, WRAPRAWOPT, but WRAPMV is the most efficient way to compress an ENCODER for multivalue storage.

As long as you understood the process, you can do some pretty complex data models with GTS of Multivalues of GTS of ENCODERS if you need to!

If you feel this could be better, contact us through our contact form or on stack overflow.

Pierre Papin

Electronics engineer, fond of computer science, embedded solution developer.