Howto: create MultiValue within WarpScript

Creation of multivalue GTS from within WarpScript needs a good understanding of how multivalue works in Warp 10.

Howto: create MultiValue within WarpScript

Multivalue was created for easy storage of multiple values in a single Time Series. We already explained how to push values on the update endpoint, how to fetch and decode them. But we never explained how to create them from within WarpScript...

Learn more about WarpScript.

MultiValue Internals

MultiValue ingestion and decoding function is a way to simplify something advanced Warp 10 users already did long before Warp 10 2.0:

  • GTS can store strings
  • GTS can be wrapped in a string

So you can take a GTS, wrap it, and store the wrapped version as the datapoint of a GTS:

You can generalize to heterogeneous types with the use of encoders rather than GTS, when you need it. That is exactly what Warp 10 does when you ingest multivalues!

1578408287040000/-4.4144541:48.4421411/120 engine{} [ 1/940.0 2/820 3/815 4/805 ]

1/940.0 2/820 3/815 4/805 is an encoder, where timestamp 1 is a DOUBLE value (940.0), timestamp 2 is a LONG value (820), timestamp 3 is a LONG (815), timestamp 4 is a LONG (805).

If you do not use indexes, then everyone has timestamp 0, and you save a few bits.

1578408287040000/-4.4144541:48.4421411/120 engine{} [ 940.0 820 ]

940.0 820 is an encoder, where timestamp 0 is a DOUBLE value (940.0), timestamp 0 is a LONG value (820). No, there is no typo here, this is perfectly legal to store several values with the same timestamp, as long as you do not try to store them in the Warp 10 storage engine.

How to use multivalue, from input format to decoding with WarpScript.

Multivalue with PARSE

A simple way to create a new multivalue GTS is to use the PARSE function:

<'
1578408287040000/-4.4144541:48.4421411/120 engine{} [ 1/940 2/820 3/815 4/805 ]
=1578408287050000/-4.4144541:48.4421411/120 [ 1/910 2/823 3/811 4/805.5 5/24.7 6/48.1 7/41 ]
=1578408287060000/-4.4144541:48.4421411/120 [ 1/905.5 2/821 3/809 4/808 ]
'>
PARSE

But building a string to parse it later is not performant for millions of data points.

Multivalue with encoders

I will create a WarpScript that does exactly the same a parsed STRING:

In detail: NEWENCODER allows adding values from different types. Here I add a LONG, another LONG, and a boolean. Then WRAPMV compresses the encoder to a byte array in the most efficient way. Since Warp 10 2.0, byte array is accepted by ADDVALUE.

For indexed Multivalues, you can do the same way with non zero timestamps:

Takeaways

  • You cannot add lists or maps directly to a GTS:
// this will never work
NEWGTS
'mvclass' RENAME
NOW NaN NaN NaN [ 820 940 true ] ADDVALUE
  • Multivalues with an index is just a compressed ENCODER.
  • Multivalues without an index is just a compressed ENCODER, where all data points have 0 timestamp.
  • A GTS can store a byte array. The value of a multivalue GTS is a byte array, represented as iso8859-1 encoded strings.
  • You can WRAPRAW, WRAPRAWOPT, but WRAPMV is the most efficient way to compress an ENCODER for multivalue storage.

As long as you understood the process, you can do some pretty complex data models with GTS of Multivalues of GTS of ENCODERS if you need to!

If you feel this could be better, contact us through our contact form or on stack overflow.