How-to: Warp 10 MultiValue, always easier

Warp 10 allows you to store strings since its very first version. In a string, you can store anything, from space separated numbers to structured JSON, or wrapped Geo Time Series (GTS). Storing multiple values for one timestamp is not a new feature. Warp 10 2.1 brought ingestion simplicity and performance. The latest version brings decoding simplicity.

Real-life example: thermal data acquisition on an engine

All the sensors are connected to a single acquisition device. It means every channel will have the same clock. Exhaust Gas Temperature is sampled every 10ms, intake air temperature every 100ms, oil and water temperature every second.

Every data point will share the same timestamps and geographic position: MultiValue makes sense.

Before Warp 10 2.1

You may store every value in a string. When a value is not present, you can store a special character. Your input format will look like this:

1578408287040000/-4.4144541:48.4421411/120 engine{} '940 820 815 805 x x x'
=1578408287050000/-4.4144541:48.4421411/120 '910 823 811 805.5 24.7 48.1 41'
=1578408287060000/-4.4144541:48.4421411/120 '905.5 821 809 808 x x x'...

Now, do the WarpScript to build one GTS per channel… Believe me, it will take you a lot of time! For each value, you have to parse the string, copy timestamp, position, data in a new GTS defined before. Maintenance of such WarpScript will be hard.

You may do something simpler to decode with a JSON input, but it will cost you more disk space.

That's why very few customers did multivalue before Warp10 2.1…

Warp 10 2.1

You can store lists of values or indexed lists of values. Note, you can store more complex recursive structures with lists of values in each value of the list, but it is not the point of this tutorial.

1578408287040000/-4.4144541:48.4421411/120 engine{} [ 1/940 2/820 3/815 4/805 ]
=1578408287050000/-4.4144541:48.4421411/120 [ 1/910 2/823 3/811 4/805.5 5/24.7 6/48.1 7/41 ]
=1578408287060000/-4.4144541:48.4421411/120 [ 1/905.5 2/821 3/809 4/808 ]

This will be parsed and stored the most efficiently possible. But how to handle it?

When you fetch such data, you will get some cryptic strings. Click execute below and look at the result:

<'
  1578408287040000/-4.4144541:48.4421411/120 engine{} [ 1/940 2/820 3/815 4/805 ]
  =1578408287050000/-4.4144541:48.4421411/120 [ 1/910 2/823 3/811 4/805.5 5/24.7 6/48.1 7/41 ]
  =1578408287060000/-4.4144541:48.4421411/120 [ 1/905.5 2/821 3/809 4/808 ]
'>
PARSE

Internally, Warp 10 stores [ 1/910 2/823 3/811 4/805.5 5/24.7 6/48.1 7/41 ] as a time series, where the first value is at timestamp 1, the second value is at timestamp 2 and so on. This time series has no classname and no labels. It is compressed and occupies 17 bytes in an ISO-8859-1 string.

Wait! There is a mix of long and doubles in the input values! This time series could not be a GTS?

Mixed types inside a time series are allowed, as long as you use ENCODERS, and not GTS. Under the hood, a GTS is just a specific encoder.

<'
  1578408287040000/-4.4144541:48.4421411/120 engine{} [ 1/940 2/820 3/815 4/805 ]
  =1578408287050000/-4.4144541:48.4421411/120 [ 1/910 2/823 3/811 4/805.5 5/24.7 6/48.1 7/41 ]
  =1578408287060000/-4.4144541:48.4421411/120 [ 1/905.5 2/821 3/809 4/808 ]
'>
PARSE
0 GET VALUES 1 GET //read string corresponding to [ 1/910 2/823 3/811 4/805.5 5/24.7 6/48.1 7/41 ]
'iso-8859-1' ->BYTES UNWRAPENCODER

Two WarpScript functions were introduced to decode the simplest multivalues: MVINDEXSPLIT and MVTICKSPLIT. When you use multivalue indexed by a timestamp, MVTICKSPLIT is your best friend:

<'
  1578408287040000/-4.4144541:48.4421411/120 engine{} [ 1/940 2/820 3/815 4/805 ]
  =1578408287050000/-4.4144541:48.4421411/120 [ 1/910 2/823 3/811 4/805.5 5/24.7 6/48.1 7/41 ]
  =1578408287060000/-4.4144541:48.4421411/120 [ 1/905.5 2/821 3/809 4/808 ]
'>
PARSE
{
  1 'egtCylinder1' 
  2 'egtCylinder2'
  3 'egtCylinder3' 
  4 'egtCylinder4' 
  5 'intake' 
  6 'oil' 
  7 'water'
}
MVTICKSPLIT

But again, there is a pitfall: as we don't know exactly the type of data you put for each type, we cannot create a GTS. So, the output of MVINDEXSPLIT and MVTICKSPLIT are encoders.

Warp 10 2.4.0

In this release, we improved the ->GTS function for an easy conversion of encoders to GTS. You just need to specify an output type in the parameters of the function:

{ 'DOUBLE' '~.*{}' } ->GTS will convert every input in GTS of doubles.

If you are 100% confident in your inputs, and you know that all values are represented as doubles (with 940.0 instead of 940 for example), then you can use the simplified signature, where the ->GTS function guesses the type from the first value encountered in the input encoder.

{} ->GTS

After this one line step, you can work with a list of GTS, as you are already used to. In the following example, I do a mean of all exhaust gas temperature. Reducing is straightforward, you know all your data are already synchronized.

<'
  1578408287040000/-4.4144541:48.4421411/120 engine{} [ 1/940 2/820 3/815 4/805 ]
  =1578408287050000/-4.4144541:48.4421411/120 [ 1/910 2/823 3/811 4/805.5 5/24.7 6/48.1 7/41 ]
  =1578408287060000/-4.4144541:48.4421411/120 [ 1/905.5 2/821 3/809 4/808 ]
'>
PARSE
{
  1 'egtCylinder1'
  2 'egtCylinder2'  
  3 'egtCylinder3' 
  4 'egtCylinder4' 
  5 'intake' 
  6 'oil'
  7 'water'
}
MVTICKSPLIT FLATTEN  //decode and rename, flatten the output list
{
  'DOUBLE' '~.*{}' 
} 
->GTS 'temperatureGTSList' STORE // encoders -> gts

// end of multivalue decoding. Easy!
[ $temperatureGTSList [] '~egtCylinder.' filter.byclass ] FILTER 'cylinderTempGTSList' STORE
[ $cylinderTempGTSList [] reducer.mean ] REDUCE

Future versions

Listening to our users is really important. Storing multiple values is something you can do since the very first versions of Warp 10, but before Warp 10 2.1, you had to be a WarpScript expert. With Warp 10 2.1, we worked on an easier way to ingest data. Then we changed WarpScript to decode multivalue easily. There is always room for improvement, so contact us if you have a specific problem.

Takeaways

If you know your data model is subject to evolve, use multivalue with a tick index. You will optimize later if needed. Your input format will look like:

1578408287040000/-4.4144541:48.4421411/120 engine{} [ 1/940 2/820 3/815 4/805 ]
=1578408287050000/-4.4144541:48.4421411/120 [ 1/910 2/823 3/811 4/805.5 5/24.7 ]
=1578408287060000/-4.4144541:48.4421411/120 [ 1/905.5 2/821 3/809 4/808 ]

Use MVTICKSPLIT. A renaming map will help you… And you can add data later.

{
  1 'egtCylinder1'
  2 'egtCylinder2'
  3 'egtCylinder3'
  4 'egtCylinder4' 
  5 'intake' 
  6 'oil' 
  7 'water'
}
MVTICKSPLIT

MVTICKSPLIT outputs are ENCODERS. To convert them to GTS, use ->GTS. If you do not trust the data input (e.g. 0 instead of 0.0 for a double), force the type.

{ 'DOUBLE' '~.*{}' } ->GTS

You're done, with just two functions! Now you can manipulate GTS as you are used to and share it with us on Twitter or Google Group.

Pierre Papin

Electronics engineer, fond of computer science, embedded solution developer.