Thinking in WarpScript – Detecting a sequence of values

WarpScript is a handy toolkit build for time series manipulation. A common use case could be to detect a sequence of values lasting a given duration

This post describes how to detect in a series of values between 0 and 5, subsequences of value greater or equal to 3 lasting at least 5 seconds.

How to easily manipulate raw binary payloads in WarpScript

First step

As a first step, we need a dataset. Here, we will generate a random GTS containing 300 values between 0 and 5 spaced by one second. Of course, you can use your own.

NEWGTS                       // Create an empty GTS
0 300 <%                     // for(i = 0; i < 300; i++)
  'i' STORE                  // store the current index
  NOW $i 1 s * +             // tick ('now' + 'i seconds')
  NaN NaN NaN                // no geo coordinate
  RAND 5 * ROUND             // generate the random value
  ADDVALUE                   // add the data point to gts
%> FOR                       // close FOR loop macro
'gts' STORE                  // store the result

Read more about:
NEWGTS, STORE, FOR, NOW, RAND, ROUND, ADDVALUE, DROP

The result is a random time series:

Second step

Now, we will build a sliding window that navigates across this series in order to detect 5 seconds of values greater or equal to 3:

[
  $gts // our random series
  <%
    'data' STORE      // store the current sliding window data
    $data 0 GET       // current timestamp
    $data 4 GET 0 GET // current lat
    $data 5 GET 0 GET // current long
    $data 6 GET 0 GET // current elev
    // in our 5 current points, we keep only those >= 3
    // and verify we have 5 in a row
    $data 7 GET <% 3 >= %> FILTERBY SIZE 5 >= // put true or false as value
  %> MACROMAPPER // custom mapper
  
  0 // pre window size
  -5 s // 5 seconds of post window size
  0 // occurrences
] MAP // Sliding window

RANGECOMPACT // keep only changing values
'result' STORE

Learn more about:
GET, FILTERBY, SIZE, MACROMAPPER, MAP, RANGECOMPACT

Here, we have a sequence of booleans.

Now, we will remove false values:

[ $result true mapper.eq 0 0 0 ] MAP 'result' STORE // retain only 'true'

Learn more about:
mapper.eq

Last step

Now, we have our original series and a second boolean series corresponding to subsequences starts:

Last thought

If you want to detect the end of subsequences, you have to invert pre- and post-window sizes of the mapper:

[
  $gts 
  <%
    'data' STORE
    $data 0 GET  // Current timestamp
    $data 4 GET 0 GET // Current lat
    $data 5 GET 0 GET // Current long
    $data 6 GET 0 GET // Current elev
    // in our 5 current points, we keep only >= 3 and verify we have 5 in a row
    $data 7 GET <% 3 >= %> FILTERBY SIZE 5 >=
  %> MACROMAPPER // custom mapper
  -5 s
  0
  0
] MAP

You can find the final code and a running example of this sequence of values here: WarpStudio

Conversions to Apache Arrow format
WarpScript has support for Apache Arrow Format. Discover what this format is with examples using WarpScript with R, Python and Spark.

An AIS data set for the 2022 Ocean Hackathon
As part of the 2022 Ocean Hackathon, SenX provides the contestants with an AIS data set covering more than 2 years and containing 500 billion data poi...

Santa asset tracking and delivery service
Build a Santa tracker from scratch with Warp 10 and Discovery. Technics for asset tracking with a real-time monitoring dashboard.