Upload Geo Time Series from a Warp 10 instance to another

How to migrate data from a Warp 10 instance to another in 2 easy steps.

There are two main strategies to sync your data between two or more Warp 10 instances: you can use a datalog or define your own mechanism using WarpScript. Here, I would like to show you how to read high-frequency data stored on an embedded Warp 10, which is hosted on a computer full of sensors for instance, downsample and store it on a remote distributed instance.

In this example, we read all GTS with a one-hour time span and downsample them. Then, we use REXECZ to upload them to the remote instance.

Discover the easiest way to detect motion and to split a Geo Time Series accordingly.


// Define token
'123456' 'localReadToken' STORE
'456789' 'localWriteToken' STORE
'azerty' 'remoteWriteToken' STORE

//
// Define an upload macro
//
<%
  'time' STORE
  $time 'start' STORE
  
  // While we have not reach present time
  <% NOW $start >= %>
  <%
    [ $localReadToken '~.*' {} $start 1 h ] FETCH 'gts' STORE
    
    // re-sample with the mean value every 15 minutes
    [ $gts bucketizer.mean NOW 15 m 0 ] BUCKETIZE
    NONEMPTY
    'outputGTS' STORE

    // if we have data
    <% $outputGTS SIZE 0 > %>
    <%
      // Wrap all the GTS to reduce space
      $outputGTS WRAPRAW 'wrapped' STORE

      // prepare the macro to be executed remotly
      <%
        $data UNWRAP
        <% '+' RENAME %> F LMAP
        $token UPDATE
      %>

      // Replace vars
      {
        'token' $remoteWriteToken
        'data' $wrapped
      }
      DEREF 'remote_macro' STORE

      $remote_macro 'http://remote.warp.10:8080/api/v0/exec' REXECZ 

      // store the new sync point
      [ $localReadToken 'upload.time' {} NOW -1 ] FETCH
      0 GET CLONEEMPTY
      $start NaN NaN NaN true ADDVALUE

      '+' RENAME  // Don't change name but allow UPDATE

      $localWriteToken UPDATE

      $start 1 h + 'start' STORE
    %> IFT
  %> WHILE
  'all updated'
%>
'upload' STORE


// First, check the last sync point
[ $localReadToken 'upload.time' {} NOW -1 ] FETCH 'uploadGTS' STORE

// if this is the first time
<% $uploadGTS SIZE 0 == %>
<%
  // Create a new GTS
  NEWGTS 'upload.time' RENAME

  // and set a very first data point
  0 NaN NaN NaN true ADDVALUE

  $localWriteToken UPDATE
  0 @upload
%>
<%
  // else, we get the last sync time
  $uploadGTS 0 GET LASTTICK @upload
%> IFTE