Upload Geo Time Series from a Warp 10™ instance to another

GTS Warp 10

There are two main strategies to sync your data between two or more Warp 10™ instances: you can use a datalog or define your own mechanism using WarpScript™. Here, I would like to show you how to read high-frequency data stored on an embedded Warp 10, which is hosted on a computer full of sensors for instance, downsample and store it on a remote distributed instance.

In this example, we read all GTS with a one hour time span and downsample them. Then, we use REXECZ to upload them to the remote instance.

'123456' 'localReadToken' STORE
'456789' 'localWriteToken' STORE
'azerty' 'remoteWriteToken' STORE

// Define an upload macro
<%
  'time' STORE
  $time 'start' STORE
  // While we have not reach present time
  <% NOW $start >= %>
  <%
    [ $localReadToken '~.*' {} $start 1 h ] FETCH 'gts' STORE
    // re-sample with the mean value every 15 minutes
    [ $gts bucketizer.mean NOW 15 m 0 ] BUCKETIZE
    'outputGTS' STORE
    // if we have data
    <%  $outputGTS SIZE 0 > %>
    <%
      // build a snapshot of the stack
      $outputGTS  SNAPSHOT 'wrapped' STORE
      // prepare the remote WarpScript
      $wrapped " <%25 DROP DUP NAME '%2B' + RENAME %25> LMAP '" + $remoteWriteToken + "' UPDATE" + 
      'http://remote.warp.10:8080/api/v0/exec' 
      REXECZ 
      // store the new sync point
      [ $localReadToken 'upload.time' {} NOW -1 ] FETCH 
      0 GET CLONEEMPTY
      $start NaN NaN NaN true ADDVALUE
      '%2B' RENAME  // Don't change name but allow UPDATE
      $localWriteToken UPDATE
      $start 1 h + 'start' STORE
    %> IFT     
  %>  WHILE
    'all updated'
%> 'upload' STORE

// First, check the last sync point
[ $localReadToken 'upload.time' {} NOW -1 ] FETCH 'uploadGTS' STORE

// if this is the first time
<% $uploadGTS SIZE 0 == %>
<%
  // Create a new GTS
  NEWGTS 'upload.time' RENAME
  // and set a very first data point
  0 NaN NaN NaN true ADDVALUE
  $localWriteToken UPDATE
  0 @upload
%>
<%
  // else, we get the last sync time
  $uploadGTS 0 GET LASTTICK @upload
%> IFTE 

To go further, refer to this WarpScript™ Tutorial.

Share

Leave a Reply

Your email address will not be published. Required fields are marked *