Upload Geo Time Series from a Warp 10 instance to another

How to migrate data from a Warp 10 instance to another in 2 easy steps.

GTS Warp 10

There are two main strategies to sync your data between two or more Warp 10 instances: you can use a datalog or define your own mechanism using WarpScript. Here, I would like to show you how to read high-frequency data stored on an embedded Warp 10, which is hosted on a computer full of sensors for instance, downsample and store it on a remote distributed instance.

In this example, we read all GTS with a one-hour time span and downsample them. Then, we use REXECZ to upload them to the remote instance.

Discover the easiest way to detect motion and to split a Geo Time Series accordingly.

// Define token
'123456' 'localReadToken' STORE
'456789' 'localWriteToken' STORE
'azerty' 'remoteWriteToken' STORE

//
// Define an upload macro
//
<%
  'time' STORE
  $time 'start' STORE
  
  // While we have not reach present time
  <% NOW $start >= %>
  <%
    [ $localReadToken '~.*' {} $start 1 h ] FETCH 'gts' STORE
    
    // re-sample with the mean value every 15 minutes
    [ $gts bucketizer.mean NOW 15 m 0 ] BUCKETIZE
    NONEMPTY
    'outputGTS' STORE

    // if we have data
    <% $outputGTS SIZE 0 > %>
    <%
      // Wrap all the GTS to reduce space
      $outputGTS WRAPRAW 'wrapped' STORE

      // prepare the macro to be executed remotly
      <%
        $data UNWRAP
        <% '+' RENAME %> F LMAP
        $token UPDATE
      %>

      // Replace vars
      {
        'token' $remoteWriteToken
        'data' $wrapped
      }
      DEREF 'remote_macro' STORE

      $remote_macro 'http://remote.warp.10:8080/api/v0/exec' REXECZ 

      // store the new sync point
      [ $localReadToken 'upload.time' {} NOW -1 ] FETCH
      0 GET CLONEEMPTY
      $start NaN NaN NaN true ADDVALUE

      '+' RENAME  // Don't change name but allow UPDATE

      $localWriteToken UPDATE

      $start 1 h + 'start' STORE
    %> IFT
  %> WHILE
  'all updated'
%>
'upload' STORE


// First, check the last sync point
[ $localReadToken 'upload.time' {} NOW -1 ] FETCH 'uploadGTS' STORE

// if this is the first time
<% $uploadGTS SIZE 0 == %>
<%
  // Create a new GTS
  NEWGTS 'upload.time' RENAME

  // and set a very first data point
  0 NaN NaN NaN true ADDVALUE

  $localWriteToken UPDATE
  0 @upload
%>
<%
  // else, we get the last sync time
  $uploadGTS 0 GET LASTTICK @upload
%> IFTE

To go further, refer to this WarpScript Tutorial.
Discover a real-world use case of this data sync technique.

Read more

Santa asset tracking and delivery service
Build a Santa tracker from scratch with Warp 10 and Discovery. Technics for asset tracking with a real-time monitoring dashboard.
12 tips to apply sliding window algorithms like an expert
We present 12 simple tips to make full use of the powerful MAP framework for manipulating time series with sliding window algorithms.
Les données dynamiques au cœur des Smart cities
Le développement des Smart Cities met en évidence la nécessité de construire des infrastructures de données neutres, performantes et sécurisées. Basée...
© Copyright 2018