Upload Geo Time Series from a Warp 10 instance to another

How to migrate data from a Warp 10 instance to another in 2 easy steps.

GTS Warp 10

There are two main strategies to sync your data between two or more Warp 10 instances: you can use a datalog or define your own mechanism using WarpScript. Here, I would like to show you how to read high-frequency data stored on an embedded Warp 10, which is hosted on a computer full of sensors for instance, downsample and store it on a remote distributed instance.

In this example, we read all GTS with a one-hour time span and downsample them. Then, we use REXECZ to upload them to the remote instance.

Discover the easiest way to detect motion and to split a Geo Time Series accordingly.

// Define token
'123456' 'localReadToken' STORE
'456789' 'localWriteToken' STORE
'azerty' 'remoteWriteToken' STORE

//
// Define an upload macro
//
<%
  'time' STORE
  $time 'start' STORE
  
  // While we have not reach present time
  <% NOW $start >= %>
  <%
    [ $localReadToken '~.*' {} $start 1 h ] FETCH 'gts' STORE
    
    // re-sample with the mean value every 15 minutes
    [ $gts bucketizer.mean NOW 15 m 0 ] BUCKETIZE
    NONEMPTY
    'outputGTS' STORE

    // if we have data
    <% $outputGTS SIZE 0 > %>
    <%
      // Wrap all the GTS to reduce space
      $outputGTS WRAPRAW 'wrapped' STORE

      // prepare the macro to be executed remotly
      <%
        $data UNWRAP
        <% '+' RENAME %> F LMAP
        $token UPDATE
      %>

      // Replace vars
      {
        'token' $remoteWriteToken
        'data' $wrapped
      }
      DEREF 'remote_macro' STORE

      $remote_macro 'http://remote.warp.10:8080/api/v0/exec' REXECZ 

      // store the new sync point
      [ $localReadToken 'upload.time' {} NOW -1 ] FETCH
      0 GET CLONEEMPTY
      $start NaN NaN NaN true ADDVALUE

      '+' RENAME  // Don't change name but allow UPDATE

      $localWriteToken UPDATE

      $start 1 h + 'start' STORE
    %> IFT
  %> WHILE
  'all updated'
%>
'upload' STORE


// First, check the last sync point
[ $localReadToken 'upload.time' {} NOW -1 ] FETCH 'uploadGTS' STORE

// if this is the first time
<% $uploadGTS SIZE 0 == %>
<%
  // Create a new GTS
  NEWGTS 'upload.time' RENAME

  // and set a very first data point
  0 NaN NaN NaN true ADDVALUE

  $localWriteToken UPDATE
  0 @upload
%>
<%
  // else, we get the last sync time
  $uploadGTS 0 GET LASTTICK @upload
%> IFTE

To go further, refer to this WarpScript Tutorial.
Discover a real-world use case of this data sync technique.

Read more

UFO sightings datasets, the truth is out there... Part 2
Explore UFO sightings datasets and find correlations with US military bases location.
All about the Geo Time Series Model
Geo Times Series are the main data model of Warp 10. It fits most of the needs for time series modeling, with optional location information.
Mobilité et données : un train de retard pour la loi LOM
Ouverture des données dans la Loi d"Orientation des Mobilités (LOM) : Nous prépare-t-elle véritablement à la transformation de la mobilité vers la mul...
© Copyright 2018