How to migrate data from a Warp 10 instance to another in 2 easy steps.
There are two main strategies to sync your data between two or more Warp 10 instances: you can use a datalog or define your own mechanism using WarpScript. Here, I would like to show you how to read high-frequency data stored on an embedded Warp 10, which is hosted on a computer full of sensors for instance, downsample and store it on a remote distributed instance.
In this example, we read all GTS with a one-hour time span and downsample them. Then, we use REXECZ to upload them to the remote instance.
Discover the easiest way to detect motion and to split a Geo Time Series accordingly. |
// Define token
'123456' 'localReadToken' STORE
'456789' 'localWriteToken' STORE
'azerty' 'remoteWriteToken' STORE
//
// Define an upload macro
//
<%
'time' STORE
$time 'start' STORE
// While we have not reach present time
<% NOW $start >= %>
<%
[ $localReadToken '~.*' {} $start 1 h ] FETCH 'gts' STORE
// re-sample with the mean value every 15 minutes
[ $gts bucketizer.mean NOW 15 m 0 ] BUCKETIZE
NONEMPTY
'outputGTS' STORE
// if we have data
<% $outputGTS SIZE 0 > %>
<%
// Wrap all the GTS to reduce space
$outputGTS WRAPRAW 'wrapped' STORE
// prepare the macro to be executed remotly
<%
$data UNWRAP
<% '+' RENAME %> F LMAP
$token UPDATE
%>
// Replace vars
{
'token' $remoteWriteToken
'data' $wrapped
}
DEREF 'remote_macro' STORE
$remote_macro 'http://remote.warp.10:8080/api/v0/exec' REXECZ
// store the new sync point
[ $localReadToken 'upload.time' {} NOW -1 ] FETCH
0 GET CLONEEMPTY
$start NaN NaN NaN true ADDVALUE
'+' RENAME // Don't change name but allow UPDATE
$localWriteToken UPDATE
$start 1 h + 'start' STORE
%> IFT
%> WHILE
'all updated'
%>
'upload' STORE
// First, check the last sync point
[ $localReadToken 'upload.time' {} NOW -1 ] FETCH 'uploadGTS' STORE
// if this is the first time
<% $uploadGTS SIZE 0 == %>
<%
// Create a new GTS
NEWGTS 'upload.time' RENAME
// and set a very first data point
0 NaN NaN NaN true ADDVALUE
$localWriteToken UPDATE
0 @upload
%>
<%
// else, we get the last sync time
$uploadGTS 0 GET LASTTICK @upload
%> IFTE
To go further, refer to this WarpScript Tutorial.
Discover a real-world use case of this data sync technique.
Read more
Working with GEOSHAPEs: code contest!
Conversions to Apache Arrow format
UFO sightings datasets, the truth is out there... Part 2
Senior Software Engineer