Warp 10 Lambda functions

Lambda functions with Warp 10. Use the power of WarpScript in simple JSON API deployments.

With Warp 10 it is easy to deploy Lambda functions using the Warp 10 HTTP plugin. Build your REST backend.

The first thing is to fetch a dataset. Here is one: https://www.kaggle.com/ahmedlahlou/accidents-in-france-from-2005-to-2016

Then, you need a Warp 10 instance with the HTTP plugin installed. Do not forget to add this to your Warp 10 configuration file: egress.clients.expose = true so the WarpScript code executed for each HTTP request can access the Warp 10™ Storage Engine.

In the zip archive, there is a caracteristics.csv file. In order to parse it and upload data against a Warp 10 instance, you can use this Gist: https://gist.github.com/Giwi/3e42d7ad1837657d9329faa619d9b664

This script will produce a couple of Geo Time Series (GTS) with "accidents" as class name and a "col" label representing the collision type:

Two vehicles – frontal
Two vehicles – from the rear
Three vehicles and more – in chain
Two vehicles – by the side
Three or more vehicles – multiple collisions
Other collision
Without collision

You normally have 8 GTS ("col" may equals "NA")

First Lambda

Lambdas are deployed as spec files for the HTTP Plugin. They are placed in the http.dir directory defined for the plugin.

<path_to_warp10>/http/last.mc2

We would like to fetch the last n accidents anywhere as a JSON result:

{
  //
  // This first part of the spec file defines when the plugin should execute 'macro'
  //
  
  ## root path
  'path' '/last'
  
  ## allow to get path info
  'prefix' true
  
  ## true: parse payload of a POST url encoded request
  ## false: to parse manually the payload
  'parsePayload' true
  
  //
  // This is the macro which will be called to serve requests to /last/xxx 
  //
  'macro' <% 
    ## save the original request into a variable
    'request' STORE
    
    ## retrieve the path info (after '/last')
    $request 'pathinfo' GET
    
    ## extract each parts of the path info
    '/' SPLIT 'res' STORE
    
    ## evince the first '/'
    $res [ 1 $res SIZE ] SUBLIST 'res' STORE
    
    ## extract the desired count
    $res 0 GET TOLONG 'count' STORE
    
    [ '<YOUR READ TOKEN>' 'accidents' {}  NOW $count -1 * ] FETCH
    MERGE
    $count SHRINK 'gts' STORE
    
    $gts TICKLIST <%
      ## drop the LFLATMAP index
      DROP
      
      ## store timestamp
      't' STORE
      
      ## fetch data from the series
      $gts $t ATTICK 'tick' STORE
      
      {
        'epoch' $t 1000 /
        'date' $t ISO8601

        <% $tick 1 GET ISNaN ! %>
        <% 'lat' $tick 1 GET %> IFT
        
        <% $tick 1 GET ISNaN ! %>
        <% 'long' $tick 2 GET %> IFT
        
        'value' $tick 4 GET
      }
    %> LFLATMAP
    'body' STORE
    
    ## build the HTTP response
    {
      'status' 200
      'body' $body ->JSON
      'headers' { 'Content-Type' 'application/json' }
    }
  %>
}

Now, open a browser and launch http://127.0.0.1:9000/last/10, you should have :

[
  {
    "epoch": 1480930500000,
    "date": "2016-12-05T09:35:00.000000Z",
    "lat": 50.25567997712642,
    "long": 2.7589399740099907,
    "value": 1
  },
  {
    "epoch": 1480930500000,
    "date": "2016-12-05T09:35:00.000000Z",
    "lat": 50.25567997712642,
    "long": 2.7589399740099907,
    "value": 1
  },  ...
]

Used functions
ISO8601, FETCH, LFLATMAP, ->JSON, ATTICK, TICKLIST, SUBLIST, TOLONG, SHRINK, MERGE, ISNaN, IFT

A more complex Lambda

We would like to have the monthly count of accidents for a given year and for a given department as a JSON result:

<path_to_warp10>/http/bydep.mc2

{
  // root path
  'path' '/bydep'

  // allow to get path info
  'prefix' true

  // true: parse payload of a POST url encoded request
  // false: to parse manually the payload
  'parsePayload' true

  'macro' <%
    // save the original request into a variable
    'request' STORE

    // retrieve the path info (after /bydep) 
    $request 'pathinfo' GET  

    // extract each parts of the path info 
    '/' SPLIT 'res' STORE 

    // evince the first '/' 
    $res [ 1 $res SIZE ] SUBLIST 'res' STORE
    
    // map pathinfo
    {} 'pathinfo' STORE 
    [ 'year' 'dep' ]
    <%
      'i' STORE
      'key' STORE 
      $pathinfo { $key $res $i GET } APPEND 'pathinfo' STORE
      $key
    %> LFLATMAP DROP  

    // build time bounds
    [ $pathinfo 'year' GET TOLONG 1 - 12 31 23 59 59 ] TSELEMENTS-> 'start' STORE
    [ $pathinfo 'year' GET TOLONG 12 31 23 59 59 ] TSELEMENTS-> 'end' STORE 
    [ '<YOUR READ TOKEN>' 'accidents' { 'dep' $pathinfo 'dep' GET } $start ISO8601 $end ISO8601 ]  FETCH 'gts' STORE  

    // sum by month   
    [ $gts bucketizer.sum $end 30 d 0 ] BUCKETIZE 
    
    // Fill missing values 
    [ NaN NaN NaN 0 ] FILLVALUE 'gts' STORE   
    
    // sum and merge all   
    [ $gts [] reducer.sum ] REDUCE 0 GET 'gts' STORE   
    
    // build the output 
    $gts TICKLIST <%   
      // drop the LFLATMAP index  
      DROP   
      // store timestamp   
      't' STORE 
      // fetch data from the series 
      $gts $t ATTICK 'tick' STORE  
      {        
      'epoch' $t 1000 /
      'date' $t ISO8601   
      'value' $tick 4 GET 
      }
    %> LFLATMAP   
    'body' STORE   

    // build the HTTP response    
    {     
    'status' 200       
    'body' $body ->JSON 
    'headers' { 'Content-Type' 'application/json' } 
    }
  %>
}

Now, open a browser and launch http://127.0.0.1:9000/bydep/2015/22, you should have :

[
  {
    "epoch": 1107561599000,
    "date": "2005-02-04T23:59:59.000000Z",
    "value": 25
  },
  {
    "epoch": 1110153599000,
    "date": "2005-03-06T23:59:59.000000Z",
    "value": 19
  },
  {
    "epoch": 1112745599000,
    "date": "2005-04-05T23:59:59.000000Z",
    "value": 18
  },  ...
]

Used functions
BUCKETIZE, REDUCE, FILLVALUE, TSELEMENTS->

Using JSON payload on a POST request

As seen, passing parameters by path parameters could be painful. You can parse any payload sent by a POST request:

{
  // root path 
  'path' '/post' 

  // allow to get path info
  'prefix' true 

  // parse manually the payload
  'parsePayload' false 

  'macro' <%   
    // save the original request's payload into a variable 
    'request' STORE

    $request 'payload' GET 'UTF-8' BYTES-> JSON-> 'payload' STORE
    [ $payload 'year' GET 1 - 12 31 23 59 59 ] TSELEMENTS-> 'start' STORE
    [ $payload 'year' GET 12 31 23 59 59 ] TSELEMENTS-> 'end' STORE  
    [
      '<YOUR READ TOKEN>'
      'accidents' { 'dep' $payload 'dep' GET }
      $start ISO8601
      $end ISO8601
    ]  FETCH 'gts' STORE

    [ $gts bucketizer.sum $end 30 d 0 ] BUCKETIZE
    [ NaN NaN NaN 0 ] FILLVALUE  'gts' STORE 
    [ $gts [] reducer.sum ] REDUCE 0 GET 'gts' STORE
    
    $gts TICKLIST <%  
      // drop the LFLATMAP index   
      DROP    
    
      // store timestamp  
      't' STORE      
    
      // fetch data from the series   
      $gts $t ATTICK 'tick' STORE    
    
      {    
        'epoch' $t 1000 / 
        'date' $t ISO8601 
        'value' $tick 4 GET   
      } 
    %> LFLATMAP  
    'body' STORE 
    
    // build the HTTP response   
    {    
      'status' 200  
      'body' $body ->JSON  
      'headers' { 'Content-Type' 'application/json' }  
    } 
  %>
}

curl -i -X POST "Content-Type:application/json" \
(out)  -d '{ "year": 2015, "dep": "29"}' \
(out)  http://localhost:9000/post

By sending this cURL POST request, you obtain the same result as the previous example.

Conclusion

You can, of course, factorize code by using the macro mechanism and/or the WarpFleet Resolver.

As you can see, it is very easy to deploy lambda functions into Warp 10 and expose a JSON API.

Connecting Tableau and Warp 10

Discover how to connect Tableau to Warp 10, an alternative to WarpView

Hosting Web content in Warp 10

Why bother with a dedicated web server when Warp 10 can host your Web content?

n8n & Warp 10 - Automate your time series manipulations

Learn how to connect the Warp 10 platform with n8n, a workflow automation tool. A guest post by Nicolas Steinmetz, Founder of CerenIT

Xavier Marin

Senior Software Engineer