Warp 10™ Lambda functions

With Warp 10™ it is easy to deploy Lambda functions using the Warp 10 HTTP plugin.

The first thing is to fetch a dataset. Here is one: https://www.kaggle.com/ahmedlahlou/accidents-in-france-from-2005-to-2016

Then, you need a Warp 10™ instance with the HTTP plugin installed. Do not forget to add this to your Warp 10™ conf file: egress.clients.expose = true so the WarpScript™ code executed for each HTTP request can access the Warp 10™ Storage Engine.

In the zip archive, there is a caracteristics.csv file. In order to parse it and upload data against a Warp 10™ instance, you can use this Gist: https://gist.github.com/Giwi/3e42d7ad1837657d9329faa619d9b664

This script will produce a couple of Geo Time Series™ (GTS) with "accidents" as class name and a "col" label representing the collision type:

  1. Two vehicles - frontal
  2. Two vehicles - from the rear
  3. Two vehicles - by the side
  4. Three vehicles and more - in chain
  5. Three or more vehicles - multiple collisions
  6. Other collision
  7. Without collision

You normally have 8 GTS ("col" may equals "NA")

First Lambda

Lambdas are deployed as spec files for the HTTP Plugin. They are placed in the http.dir directory defined for the plugin.

<path_to_warp10>/http/last.mc2

We would like to fetch the last n accidents anywhere as a JSON result:

{
  //
  // This first part of the spec file defines when the plugin should execute 'macro'
  //
  ## root path
  'path' '/last'
  ## allow to get path info
  'prefix' true
  ## true: parse payload of a POST url encoded request
  ## false: to parse manually the payload
  'parsePayload' true   

  //
  // This is the macro which will be called to serve requests to /last/xxx
  //
  'macro' <%
    ## save the original request into a variable
    'request' STORE 
    ## retrieve the path info (after '/last')
    $request 'pathinfo' GET 
    ## extract each parts of the path info
    '/' SPLIT 'res' STORE 
    ## evince the first '/'
    $res [ 1 $res SIZE ] SUBLIST 'res' STORE
    ## extract the desired count
    $res 0 GET TOLONG 'count' STORE
    [
        '<YOUR READ TOKEN>'
        'accidents' {}
        NOW $count -1 *
    ]  FETCH MERGE $count SHRINK 
    'gts' STORE
    $gts TICKLIST <%
      ## drop the LFLATMAP index
      DROP 
      ## store timestamp
      't' STORE
      ## fetch data from the series
      $gts $t ATTICK 'tick' STORE 
      { 
        'epoch' $t 1000 /
        'date' $t ISO8601
        <% $tick 1 GET ISNaN ! %> <% 'lat' $tick 1 GET %> IFT
        <% $tick 1 GET ISNaN ! %> <% 'long' $tick 2 GET %> IFT
        'value' $tick 4 GET
      }
    %> LFLATMAP
    'body' STORE
    ## build the HTTP response
    { 
      'status' 200  
      'body' $body ->JSON
      'headers' { 'Content-Type' 'application/json' }
    }
  %>
}

Now, open a browser and launch http://127.0.0.1:9000/last/10, you should have :

[
   {
      "epoch":1480930500000,
      "date":"2016-12-05T09:35:00.000000Z",
      "lat":50.25567997712642,
      "long":2.7589399740099907,
      "value":1
   },
   {
      "epoch":1480930500000,
      "date":"2016-12-05T09:35:00.000000Z",
      "lat":50.25567997712642,
      "long":2.7589399740099907,
      "value":1
   },
  ...
]

Used functions

ISO8601, FETCH, LFLATMAP, ->JSON, ATTICK, TICKLIST, SUBLIST, TOLONG, SHRINK, MERGE, ISNaN, IFT

A more complex Lambda

We would like to have the monthly count of accidents for a given year and for a given department as a JSON result:

<path_to_warp10>/http/bydep.mc2

{
  // root path
  'path' '/bydep'
  // allow to get path info
  'prefix' true
  // true: parse payload of a POST url encoded request
  // false: to parse manually the payload
  'parsePayload' true   
  'macro' <%
    // save the original request into a variable
    'request' STORE 
    // retrieve the path info (after /bydep)
    $request 'pathinfo' GET 
    // extract each parts of the path info
    '/' SPLIT 'res' STORE 
    // evince the first '/'
    $res [ 1 $res SIZE ] SUBLIST 'res' STORE
    // map pathinfo
    {} 'pathinfo' STORE
    [ 'year' 'dep' ] <%
      'i' STORE
      'key' STORE
      $pathinfo { $key $res $i GET } APPEND 'pathinfo' STORE
      $key
    %> LFLATMAP DROP
     // build time bounds
    [ $pathinfo 'year' GET TOLONG 1 - 12 31 23 59 59 ] TSELEMENTS-> 'start' STORE
    [ $pathinfo 'year' GET TOLONG 12 31 23 59 59 ] TSELEMENTS-> 'end' STORE
    [
        '<YOUR READ TOKEN>'
        'accidents' { 'dep' $pathinfo 'dep' GET }
        $start ISO8601
        $end ISO8601
    ]  FETCH 'gts' STORE
    // sum by month
    [ $gts bucketizer.sum $end 30 d 0 ] BUCKETIZE
    // Fill missing values
    [ NaN NaN NaN 0 ] FILLVALUE 'gts' STORE
    // sum and merge all
    [ $gts [] reducer.sum ] REDUCE 0 GET 'gts' STORE
    // build the output
    $gts TICKLIST <%
      // drop the LFLATMAP index
      DROP 
      // store timestamp
      't' STORE
      // fetch data from the series
      $gts $t ATTICK 'tick' STORE 
      { 
        'epoch' $t 1000 /
        'date' $t ISO8601
        'value' $tick 4 GET
      }
    %> LFLATMAP
    'body' STORE 
    // build the HTTP response
    { 
      'status' 200  
      'body' $body ->JSON
      'headers' { 'Content-Type' 'application/json' }
    }
  %>
}

Now, open a browser and launch http://127.0.0.1:9000/bydep/2015/22, you should have :

[
  {
    "epoch": 1107561599000,
    "date": "2005-02-04T23:59:59.000000Z",
    "value": 25
  },
  {
    "epoch": 1110153599000,
    "date": "2005-03-06T23:59:59.000000Z",
    "value": 19
  },
  {
    "epoch": 1112745599000,
    "date": "2005-04-05T23:59:59.000000Z",
    "value": 18
  },
  ...
]

Used functions

BUCKETIZE, REDUCE, FILLVALUE, TSELEMENTS->

Using JSON payload on a POST request

As seen, passing parameters by path parameters could be painful. You can parse any payload sent by a POST request:

{
  // root path
  'path' '/post'
  // allow to get path info
  'prefix' true
  // parse manually the payload
  'parsePayload' false   
  'macro' <%
    // save the original request's payload into a variable
    'request' STORE
    $request 'payload' GET 'UTF-8' BYTES-> JSON-> 'payload' STORE 
    [ $payload 'year' GET 1 - 12 31 23 59 59 ] TSELEMENTS-> 'start' STORE
    [ $payload 'year' GET 12 31 23 59 59 ] TSELEMENTS-> 'end' STORE
    [
        '<YOUR READ TOKEN>'
        'accidents' { 'dep' $payload 'dep' GET }
        $start ISO8601
        $end ISO8601
    ]  FETCH 'gts' STORE
    [ $gts bucketizer.sum $end 30 d 0 ] BUCKETIZE [ NaN NaN NaN 0 ] FILLVALUE  'gts' STORE
    [ $gts [] reducer.sum ] REDUCE 0 GET 'gts' STORE
    $gts TICKLIST <%
      // drop the LFLATMAP index
      DROP 
      // store timestamp
      't' STORE
      // fetch data from the series
      $gts $t ATTICK 'tick' STORE 
      { 
        'epoch' $t 1000 /
        'date' $t ISO8601
        'value' $tick 4 GET
      }
    %> LFLATMAP
    'body' STORE 
    // build the HTTP response
    { 
      'status' 200  
      'body' $body ->JSON
      'headers' { 'Content-Type' 'application/json' }
    }
  %>
}
$ curl -i -X POST "Content-Type:application/json" \
   -d \
'{
  "year": 2015,
  "dep": "29"
}' \
 'http://localhost:9000/post'

By sending this cURL POST request, you obtain the same result as the previous example.

Conclusion

You can, of course, factorize code by using the macro mechanism and/or the WarpFleet™ Resolver.

As you can see, it is very easy to deploy lambda functions into Warp 10™ and expose a JSON API.

Share