Lambda functions with Warp 10. Use the power of WarpScript in simple JSON API deployments.
![Warp 10 Lambda functions](https://blog.senx.io/wp-content/uploads/2019/07/tuto_lambda.png)
With Warp 10 it is easy to deploy Lambda functions using the Warp 10 HTTP plugin. Build your REST backend.
The first thing is to fetch a dataset. Here is one: https://www.kaggle.com/ahmedlahlou/accidents-in-france-from-2005-to-2016
Then, you need a Warp 10 instance with the HTTP plugin installed. Do not forget to add this to your Warp 10 configuration file: egress.clients.expose = true
so the WarpScript code executed for each HTTP request can access the Warp 10™ Storage Engine.
In the zip archive, there is a caracteristics.csv
file. In order to parse it and upload data against a Warp 10 instance, you can use this Gist: https://gist.github.com/Giwi/3e42d7ad1837657d9329faa619d9b664
This script will produce a couple of Geo Time Series (GTS) with "accidents" as class name and a "col" label representing the collision type:
- Two vehicles – frontal
- Two vehicles – from the rear
- Three vehicles and more – in chain
- Two vehicles – by the side
- Three or more vehicles – multiple collisions
- Other collision
- Without collision
You normally have 8 GTS ("col" may equals "NA")
First Lambda
Lambdas are deployed as spec files for the HTTP Plugin. They are placed in the http.dir
directory defined for the plugin.
<path_to_warp10>/http/last.mc2
We would like to fetch the last n accidents anywhere as a JSON result:
{
//
// This first part of the spec file defines when the plugin should execute 'macro'
//
## root path
'path' '/last'
## allow to get path info
'prefix' true
## true: parse payload of a POST url encoded request
## false: to parse manually the payload
'parsePayload' true
//
// This is the macro which will be called to serve requests to /last/xxx
//
'macro' <%
## save the original request into a variable
'request' STORE
## retrieve the path info (after '/last')
$request 'pathinfo' GET
## extract each parts of the path info
'/' SPLIT 'res' STORE
## evince the first '/'
$res [ 1 $res SIZE ] SUBLIST 'res' STORE
## extract the desired count
$res 0 GET TOLONG 'count' STORE
[ '<YOUR READ TOKEN>' 'accidents' {} NOW $count -1 * ] FETCH
MERGE
$count SHRINK 'gts' STORE
$gts TICKLIST <%
## drop the LFLATMAP index
DROP
## store timestamp
't' STORE
## fetch data from the series
$gts $t ATTICK 'tick' STORE
{
'epoch' $t 1000 /
'date' $t ISO8601
<% $tick 1 GET ISNaN ! %>
<% 'lat' $tick 1 GET %> IFT
<% $tick 1 GET ISNaN ! %>
<% 'long' $tick 2 GET %> IFT
'value' $tick 4 GET
}
%> LFLATMAP
'body' STORE
## build the HTTP response
{
'status' 200
'body' $body ->JSON
'headers' { 'Content-Type' 'application/json' }
}
%>
}
Now, open a browser and launch http://127.0.0.1:9000/last/10, you should have :
[
{
"epoch": 1480930500000,
"date": "2016-12-05T09:35:00.000000Z",
"lat": 50.25567997712642,
"long": 2.7589399740099907,
"value": 1
},
{
"epoch": 1480930500000,
"date": "2016-12-05T09:35:00.000000Z",
"lat": 50.25567997712642,
"long": 2.7589399740099907,
"value": 1
}, ...
]
Used functions
ISO8601, FETCH, LFLATMAP, ->JSON, ATTICK, TICKLIST, SUBLIST, TOLONG, SHRINK, MERGE, ISNaN, IFT
A more complex Lambda
We would like to have the monthly count of accidents for a given year and for a given department as a JSON result:
<path_to_warp10>/http/bydep.mc2
{
// root path
'path' '/bydep'
// allow to get path info
'prefix' true
// true: parse payload of a POST url encoded request
// false: to parse manually the payload
'parsePayload' true
'macro' <%
// save the original request into a variable
'request' STORE
// retrieve the path info (after /bydep)
$request 'pathinfo' GET
// extract each parts of the path info
'/' SPLIT 'res' STORE
// evince the first '/'
$res [ 1 $res SIZE ] SUBLIST 'res' STORE
// map pathinfo
{} 'pathinfo' STORE
[ 'year' 'dep' ]
<%
'i' STORE
'key' STORE
$pathinfo { $key $res $i GET } APPEND 'pathinfo' STORE
$key
%> LFLATMAP DROP
// build time bounds
[ $pathinfo 'year' GET TOLONG 1 - 12 31 23 59 59 ] TSELEMENTS-> 'start' STORE
[ $pathinfo 'year' GET TOLONG 12 31 23 59 59 ] TSELEMENTS-> 'end' STORE
[ '<YOUR READ TOKEN>' 'accidents' { 'dep' $pathinfo 'dep' GET } $start ISO8601 $end ISO8601 ] FETCH 'gts' STORE
// sum by month
[ $gts bucketizer.sum $end 30 d 0 ] BUCKETIZE
// Fill missing values
[ NaN NaN NaN 0 ] FILLVALUE 'gts' STORE
// sum and merge all
[ $gts [] reducer.sum ] REDUCE 0 GET 'gts' STORE
// build the output
$gts TICKLIST <%
// drop the LFLATMAP index
DROP
// store timestamp
't' STORE
// fetch data from the series
$gts $t ATTICK 'tick' STORE
{
'epoch' $t 1000 /
'date' $t ISO8601
'value' $tick 4 GET
}
%> LFLATMAP
'body' STORE
// build the HTTP response
{
'status' 200
'body' $body ->JSON
'headers' { 'Content-Type' 'application/json' }
}
%>
}
Now, open a browser and launch http://127.0.0.1:9000/bydep/2015/22, you should have :
[
{
"epoch": 1107561599000,
"date": "2005-02-04T23:59:59.000000Z",
"value": 25
},
{
"epoch": 1110153599000,
"date": "2005-03-06T23:59:59.000000Z",
"value": 19
},
{
"epoch": 1112745599000,
"date": "2005-04-05T23:59:59.000000Z",
"value": 18
}, ...
]
Used functions
BUCKETIZE, REDUCE, FILLVALUE, TSELEMENTS->
Using JSON payload on a POST request
As seen, passing parameters by path parameters could be painful. You can parse any payload sent by a POST request:
{
// root path
'path' '/post'
// allow to get path info
'prefix' true
// parse manually the payload
'parsePayload' false
'macro' <%
// save the original request's payload into a variable
'request' STORE
$request 'payload' GET 'UTF-8' BYTES-> JSON-> 'payload' STORE
[ $payload 'year' GET 1 - 12 31 23 59 59 ] TSELEMENTS-> 'start' STORE
[ $payload 'year' GET 12 31 23 59 59 ] TSELEMENTS-> 'end' STORE
[
'<YOUR READ TOKEN>'
'accidents' { 'dep' $payload 'dep' GET }
$start ISO8601
$end ISO8601
] FETCH 'gts' STORE
[ $gts bucketizer.sum $end 30 d 0 ] BUCKETIZE
[ NaN NaN NaN 0 ] FILLVALUE 'gts' STORE
[ $gts [] reducer.sum ] REDUCE 0 GET 'gts' STORE
$gts TICKLIST <%
// drop the LFLATMAP index
DROP
// store timestamp
't' STORE
// fetch data from the series
$gts $t ATTICK 'tick' STORE
{
'epoch' $t 1000 /
'date' $t ISO8601
'value' $tick 4 GET
}
%> LFLATMAP
'body' STORE
// build the HTTP response
{
'status' 200
'body' $body ->JSON
'headers' { 'Content-Type' 'application/json' }
}
%>
}
curl -i -X POST "Content-Type:application/json" \
(out) -d '{ "year": 2015, "dep": "29"}' \
(out) http://localhost:9000/post
By sending this cURL POST request, you obtain the same result as the previous example.
Conclusion
You can, of course, factorize code by using the macro mechanism and/or the WarpFleet Resolver.
As you can see, it is very easy to deploy lambda functions into Warp 10 and expose a JSON API.
Read more
WarpStudio in standalone mode
Using Warp 10 as a map tile server for Discovery
n8n & Warp 10 - Automate your time series manipulations
![Xavier Marin](https://blog.senx.io/wp-content/uploads/2019/04/IMG_8993.jpg)
Senior Software Engineer