May 2021: Warp 10 releases 2.8.0 and 2.8.1

The 2.8.0 version of Warp 10 was released on the 5th of May 2021, 8 months after the release of the 2.7.0 version. This is far longer than usual between two minor versions, so expect a lot of new functionalities! 130 Pull Requests have been merged since the 2.7.0, adding and extending functions, improving the performance, and fixing bugs on the platform.

2.8.1

The 2.8.1 is already out! It only fixes a publication issue which makes impracticable to have Gradle projects depends on the WarpScript lib.

New Features

The Warp 10 platform is built with security in mind. Tokens already allow to finely control the access to the data: who can read, update or delete the data. Warp 10 is also a platform to execute scripts and many instances are multitenant. We added capabilities to the tokens to also control which functions are accessible and to what extent. By adding capability attributes in a token, you can check and manipulate them with CAPCHECK, CAPADD, CAPDEL and CAPGET. More on that in an upcoming blog post.

As you may want to generate capabilities-only tokens, new attributes are also available in tokens. .noauth disallow authentication, .nofind and .nofetch disallow respectively find and fetch operations for a read token. You can truly craft tokens to your very need.

The 2.7.0 version is FLoWS-ready, which means you can add the FLoWS extension and execute FLoWS scripts. In the 2.8.1 version, we added a MacroResolver mechanism to be able to deploy FLoWS macros. This brings us a step closer to Warp 10 instances only using FLoWS.

New Functions

A lot of new functions have been added to be able to work with the Ethereum blockchain:

• ECRECOVER to find possible public keys from a hash and an associated ECDSA signature
• The KECCAK.256 hash function and also 20 other hash functions
• ->B58, ->B58C, B58-> and B58C-> to convert from/to base58
• ->RLP and RLP-> to convert from/to RLP

Other crypto-related functions have been added, like the Shamir's Secret Sharing Scheme ->SSSS and SSSS-> and Elliptic-curve Diffie–Hellman ECDH to share secrets.

Still, on the side of cryptography and security, functions have been added to sign and verify macros: MSIGN, MVERIFY, MSIG, MDETACH, MSIGCOUNT and MSIGINFO. This can be particularly useful to make sure macros from a remote MacroRepository were not tampered with.

As you may need to access resources outside your instance, the HTTP function was introduced to make HTTP requests easy and flexible. It offers much more options than URLFETCH and relies on capabilities.

In addition to JSONSTRICT and JSONLOOSE, you can now choose whether your script output pretty or not JSON with the JSONPRETTY and JSONCOMPACT functions.

You can now resize your images using bilinear interpolation using Presize.

To simplify the use of the shared memory extension, the SHMDEFINED has been added, which tells whether a symbol is defined or not.

To complete mapper.geo.within and mapper.geo.outside, we added mapper.geo.fence to tell if a point is inside a shape or not. This is especially useful to detect when a device enters or exits a particular area.

New utility functions are now available: ->MACRO and MACRO-> functions convert macros from/to lists and FUNCREF gets the function from a name.

Updated Functions

On top of new functions, the 2.8.1 release also improve functions:

• FETCH can return GTSEncoders and HBase TTLs
• DELETE can do a dry-run to return would-be deleted GTS
• TOKENDUMP can dump tokens defined in the tokens file
• TOKENDUMP and TOKENGEN can be used with custom keys to manipulate other instances' tokens
• DTW can work on locations, elevations and a window can be specified
• ** can work on a GTS and Lists
• Most math functions, like FLOOR, can work on GTS
• Pencode can encode an Image
• mapper.percentile, reducer.percentile and bucketizer.percentile implement Hyndman's 9 types of percentile computation
• GET can get a single character in a String
• SPLIT now accepts a limit parameter to early-abort splitting
• GEO.INTERSECTS can tell if two GeoShapes intersect
• == works on bytes and != works on GeoShapes
• STRICTMAPPER can operate on macros which are used in MAP
• TYPEOF now returns FUNCTION for a Function
• IMMUTABLE can work on Map
• TOBIN and ->BIN, as well as TOHEX and ->HEX have been merged
• ECPRIVATE accepts hexadecimal parameter
• SIZE can work on Macros
• PcreateFont always defaults on 12 size
• + works on secure macros
• ~= is now less strict on valid parameter types

Improvements

To simplify debugging, we improved error messages, when using the /fetch and /find endpoints for instance. We also named each and every thread created to be able to better understand thread dumps.

We made performance improvements, like context restoration for instance, which further improve WarpScript and FLoWS performances.

The HTTP plugin now has the option to accept streams of data, which can be used to create new ingestion endpoints with custom formats.

The last Python dependency was removed, so Warp 10 can be initialized and run without Python. This makes the Docker image even slimmer with this dependency removed!

Snapshots have been improved with more compatible objects and better formatting.

More low-level configurations are exposed for LevelBD, like leveldb.create.if.missing and leveldb.maxopenfiles.

Bug Fixes

The 2.8.1 release fixes bugs which were present in the 2.7.0 version. Even if most of the fixes are already present in the 2.7.5, we encourage you to upgrade to the 2.8.1 version. Here are the bug fixes:

• Fix the skip and step parameters in FETCH on the Standalone version
• Fix race condition in GTSDecoderIteratorRunnable
• Fix concurrency in StandaloneDirectoryClient
• Fix possible hint overflow in InMemoryChunkSet
• Fix Shadow context save and restore
• Fix wrongly caught ATC Exception in MAP and SORTWITH which made STOP work improperly
• Fix rare collision bug in DataLog file names
• Fix NPE when doing a DELETE with uninitialized memory chunks
• Fix WarpScriptMacroLibrary TTL
• Fix NOT when applied on a GTS only containing false values
• Fix retrieval of post boundary when at the end of LevelDB keyspace
• Fix INTERPOLATE formula
• Fix op.add and op.mul cast in certain case
• Fix NPE in != when one of the two operands was null
• Fix current tick for mappers when GTS has duplicate ticks
• Fix NPE in URLFETCH when response has no body
• Fix NPDF snapshot

Fabien Tencé

WarpScript™ Doctor