October 2021: Warp 10 release 2.9.0

The 2.9.0 version of Warp 10 was released on the 22nd of October 2021, 5 months after the release of the 2.8.0 and 2.8.1 versions. Like the last minor version, this release delivers a lot of improvements and new ways of customizing the platform to your very needs! 110 Pull Requests have been merged since the 2.8.0, adding and extending functions, improving the performance, and fixing bugs on the platform.

New Features

You can now limit the duration of execution of all scripts submitted on /exec. To do so, specify egress.maxtime in your configuration file. To allow specific users to go over this limit either to some longer duration or remove the limit altogether, you can use token capabilities, see the part on TIMEBOX below.

To further constrain what your users can do, you can limit /fetch and FETCH parameters using token attributes. You can restrict a user to access to a specific period of time or greatly reduce the number of points this user can fetch.

While it is very easy to dump data out of Warp 10, it can be less trivial to recover from a connection lost or to parallelize the process. With the two new parameters gskip and gcount for /find, FIND, /fetch and FETCH, you can paginate your dump. gcount will limit the number of returned GTS while gskip will skip the given number of GTS before returning the result. Of course, this will only work if no new GTS is created during the process.

Would you need to rapidly generate a lot of realistic data for benchmarking purposes, for instance, you can now shift all the timestamps sent to /update. By setting the X-Warp10-Timeshift header, this shift will be applied to all the parsed timestamps. You can then easily submit the same file multiple time with different shifts to simulate a larger dataset.

This one is for advanced users only: it is possible to delete all points from a GTS without removing the metadata. This can be particularly useful if you want to clear a GTS just before repopulating it with fresh points. This is the counterpart of metaonly.

New Functions

When a server-side macro manipulates sensitive information you want to make sure that none leaks. The GUARD function makes sure of that by cleaning variables, registers, stack and error messages.

Warp 10 already supports GeoJSON, WKT and WKB. It now also supports KML and GML with the ->KML, ->GML, GEO.KML, GEO.KML.UNIFORM, GEO.GML and GEO.GML.UNIFORM functions.

To simplify the modification of Lists and Maps, in the same way LMAP and MMAP do, we introduced LFILTER and MFILTER. As you may already have guessed, they allow the filtering of elements in Lists and Maps according to the value returned by a Macro.

Two new functions have been added to do polynomial interpolation of GTSs: POLYFIT, to learn the polynomial coefficients and POLYFUNC to interpolate a point on this polynomial.

The last added functions are variance and standard deviation aggregators. bucketizer.var was missing so we exposed it. As the algorithm used for the computation of variance and standard deviation can suffer from numerical instability, especially when the variance is low compared to the mean, we added Welford's method to compute them. As it is 10% slower, we opted to expose the functions using a different name to leave the choice of the algorithm to you. Those new aggregators are: mapper.var.welford, mapper.sd.welford, bucketizer.var.welford, bucketizer.sd.welford, reducer.var.welford, reducer.var.welford.forbid-nulls, reducer.sd.welford and reducer.sd.welford.forbid-nulls.

Updated Functions

You can now use REMOVE to delete an element in a SET, in BYTES or in a STRING.

REDUCE will now keep common attributes in GTS partitions, in the same way it keeps common labels.

HHCODE.BBOX, HHCODE.CENTER, HHCODE.NORTH, HHCODE.SOUTH, HHCODE.EAST, HHCODE.WEST, HHCODE.NORTH.EAST, HHCODE.NORTH.WEST, HHCODE.SOUTH.EAST and HHCODE.SOUTH.WEST can now work with odd HHCode resolutions. Although odd-resolution HHCodes are less practical because they cannot be represented using hexadecimal or used in GeoShapes, there are totally valid.

Functions outputting text on standard output or error flush after each write to ensure it does not stay in the buffer. Affected functions are in the debug extension: LOGMSG, STDOUT and STDERR.

TIMEBOX time limit can now be constrained with capabilities. Set the warpscript.timebox.maxtime.capname configuration key to define the capability name.

METASORT will now also use attributes to order GTSs, if class and labels are the same.

In case there are missing locations in a GTS, MOTIONSPLIT will now ignore the points.

Improvements

Capabilities can be specified using the X-Warp10-Capabilities header and a comma-separated list of tokens on the /exec endpoint.

Access-Control-Allow-Private-Network to CORS headers and Access-Control-Request-Private-Network to Access-Control-Allow-Headers for RFC1918 have been added to allow browsers to connect to local instances.

A read timeout configuration can be set for the TCP plugin using the socketReadTimeout key in the configuration Map.

The changelog is now much nicer to read being based on PRs. It can be found in the root of the Warp 10 release archive.

As always some optimizations and better error handling and messages.

Bug Fixes

The 2.9.0 fixes some bugs present in the 2.8.1 version. We strongly encourage you to update to the 2.9.0 version because of a memory leak which appears when doing a lot of deletes. Here is the list of fixes:

• Fix the closing of a batch which caused a memory leak.
• Fix the closing of a decoder in the accelerator preloading phase which could make some persisted data unavailable.
• Fix a bug where some TCPClients could connect but were not handled instead of being immediately disconnected.
• Fix a NullPointerException when handling error messages which lead to vague error messages.
• Fix context handling in the Shadow extension.
• Fix macro TTL extraction for WarpFleet macro repositories.
• Fix bugs that made STL fail with cryptic error messages.
• Fix DEVAL shard modulus assignment.

Fabien Tencé

WarpScript™ Doctor