September 2020: Warp 10 release 2.7.0, ready for FLoWS

On Thursday, September the 10th of 2020, three and a half months after the release of the 2.6.0, we published the 2.7.0 release of Warp 10. Some exciting new features and functions, as well as a few optimizations and several bug fixes, come with this new release.

New Features

Fetching the right data is the first step in creating fast and concise scripts for data analytics. That’s why Warp 10 offers multiple parameters to precisely select the data you want. In the 2.3.0 release, we introduced skip, sample, and boundaries.

In this 2.7.0 release, we introduce step and timestep. Step allows you to get one point every N points by stepping over points. Timestep works similarly, but on time, allowing you to get one point every at least N time unit. These 2 parameters as well as the sample parameter let you choose a technique to subsample your data when using all of it is irrelevant.

While adding these new parameters, we noticed Warp10InputFormat was lagging behind in terms of functionalities. It was not offering the same parameters fetch had, so we updated it to make it on par with standard fetching. In Spark or Pig, you can now use both a time window and a count as well as boundaries and subsampling to fetch data from Warp 10.

For those seeking performance while having a very high number of GTSs, we added the labels priority feature. This lets you define the order with which the labels are used to select the GTS you requested by a FIND or a FETCH. By prioritizing the most discriminating labels, you can improve the performance of FIND and FETCH. Remember that the benefit is only visible for a very high number of GTS, more than millions.

Finally, we modified the mechanism of custom JSON serializers to make it easier to implement. Instead of returning a raw string, you can choose between a raw string or a serializable object. This makes the serialization code easier to write and understand. 

FLoWS is Coming

We heard you!

WarpScript has a too steep learning curve for some new users. We will introduce the FLoWS language, which will simplify the onboarding of new users by making functions input and output clearer.

What differentiates FLoWS from WarpScript is its functional approach, hence the name FLoWS for Functional Lineage of WarpScript. FLoWS is fully compatible with WarpScript, enabling the use of all core functions plus all those from extensions.

The 2.7.0 release is the first to be able to support FLoWS by providing the necessary functionalities for FLoWS to operate. Be sure to update to this release if you plan to use FLoWS!

We will release it very soon, be sure to keep an eye on the blog, or subscribe to the newsletter to be notified!

New and Updated Functions

Maps

• SORTBY and SORTWITH can now sort LinkedHashMaps as created by {...}.
• We added MMAP which works like LMAP but on Maps.

Geo

• The new function GEO.BUFFER lets you "inflate" a GeoJSON, WKT or WKB by creating a buffer around the shape.
• Two more conversion functions ->WKT and ->WKB are available to convert GeoShapes and GeoJSON to either WKT or WKB.
• ->GEOJSON can now convert from WKT or WKB.

Registers

• You can now use RUNR to run macros from registers which is faster than calling RUN with a register number.
• ASREGS now optimizes, even more, your macros with POPRs, CPOPRs, PUSHRs, and RUNRs.

Processing

• PupdatePixels and Ppixels now both support PImage as a parameter.
• Pencode is updated to handle PNG generation with tEXt, zTXt, and iTXt chunks.
• The new function Pmask allows masking a PImage with another.

Misc

• The HIDE and SHOW new functions let you hide the deepest elements on the stack. They are mainly used by the new FLoWS language.
• A macro or a mapper can now be given to DEDUP to determine how duplicates are handled.
• REMOVE is improved to remove a point in GTS according to its index.
• STORE called on a list of variable names with duplicates only associates the last occurrence instead of the first.
• URLFETCH can now be given a map of headers.
• SHAPE is modified to work on an empty list and, in this case, returns [0].
• All 6 comparison functions can now be given a constant and a GTS in any order.

Performance Improvements

Two modifications are packed with this 2.7.0 release to improve performance. Their goal is to reduce unnecessary copies to reduce CPU and memory consumption:

• Metadatas are no longer deep copied when cloning FetchRequest
• Calls to ImmutableMap.copyof are replaced by Collections.unmodifiableMap when possible.

Bug Fixes

• Fetching only by count in a chunked in-memory no longer throws an error #780.
• Calling FETCH with MINLONG timespan does not overflow count anymore #781.
• The StreamUpdate endpoint is now functional again thanks to a fixed parsing error #783.
• Stacks now honor MAXDEPTH even if smaller than the array backing MemoryWarpScriptStack #785.
• UNBUCKETIZE.CALENDAR no longer fails on empty GTSs #793.
• BUCKETIZE.CALENDAR now sets the correct bucketcount #792.
• Comparison function does not throw an NPE anymore when the first operand is null 09d513.
• Made sure parallel scanners won’t throw an NPE on interruption #806.
• VARS now returns variables that are only used for storage #809.
• Distributed Stores no longer starve on machines with a high number of cores #813.
• Made sure the GeoXPShapes contain sorted and deduped lists of geocells #823.

Fabien Tencé

WarpScript™ Doctor