Warp 10™ version 2.1

Almost eight months after our major 2.0 release, the team at SenX is very happy to announce the release of version 2.1 of Warp 10™, confirming it as the most advanced time series platform.

This release could almost have been a major one given how many changes it includes but we decided to remain on the 2.x path, after all 2.1 is one twentieth of 42!

Here is a list of features and enhancements added to 2.1. We highly encourage you to migrate to 2.1 to benefit from all of them, performance improvements make the move way worth it.

New features

Native SSL

All the external HTTP endpoints (ingress, egress, plasma) can now be configured with SSL support enabled. This requires that you have a certificate and associated private key available in a Java Key Store. You can use separate JKS and certificates for each endpoint.

Binary values

Geo Time Series™ can contain LONG, DOUBLE, BOOLEAN or STRING values. With release 2.1, STRING values can now either be standard UTF-8 encoded strings or ISO-8859-1 representations of raw byte arrays, we call this subtype BINARY. Encoding binary data in base64 is no longer required thus leading to a 25% reduction of the storage space required for blobs. The FETCH and ->GTS functions have been modified to correctly return data that was stored as binary.

Multivariate values

The support of binary values enabled the automatic encoding of multivariate values. This new type of values enables Warp 10™ to reach tremendous levels in terms of performance. This is the addition which made our record breaking Raspberry Pi 4 benchmark possible.

Multi Conf

The configuration mechanism for Warp 10™ has been modified to support multiple configuration files. This makes it easier to store the configuration for a given extension in its own file for example and allows for an overall better organization of your config.

Datalog++

The datalog mechanism has been enhanced to support forwarding to multiple targets from within a single Warp 10™ instance, and to support sharding the dataset across targets. Coupled with a dedicated extension, you can configure a set of standalone Warp 10™ instances to act as a mini-cluster, ensuring high availability (datalog replication) and scalability (sharding).

Warp Studio

We retired the aging quantum web interface to Warp 10™ has been and we replaced with a richer, less buggy and more performant web app dubbed Warp Studio. Warp Studio is also accessible online.

New functions

Release 2.1 has a certain number of new functions which were added in different domains to make you more efficient when working with WarpScript™.

The set of operators working on Geo Time Series™ has been expanded and now includes comparison operators <, >, <=, >=, == and !=, bitwise operators <<, >>, >>>, & (AND), | (OR) and ^ (XOR) and boolean operators ! (NOT), || (OR) and && (AND).

Function for working with multidimensional arrays (i.e. tensor like structures) have been added: SHAPE, RESHAPE, CHECKSHAPE, HULLSHAPE and PERMUTE.

New timing functions have appeared to help you tune your WarpScript™ code and control its execution time: CHRONOSTART, CHRONOEND, CHRONOSTATS, TIMED and TIMEBOX.

New compression/decompression functions DEFLATE and INFLATE use the Zlib compression library.

Functions for working with controlling the WarpFleet™ Resolver: IMPORT, WFON and WFOFF.

Functions for dynamically configuring macros: MACROCONFIG, SETMACROCONFIG and MACROCONFIGSECRET.

Function to produce a summary of the Warp 10™ instance configuration and status, useful when reporting a problem: REPORT.

Handy function to shuffle a list: SHUFFLE.

Functions for disabling/enabling data ingestion and deletion and metadata updates: UPDATEON UPDATEOFF METAON METAOFF DELETEON DELETEOFF.

Conversion function ->LONGBYTES to convert LONGs to byte arrays.

Macros can now be executed from registers via an enhanced RUN function.

Functions for working with multivariate value format: PARSEVALUE, ->MVSTRING, MVINDEXPLIT, MVTICKSPLIT, MVVALUES, MVTICKS, MVLOCATIONS, MVELEVATIONS, MVHHCODES, WRAPMV

Lastly, many functions operating on Geo Time Series™ can now also operate on Encoders.

New Configurations

We added the following new configuration

# Secret for REPORT, if unset, a random secret will be generated and output in the logs and on stdout
#warp10.report.secret = 

# Manager secret, must be set to use the managing functions
#warp10.manager.secret =

# Set to a message indicating the reason why updates are disabled, they are enabled if this is not set
#warp.update.disabled = Updates are momentarily disabled
  
# Set to a message indicating the reason why deletes are disabled, they are enabled if this is not set
#warp.delete.disabled = Deletes are momentarily disabled
  
# Set to a message indicating the reason why meta updates are disabled, they are enabled if this is not set
#warp.meta.disabled = Meta updates are momentarily disabled

# Maximum time that TIMEBOX can wait for an execution (in ms), defaults to 30s
#warpscript.timebox.maxtime=30000

# Secret to use for setting configuration accessible via MACROCONFIG
#warpscript.macroconfig.secret =

# Default TTL for macros loaded on demand
#warpscript.repository.ttl = 600000

# TTL to use for failed macros, a new on-demand loading will occur after this delay.
# Defaults to the max of 10s or half of warpscript.repository.refresh
#warpscript.repository.ttl = 

# Maximum TTL that can be set using MACROTTL (defaults to 2**62 ms)
#warpscript.repository.ttl.hard = 

#
# WarpFleet

# Comma separated list of default WarpFleet™ repositories
warpfleet.macros.repos = https://warpfleet.senx.io/macros

# Read timeout when fetching macro source code from a repository, in ms. Defaults to 10s.
#warpfleet.timeout.read =

# Connection timeout when fetching macro source code from a repository, in ms. Defaults to 5s.
#warpfleet.timeout.connect =

# Configure this property to 'true' to disable the function WF.GETREPOS. This is useful when some of your repo URLs have sensitive information.
#warpfleet.getrepos.disable = true

# Maximum number of macros from WarpFleet™ to cache, defaults to 10000
#warpfleet.cache.size = 10000

# Default TTL (in ms) for macros loaded from a WarpFleet repository (defaults to 10 minutes)
#warpfleet.macros.ttl = 600000

# Lower limit for TTL (in ms) of macros loaded from a WarpFleet repository (defaults to 60 seconds)
#warpfleet.macros.ttl.min = 60000

# Upper limit for TTL (in ms) of macros loaded from a WarpFleet repository (defaults to 24 hours)
#warpfleet.macros.ttl.max = 

# Default TTL (in ms) for WarpFleet macros which had errors (defaults to 10 seconds)
#warpfleet.macros.ttl.failed = 10000

# Default TTL (in ms) for WarpFleet macros which were not found. If > 0, a dummy macro
# will be generated which will fail with an informative error message
#warpfleet.macros.ttl.unknown = 0

# Name of WarpFleet repository macro. This macro consumes a URL and emits a boolean.
#warpfleet.macros.validator = 

# REXEC connect timeout in ms (default = 0, no timeout)
#warpscript.rexec.timeout.connect = 0
# REXEC read timeout in ms (default = 0, no timeout)
#warpscript.rexec.timeout.read = 0

#
# Set datalog.sync to true to force a call to fsync upon closing each datalog file, thus ensuring
# the file blocks are written to the underlying device. This defaults to false if the property is not set.
#
datalog.sync = true

# Set to false to disable support for gzipped responses
#http.gzip = false
# Set to true to convert header names to lower case in the request map
#http.lcheaders = true

##
## SSL Configuration
##

## Standalone

# Host/IP Address to bind to for the SSL endpoint
#standalone.ssl.host =
# Port the SSL endpoint should bind to
#standalone.ssl.port =
# Number of Jetty acceptors (defaults to 2)
#standalone.ssl.acceptors =
# Number of Jetty selectors (defaults to 4)
#standalone.ssl.selectors =
# Jetty idle timeout in ms
#standalone.ssl.idletimeout =
# Path to the JKS keystore file
#standalone.ssl.keystore.path =
# Password for accessing the keystore file
#standalone.ssl.keystore.password =
# Alias associated with the SSL X.509 certificate
#standalone.ssl.cert.alias =
# Password for accessing the X.509 certificate
#standalone.ssl.keymanager.password =

## Distributed

## Ingress

# Host/IP Address to bind to for the SSL endpoint
#ingress.ssl.host =
# Port the SSL endpoint should bind to
#ingress.ssl.port =
# Number of Jetty acceptors (defaults to 2)
#ingress.ssl.acceptors =
# Number of Jetty selectors (defaults to 4)
#ingress.ssl.selectors =
# Jetty idle timeout in ms 
#ingress.ssl.idletimeout =
# Path to the JKS keystore file
#ingress.ssl.keystore.path =
# Password for accessing the keystore file
#ingress.ssl.keystore.password =
# Alias associated with the SSL X.509 certificate
#ingress.ssl.cert.alias =
# Password for accessing the X.509 certificate
#ingress.ssl.keymanager.password =

## Egress/Fetcher

# Host/IP Address to bind to for the SSL endpoint
#egress.ssl.host =
# Port the SSL endpoint should bind to
#egress.ssl.port =
# Number of Jetty acceptors (defaults to 2)
#egress.ssl.acceptors =
# Number of Jetty selectors (defaults to 4)
#egress.ssl.selectors =
# Jetty idle timeout in ms
#egress.ssl.idletimeout =
# Path to the JKS keystore file
#egress.ssl.keystore.path =
# Password for accessing the keystore file
#egress.ssl.keystore.password =
# Alias associated with the SSL X.509 certificate
#egress.ssl.cert.alias =
# Password for accessing the X.509 certificate
#egress.ssl.keymanager.password =

## Plasma Frontend

# Host/IP Address to bind to for the SSL endpoint
#plasma.frontend.ssl.host =
# Port the SSL endpoint should bind to
#plasma.frontend.ssl.port =
# Number of Jetty acceptors (defaults to 2)
#plasma.frontend.ssl.acceptors =
# Number of Jetty selectors (defaults to 4)
#plasma.frontend.ssl.selectors =
# Jetty idle timeout in ms
#plasma.frontend.ssl.idletimeout =
# Path to the JKS keystore file
#plasma.frontend.ssl.keystore.path =
# Password for accessing the keystore file
#plasma.frontend.ssl.keystore.password =
# Alias associated with the SSL X.509 certificate
#plasma.frontend.ssl.cert.alias =
# Password for accessing the X.509 certificate
#plasma.frontend.ssl.keymanager.password =

## HTTP Plugin

# Host/IP Address to bind to for the SSL endpoint
#http.ssl.host =
# Port the SSL endpoint should bind to
#http.ssl.port =
# Number of Jetty acceptors (defaults to 2)
# Adapt value of http.maxthreads accordingly
#http.ssl.acceptors =
# Number of Jetty selectors (defaults to 4)
# Adapt value of http.maxthreads accordingly
#http.ssl.selectors =
# Jetty idle timeout in ms
#http.ssl.idletimeout =
# Path to the JKS keystore file
#http.ssl.keystore.path =
# Password for accessing the keystore file
#http.ssl.keystore.password =
# Alias associated with the SSL X.509 certificate
#http.ssl.cert.alias =
# Password for accessing the X.509 certificate
#http.ssl.keymanager.password =

Performance improvements

Bug fixes and code refactoring have made some operations way faster. As an example APPLY and FILTER are now faster when called on several 10s of thousands of GTS. Also all BUCKETIZE operations following a FETCH should see performance improvements and MAP operations were sped up thanks to faster sub-series extraction.

On the storage engine side, the support for multiple values in the input format enables blazing-fast ingestion performance even on very limited hardware. A future blog post will show how this probably makes Warp 10™ also the fastest time series platform!

How to migrate an existing Warp 10™ instance to 2.1?

The 2.1 release of Warp 10™ is compatible with previous releases, but slight changes to the layout of the configuration files requires that you proceed with the steps below. The configuration change is that Warp 10™ can now use multiple configuration files sitting in the etc/conf.d directory, this enables you to create separate configuration files for separate components of Warp 10™, thus making the configuration easier to read. This also makes it possible for the WarpFleet™ packages and plugins to have their own separate configuration file.

  1. Get the latest version of Warp 10™: https://www.warp10.io/download
  2. Extract the jar (ie: warp10-2.1.0.jar) from the archive
  3. Extract the warp10-standalone.sh file from the archive
  4. Stop your current Warp 10™ instance: ${WARP10_HOME}/bin/warp10-standalone.init stop
  5. Copy the new jar and new warp10-standalone.sh in your previous Warp 10™ bin directory
  6. Adjust Java Heap size if needed
  7. Create a new directory ${WARP10_HOME}/etc/conf.d
  8. Move old configuration file the this new directory
  9. Start your new Warp 10™ instance: ${WARP10_HOME}/bin/warp10-standalone.init start
Share