Warp 10 Office Hours #1 Recap

On June 13th 2019 we held our first Warp 10 Office Hours session. A small crowd gathered during an hour to ask questions to the SenX team about Warp 10. This is a recap of this hour of Q&A.

Question 1

Warp 10 was not always Open Source. The change of license became effective on January 29, 2016. The product development started in summer of 2013 and the analytics language was then called Einstein. The name Einstein was chosen because Warp (not yet called Warp 10) was enabling the management and analytics of spatio-temporal data and when we looked for a name we thought that Einstein was the best person to make sense of the spacetime continuum, so we picked that name and quite naturally the .mc2file extension came to mind and was chosen.

When we decided to go Open Source at the end of 2015, we addressed the associated legal issues. One of them was about trademarks. We then found that the Einstein name was actually a trademark of the Hebraic University of Jerusalem and therefore could not be used freely as we thought, we then changed the name to WarpScript which is a trademark of SenX, but we kept the .mc2 extension as a reminder of the Einstein days. We have registered this extension with the IANA, so it is the official WarpScript file extension you should use.

If you look into the Warp 10 source code, you might still find some references to Einstein here or there.

A fun fact about the Einstein story is that shortly after we changed the name from Einstein to WarpScript, Salesforce announced its IoT/AI product line, which it dubbed… Einstein, obviously if you have deep pockets you can use that name 🙂

Question 2

As you know, there are lots of functions in WarpScript. The latest count is between 900 and 1000. So it can be intimidating, and many people don't use most of the functions as their use cases might not require them. Nevertheless, there are a few gems within that function forest, here is a short selection.

RTFM

The RTFM function is a fun one which returns the URL of the Warp 10 doc site 🙂

INFO

The INFO function is used to document macros. It expects a MAP with fields describing various aspects of a macro, such as its name, its configuration, its description, the set of signatures and the description of parameters, a set of examples, a list of related macros and various other fields. You can view an example of such a parameter map in the warp10-ext-barcode extension.

When encountered during the execution of WarpScript code, INFO will simply discard its parameter map. But if INFOMODE was previously called, INFO will stop the current execution, returning its parameter map as a result.

We encourage you to document your macros this way. We are working on a WarpStudio add-on to visualize a macro's documentation by executing the said macro after calling INFOMODE.

Encoder related functions

Encoders are low level structures which can store data points (timestamp, latitude, longitude, elevation, value). They are more compact than Geo Time Series when used inside WarpScript, can store values of multiple types, but are less flexible in terms of how you can access the data points inside them.

They are the data structures underlying functions such as WRAP or WRAPRAW which allow you to compact a Geo Time Series.

You can also have a look at the NEWENCODER function which allows you to create a GTSEncoder inside WarpScript code.

Snapshot related functions

The SNAPSHOT function and its siblings are a very handy set of functions which will generate WarpScript code to reproduce the state of the stack when SNAPSHOT was called. The resulting WarpScript code can be persisted or transmitted. Later, you can use it to recreate a stack in a given past state.

Quaternions

For video games lovers, a family of functions, such as ->Q, allows you to manipulate quaternions in WarpScript.

Processing

Little known fact, the WarpScript language contains functions which allow use to create images using the Processing syntax. Those functions, around a hundred, range from drawing points and lines to Bézier curves, stamping images or loading SVG content. This is a very useful set of functions to create custom visualizations. The Processing related functions all start with a P and are listed in their category page.

Question 3

We hope to be able to release version 2.1 during the summer. We still have a set of pull requests we need to review and some dependencies we would like to update before we can ship it.

But we are eager to push 2.1 to GA as there are many improvements, both in terms of features and in terms of performance.

Among the new features is the ability to configure Warp 10 natively to do SSL, and a new version of datalog with sharding and easy configuration for multiple targets. There is the ability to store multiple values and binary values in Geo Time Series. You will find a whole set of new functions and performance improvements up to 300x for some common WarpScript patterns. And a lot more.

** UPDATE ** The 2.4 version is now available.

Question 4

We have a branch with support for the Kafka 1.x API. is compatible with Kafka 2.x as we do not use any feature specific to Kafka 2. That branch was initially created as a joint effort to also support MapR Streams and MapR DB, but the branch supports Apache Kafka 1.x+.

For Pulsar, the use of the Pulsar adaptor for Apache Kafka should do the trick.

Note that during the Office Hours, we said it was not possible to use that branch to use Pulsar instead of Kafka. But it seems recent versions of the Pulsar adaptor for Kafka are now fully compatible with the Kafka API and thus do not require the use of specific constructors.

Question 5

We also have a branch for JDK11 compatibility. That branch includes changes we have done to the Warp 10 code to make it compatible with JDK11. But there are some unsolved issues with a number of dependencies that Warp 10 relies on, which make them incompatible with JDK11. This is for example the case of dependencies which make use of Unsafe to access String internal fields for performance reasons, starting with JDK9, the String class no longer has a field of type char[] but instead stores its content in a byte[], so those libraries accessing the char[] all fail. Another issue is the way recent JDK report their version. Many libraries parse the version name expecting it to be 1.x where x is the JDK release (7,8,9, …) but this is no longer the format used for the JDK version.

So we will keep working on that branch whenever we foresee solutions for some dependencies. Be sure that whenever we become JDK11 compatible, we will let you know!

Next Office Hours

We intend to hold recurrent office hours sessions, follow our Twitter account to be notified of the next session. In the meantime, you can ask questions on Stack Overflow, Twitter or the Warp 10™ Google Group.

Mathias Herberts

Co-Founder & Chief Technology Officer