All you need to know about interactions between Warp 10 and Python

Hello Warp 10 from Python

The first way to interact between Python and Warp 10 is through the WarpScript egress endpoint. For example:

# pip install requests
import requests
mc2 = "'Hello Warp 10!'"

requests.post('https://sandbox.senx.io/api/v0/exec', mc2).json()
# ['Hello Warp 10!']

Inter-operating WarpScript and Python

Using a Py4J gateway, you can access a WarpScript execution environment (a stack) directly from Python.

There are two ways to launch a gateway that gives access to a stack:

• either it is launched by the Warp 10 platform via the Py4J plugin (available here);
• or it can be launched manually from a Python script, but in this case no Warp 10 instance is connected to it (see this blog post, section 4, or this one)

Using a Py4J gateway has multiple advantages:

• you don't need to use POST requests to execute your code
• you can access WarpScript objects from Python without them being serialized in JSON format
• It's possible to go back and forth between Python and the WarpScript stack that is in the JVM
• and you can process with Python server-side close to Warp 10 data

However, the Py4J gateway gives access to all classes loaded in the JVM of the WarpScript execution environment. So it is recommended not to use it in a standalone multi-tenant installation, or at least to restrict its access to privileged users.

The Py4J plugin

The Py4J plugin launches a gateway in the same JVM running Warp 10.
You can install it using WarpFleet from the home folder.

WarpFleet is Warp 10 package manager, find more details here: Introducing WarpFleet

wf g -w /path/to/warp10 io.warp10 warp10-plugin-py4j

Once the plugin is installed and registered in the configuration files, restart Warp 10. Then, you can set a connection with the gateway from Python:

# pip install Py4J
from py4j.java_gateway import (JavaGateway, GatewayParameters)

# if you need to launch a gateway manually, you can use py4j.launch_gateway() function

# Connect to the gateway
params = GatewayParameters('127.0.0.1', 25333, auto_convert=True)
gateway = JavaGateway(gateway_parameters=params)

Then, you can create a WarpScript stack and start querying the Warp 10 platform.

# Instanciates a WarpScript stack
stack = gateway.entry_point.newStack()
stack = stack.execMulti('<My_WarpScript_code_here>')

# Extract top of the stack and store it in a Python variable
# @see https://www.py4j.org/advanced_topics.html#collections-conversion
my_var = stack.pop()

The following methods are useful:

• execMulti(str): executes a multiline WarpScript code on the stack
• pop(): extract the top of the stack
• get(int): get the object at a certain level of the stack
• clear(): reset the stack
• load(str): get the object that was stored under a certain variable in WarpScript
• getStackAttribute(key): get an attribute
• setStackAttribute(key,value): set an attribute

Py4J with a Jupyter Server

The python package warp10-jupyter contains the cell magic called %%warpscript that executes WarpScript code through a gateway of the Py4J plugin. You can install it with pip install warp10-jupyter, or from the source code available here.

For example, you can load it in a Jupyter notebook with:

%load_ext warpscript

Then you can start executing WarpScript code in the following cells:

%%warpscript --stack s
<My_WarpScript_code_here>

The newly created WarpScript stack is stored under the Python variables. Therefore, its objects can be accessed with the aforementioned methods.

After the cell has been executed, the stack is still in the JVM so subsequent cells starting with %%warpscript --stack s will execute WarpScript code on the same stack.

Recall that if needed, the docstring of this cell magic can be invoked within a notebook with %%warpscript?.

Conversions

WarpScript supports the binary Pickle format with the functions ->PICKLE and PICKLE->.

The binary Apache Arrow columnar format is also supported through the Warp 10 Arrow extension.

Note that if you are using the egress endpoint, binary data will be encoded with charset ISO-8859-1 in the JSON output.

If you are using a Py4J gateway, common types are already converted (see Py4J automatic conversions), and binary data need not an additional encoding round.

GTS to DataFrame

We provide two macros to help convert GTS into DataFrame:

• The macro @senx/python/GTStoPickledDict takes a GTS and converts it to pickled dict.
• The macro @senx/python/ListGTStoPickledDict converts a list of GTS into a pickled dict, and also joins the timestamps into a single field.

They also take a boolean as a second parameter indicating whether the classname (false) or the full selector (true) is used as the field keys of the dict.

Then after unpickling, Pandas can read the data into a dataframe with the method pandas.DataFrame.from_dict().

With the Warp 10 Arrow extension, it is also possible to use the Arrow columnar format. See for example this blog post: Conversions to Apache Arrow Format.

Wrapping Python processes inside WarpScript execution

On the other way, it is also possible to make a call to Python from WarpScript, thanks to the function CALL.

For example, in Forecast with Facebook Prophet and CALL, Fabien shows how to integrate a Python library inside WarpScript.

Read more

If you want to know more, you can have a look at the Warp 10 documentation.

In the meantime, we have updated the documentation page about Python.

Also, feel free to check other articles from our blog.

Jean-Charles Vialatte

Machine Learning Engineer