The Py4J plugin for Warp 10™

tutorial Python

The Py4J plugin allows a Python script to interact with a Warp 10™ instance through the Py4J protocol.

1. Installation

First, clone the GitHub repository of the plugin.

git clone https://github.com/senx/warp10-plugin-py4j/

Then, compile and place the produced jar in the lib folder of your Warp 10 platform.

cd warp10-plugin-py4j
./gradlew ShadowJar
cp build/libs/*.jar <Warp10>/lib/

Next, declare the Py4J plugin in the Warp 10 configuration file.

warp10.plugin.py4j = io.warp10.plugins.py4j.Py4JWarp10Plugin

Then, if you want to be able to fetch data stored in Warp 10 from Python, set the following property to true:

egress.clients.expose = true

After that, restart your Warp 10™ instance for the plugin to be taken into account.

In the log file <Warp10>/logs/warp10.log, check that the following line appears:

LOADED plugin 'io.warp10.plugins.py4j.Py4JWarp10Plugin'

Congratulations, you can now interact with Warp 10 from a Python script.

If the latter is not installed, you can install it with pip install py4j (or conda install py4j if you use conda).

2. Basic example

The Py4J plugin launches a gateway in same JVM running Warp 10™. Instances of Java objects are accessible from Python through this gateway. By default, the gateway address is 127.0.0.1 and its port is 25333.

The following example shows how to connect to the running gateway:

from py4j.java_gateway import (JavaGateway, GatewayParameters)

addr = '127.0.0.1'
port = 25333

# @see https://www.py4j.org/advanced_topics.html#collections-conversion
params = GatewayParameters(addr, port, auto_convert=True)

# Connect to the gateway created by the plugin
gateway = JavaGateway(gateway_parameters=params)

The plugin allows to create WarpScript™ stack instances through the gateway like below:

stack = gateway.entry_point.newStack()

In addition, the stack can be accessed and its methods invoked from Python. For example:

# Push objects onto the stack
stack.push(0)
stack.push([1,2,3,4])

# Execute WarpScript
stack.execMulti('<% + %> FOREACH')

# Extract top of the stack and print it
print(stack.pop())
10
# Print fields and methods
print(dir(stack))
['checkBalanced', 'clear', 'define', 'depth', 'drop', 'dropn', 'dump', 'dup', 'dupn', 'equals', 'exec', 'execMulti', 'find', 'forget', 'get', 'getAttribute', 'getClass', 'getCounter', 'getDefined', 'getDirectoryClient', 'getGeoDirectoryClient', 'getStoreClient', 'getSubStack', 'getSymbolTable', 'getUUID', 'hashCode', 'inMultiline', 'incOps', 'isAuthenticated', 'load', 'maxLimits', 'notify', 'notifyAll', 'peek', 'peekn', 'pick', 'pop', 'popn', 'progress', 'push', 'reset', 'restore', 'roll', 'rolld', 'rot', 'run', 'save', 'setAttribute', 'store', 'swap', 'toString', 'wait']

3. Configuration options

When the Py4J plugin is used, it adds these additional configuration properties to Warp 10™ (values given are the defaults):

# Address and port of the gateway
py4j.host = '127.0.0.1'
py4j.port = 25333

# Address and port used for Python callbacks
py4j.python.host = '127.0.0.1'
py4j.python.port = 25334

# Timeout specifications
py4j.timeout.read = 0
py4j.timeout.connect = 0

# Whether to use imposed limits on the stack or not
py4j.stack.nolimits = false

# If set, GatewayParameters must set the argument auth_token to this value to authorize the connection
#py4j.authtoken = 

As already mentioned, if you want to be able to fetch data from a stack created through a gateway, you must modify the Warp 10™ configuration property egress.clients.expose and set it to true.

4. Without a running Warp 10™ instance

You can also use WarpScript™ from Python without the need to have a running Warp 10™ instance (however you won’t be able to use the functions FETCH, FIND, or FINDSTATS).

To do that, the Py4j library provides the function launch_gateway that starts a gateway server in a JVM from Python.

from py4j.java_gateway import launch_gateway
from py4j.java_gateway import JavaGateway
from py4j.java_gateway import GatewayParameters
#
# Arguments of launch_gateway
#

ENABLE_AUTH=True
JAVA_OPTS = [] # specific java option
WARP10_JAR = # insert the path of your warp10 jar here
PY4J_JAR = # insert the path of the java part of the py4j library here. The jar of the the py4j warp10 plugin can do as well.
JAVA_CLASSPATH = WARP10_JAR + ':' + PY4J_JAR # the java classpath. You can include any other java library you would need.

#
# Launch the gateway server in a JVM and retrieve its port
#

if ENABLE_AUTH:
 (port,token) = launch_gateway(enable_auth=ENABLE_AUTH,die_on_exit=True,javaopts=JAVA_OPTS,classpath=JAVA_CLASSPATH)
else:
 port = launch_gateway(enable_auth=ENABLE_AUTH,die_on_exit=True,javaopts=JAVA_OPTS,classpath=JAVA_CLASSPATH)
 token = None

#
# Connect to it
#

gateway = JavaGateway(gateway_parameters=GatewayParameters(auto_convert=True,port=port,auth_token=token))

Then, you will be able to instanciate an entry point to a WarpScript™ stack and call the method newStack like in the previous example.

# Create an entry point with class Py4JEntryPoint included in the warp 10 jar
conf = {} # holds the Warp10 configuration properties
conf['warp.timeunits'] = 'us'
entry_point = gateway.jvm.io.warp10.Py4JEntryPoint(conf)

# create a new stack
stack = entry_point.newStack()

Conclusion

From now on, you should be able to launch a WarpScript™ stack from Python, and to go back and forth between Python and WarpScript functions.

The WarpScript reference is available on the Warp 10™ website.

For more information on the possibilities of Py4J, you can see their website.

In the next blog post, we will see how to efficiently use WarpScript™ in Jupyter.

Share