The Py4J plugin for Warp 10

The Py4J plugin allows a Python script to interact with a Warp 10 instance through the Py4J protocol.

• 1. Installation
• 2. Basic example
• 3. Configuration options
• 4. Without a running Warp 10 instance
• Conclusion

1. Installation

(Outdated: now you can simply use WarpFleet to install this plugin)

First, clone the GitHub repository of the plugin.

git clone https://github.com/senx/warp10-plugin-py4j/

Then, compile and place the produced jar in the lib folder of your Warp 10 platform.

cd warp10-plugin-py4j
./gradlew ShadowJar
cp build/libs/*.jar <Warp10_home>/lib/

Next, declare the Py4J plugin in the Warp 10 configuration file.

warp10.plugin.py4j = io.warp10.plugins.py4j.Py4JWarp10Plugin

Then, if you want to be able to fetch data stored in Warp 10 from Python, set the following property to true in <Warp10_home>/etc/conf.d/10-egress.conf:

egress.clients.expose = true

After that, restart your Warp 10 instance for the plugin to be taken into account.

Congratulations, you can now interact with Warp 10 from a Python script!

If the latter is not installed, you can install it with pip install py4j (or conda install py4j if you use conda).

Also, in the log file <Warp10_home>/logs/warp10.log, the following line should have appeared:

LOADED plugin 'io.warp10.plugins.py4j.Py4JWarp10Plugin'

If it is not the case, it is probably because the log configuration file is not properly set up (per default, that was the case if the initial revision of Warp 10 you installed was between version 2.0.3 and 2.4).

To fix it, append the following lines to the file <Warp10_home>/etc/log4j.properties then restart the platform.

log4j.additivity.io.warp10.warp.sdk.AbstractWarp10Plugin = false
log4j.logger.io.warp10.warp.sdk.AbstractWarp10Plugin = INFO, stdout
log4j.additivity.io.warp10.script.WarpScriptLib = false
log4j.logger.io.warp10.script.WarpScriptLib = INFO, stdout

2. Basic example

The Py4J plugin launches a gateway in the same JVM running Warp 10. Instances of Java objects are accessible from Python through this gateway. By default, the gateway address is 127.0.0.1 and its port is 25333.

The following example shows how to connect to the running gateway:

from py4j.java_gateway import (JavaGateway, GatewayParameters)

addr = "127.0.0.1"
port = 25333

# @see https://www.py4j.org/advanced_topics.html
#collections-conversion
params = GatewayParameters(addr, port, auto_convert=True)

# Connect to the gateway created by the plugin
gateway = JavaGateway(gateway_parameters=params)

The plugin allows to create WarpScript stack instances through the gateway like below:

stack = gateway.entry_point.newStack()

In addition, the stack can be accessed and its methods invoked from Python. For example:

# Push objects onto the stack
stack.push(0)
stack.push([1,2,3,4])

# Execute WarpScript
stack.execMulti("<% + %> FOREACH")

# Extract top of the stack and print it
print(stack.pop())

10

# Print fields and methods
print(dir(stack))

["checkBalanced", "clear", "define", "depth", "drop", "dropn", "dump", "dup", "dupn", "equals", "exec", "execMulti", "find", "forget", "get", "getAttribute", "getClass", "getCounter", "getDefined", "getDirectoryClient", "getGeoDirectoryClient", "getStoreClient", "getSubStack", "getSymbolTable", "getUUID", "hashCode", "inMultiline", "incOps", "isAuthenticated", "load", "maxLimits", "notify", "notifyAll", "peek", "peekn", "pick", "pop", "popn", "progress", "push", "reset", "restore", "roll", "rolld", "rot", "run", "save", "setAttribute", "store", "swap", "toString", "wait"]

3. Configuration options

When the Py4J plugin is used, it adds these additional configuration properties to Warp 10 (values given are the defaults):

# Address and port of the gateway
py4j.host=127.0.0.1
py4j.port=25333

# Address and port used for Python callbacks
py4j.python.host=127.0.0.1
py4j.python.port=25334

# Timeout specifications
py4j.timeout.read=0
py4j.timeout.connect=0

# Whether to use imposed limits on the stack or not
py4j.stack.nolimits=false

# If set, GatewayParameters must set the argument auth_token to this value to authorize the connection
#py4j.authtoken= 

As already mentioned, if you want to be able to fetch data from a stack created through a gateway, you must modify the Warp 10 configuration property egress.clients.expose and set it to true.

4. Without a running Warp 10 instance

You can also use WarpScript from Python without the need to have a running Warp 10 instance (however, you won't be able to use the functions FETCH, FIND, or FINDSTATS).

To do that, the Py4j library provides the function launch_gateway that starts a gateway server in a JVM from Python.

from py4j.java_gateway import launch_gateway
from py4j.java_gateway import JavaGateway
from py4j.java_gateway import GatewayParameters

#
# Arguments of launch_gateway
#
ENABLE_AUTH = True
JAVA_OPTS = []

# specific java option
WARP10_JAR = # insert the path of your warp10 jar here
PY4J_JAR = # insert the path of the java part of the py4j library here. The jar of the py4j warp10 plugin can do as well.
JAVA_CLASSPATH = WARP10_JAR + ':' + PY4J_JAR # the java classpath. You can include any other java library you would need.

#
# Launch the gateway server in a JVM and retrieve its port
#
if ENABLE_AUTH:
    (port, token) = launch_gateway(enable_auth=ENABLE_AUTH,
                                   die_on_exit=True,
                                   javaopts=JAVA_OPTS,
                                   classpath=JAVA_CLASSPATH)
else:
    port = launch_gateway(enable_auth=ENABLE_AUTH,
                          die_on_exit=True,
                          javaopts=JAVA_OPTS,
                          classpath=JAVA_CLASSPATH)
    token = None

#
# Connect to it
#

gateway = JavaGateway(gateway_parameters=GatewayParameters(auto_convert=True,
                                                           port=port,
                                                           auth_token=token))

Then, you will be able to instanciate an entry point to a WarpScript stack and call the method newStack like in the previous example.

# Create an entry point with class Py4JEntryPoint included in the warp 10 jar
conf = {}

# holds the Warp10 configuration properties
conf["warp.timeunits"] = "us"
entry_point = gateway.jvm.io.warp10.Py4JEntryPoint(conf)

# create a new stack
stack = entry_point.newStack()

Conclusion

From now on, you should be able to launch a WarpScript stack from Python, and to go back and forth between Python and WarpScript functions.

The WarpScript reference is available on the Warp 10 website.

For more information on the possibilities of Py4J, you can see their website.

In the next blog post, we will see how to efficiently use WarpScript in Jupyter.

Jean-Charles Vialatte

Machine Learning Engineer