Build a WarpScript extension which embeds a C library

Learn how to embed Java library or C code in a WarpScript/FLoWS function to enlarge the WarpLib function corpus.

With WarpScript (and obviously FLoWS), you have access to a huge corpus of functionalities to handle your time series. But sometimes, we miss a special algorithm or a proprietary library.

You can encapsulate it in an extension dedicated to Warp 10 and gain access to new WarpScript/FLoWS functions.

We will show you how to build an extension containing a Java implementation of a function and an external C program.

Initiate your project

The easiest way is to use WarpFleet CLI.

$ npm install -g @senx/warpfleet
$ wf i
       ___       __                     _______________         _____     
       __ |     / /_____ __________________  ____/__  /___________  /_    
       __ | /| / /_  __ `/_  ___/__  __ \_  /_   __  /_  _ \  _ \  __/    
       __ |/ |/ / / /_/ /_  /   __  /_/ /  __/   _  / /  __/  __/ /_      
       ____/|__/  \__,_/ /_/    _  .___//_/      /_/  \___/\___/\__/      
                                /_/                                        ™
 version: 1.0.41
? What do you want to develop? Extension
? Enter project's group: io.warp10.ext.test
? Enter assets's name: test-extension
? Enter assets's version: 0.0.1
? Enter project's description: 
? Enter author's name: 
? Enter email address: 
? Enter license: 
? Enter GIT repo url: 
? Enter Maven repo url root: https://dl.bintray.com/myCompany/maven/
? Comma separated list of tags: 
? Is all OK, dude?
{
    "type": "ext",
    "group": "io.warp10.ext.test",
    "artifact": "test-extension",
    "version": "0.0.1",
    "description": "",
    "author": "",
    "email": "",
    "license": "",
    "git": "",
    "repoUrl": "https://dl.bintray.com/myCompany/maven/",
    "tags": [
        ""
    ]
} Yes
✔ Bootstrap generated
? test-extension/README.md already exists, do you want to overwrite it? Yes
✔ Artifact generated
✔ Asset generated

Open the test-extension folder with your favorite Java IDE.

Java implementation

Delete those two classes and create in this directory a file named MAXABS.java

package io.warp10.ext.test;

import io.warp10.script.NamedWarpScriptFunction;
import io.warp10.script.WarpScriptException;
import io.warp10.script.WarpScriptStack;
import io.warp10.script.WarpScriptStackFunction;

public class MAXABS extends NamedWarpScriptFunction implements WarpScriptStackFunction {
  public MAXABS(String name) {
    super(name);
  }

  @Override
  public Object apply(WarpScriptStack stack) throws WarpScriptException {
    int a = Integer.parseInt(stack.pop().toString());
    int b = Integer.parseInt(stack.pop().toString());
    stack.push(Math.max(Math.abs(a), Math.abs(b)));
    return stack;
  }
}

This is the implementation of a function called MAXABS which computes the maximum absolute value of 2 parameters. I agree, this is a dummy function and I do not test input parameters.

The process is easy:

extract parameters with stack.pop()
compute the maximum of absolute values
push the result back on the stack with stack.push()

Create TestWarpScriptExtension.java:

package io.warp10.ext.test;

import io.warp10.warp.sdk.WarpScriptExtension;

import java.util.HashMap;
import java.util.Map;

public class TestWarpScriptExtension extends WarpScriptExtension {
  private static final Map<String, Object> functions;

  static {
    functions = new HashMap<String, Object>();
    functions.put("MAXABS", new MAXABS("MAXABS"));
  }

  @Override
  public Map<String, Object> getFunctions() {
    return functions;
  }
}

This is the entry point of our extension. We must declare function names (ie: MAXABS) and associate them to real implementations (ie: new MAXABS("MAXABS"))

You can now package the lib:

$ ./gradlew shadowJar

Copy the resulting jar in /path/to/warp10/lib/. activate your extension in /path/to/warp10/etc/conf.d/70--extensions.conf:

warpscript.extension.test=io.warp10.ext.test.TestWarpScriptExtension

Finally, restart your Warp 10 instance and test your new MAXABS function.

C implementation

Create test-extension/src/main/c/helloworld.h:

char *hello(char* ch);

Create test-extension/src/main/c/helloworld.c:

#include "helloworld.h"
#include <stdlib.h>
#include <stdio.h>
#include <string.h>

char* hello(char* ch) {
    char prefix[] = "Hello ";
    char *res;
    res = malloc(strlen(prefix) + strlen(ch));
    strcpy(res, prefix);
    strcat(res,ch);
    return res;
}

Disclaimer: I haven't coded with C since ages, so sorry for that

Hey wait, there is a malloc without a free… Ok, we will see how to free later.

Compile it:

$ cd src/main/c 
$ gcc -c helloworld.c 
$ gcc -shared -dynamiclib helloworld.o -o ../resources/helloworld.so

Now we will bind this small program with our extension thanks to JNA.

Add the dependence to your build.gradle:

implementation group: 'net.java.dev.jna', name: 'jna', version: '5.10.0'

Create interface CHelloWorld.java:

package io.warp10.ext.test;

import com.sun.jna.Library;
import com.sun.jna.Native;
import com.sun.jna.Pointer;

public interface CHelloWorld extends Library {
  CHelloWorld INSTANCE = Native.load("helloworld.so", CHelloWorld.class);

  Pointer hello(String g);
  void free(Pointer p);  // JNA provides it 
}

Here is our binding, now, we have to embed it into a new function: HELLO.class

package io.warp10.ext.test;

import io.warp10.script.NamedWarpScriptFunction;
import io.warp10.script.WarpScriptException;
import io.warp10.script.WarpScriptStack;
import io.warp10.script.WarpScriptStackFunction;

public class HELLO extends NamedWarpScriptFunction implements WarpScriptStackFunction {
  public HELLO(String name) {
    super(name);
  }

  @Override
  public Object apply(WarpScriptStack stack) throws WarpScriptException {
    Pointer ptr = null;
    try {
      String str = stack.pop().toString();
      ptr = CHelloWorld.INSTANCE.hello(str); // invoke the C function
      stack.push(ptr.getString(0));
    } catch (Exception e) {
      throw new WarpScriptException(e);
    } finally {
      if (null != ptr) {
        CHelloWorld.INSTANCE.free(ptr); // Free the allocated memory
      }
    }
    return stack;
  }
}

We use the same principle as a standard java library. And the declare this function:

public class TestWarpScriptExtension extends WarpScriptExtension {
  private static final Map<String,Object> functions;
  static {
    functions = new HashMap<String,Object>();
    functions.put("MAXABS", new MAXABS("MAXABS"));
    functions.put("HELLO", new HELLO("HELLO")); // <----- new code
  }

Re-build your jar, deploy it, restart Warp 10 and test it: