Introducing FLoWS, a functional language for Time Series Analytics

The Warp 10 Time Series Platform offers a rich analytics environment which makes it the most advanced platform of its kind. This environment relies on a library (WarpLib) of over 1000 functions and a concatenative language called WarpScript.

WarpScript is really powerful and concise. But it has proved challenging to learn for people not familiar with languages of its kind such as FORTH, PostScript, or RPL.

The most difficult part of learning WarpScript was reported as being the mental gymnastics required to visualize the parameters and results of function calls.

Well, this hard time is over. As we announced in July during the last Paris Time Series Meetup before the summer break, we are introducing a companion language to WarpScript which makes those challenges disappear.

We called this language FLoWS, for Functional Lineage of WarpScript.

It is a purely functional language, fully compatible with WarpScript. And it meant to leverage the whole of WarpLib without the steep learning curve.

You can use FLoWS wherever WarpScript can, including in the various tools with which WarpScript is integrated, such as Spark, Pig, Flink, Kafka Streams, NiFi, and many others.

This article will introduce you to the syntax of FLoWS. You will be able to quickly put it to work to solve your Time Series problems.

FLoWS 101

The syntax of FLoWS is easy to grasp, all there is to know about FLoWS fits on the rest of this page.

Comments

FloWS supports C and C++ style comments.

// C++ Style comments
/*
C-Style comments
*/

Supported types

FLoWS supports LONGs and DOUBLEs.

42        // LONG
0x2a      // LONG
3.14      // DOUBLE
1.0E-12   // DOUBLE

BOOLEANs are also supported.

true  // Not False
false // Not True

Percent encoded STRINGs using UTF-8 are enclosed in single or double quotes.

'Hello'
"%F0%9F%96%96"	// 🖖
'Multiline
Strings'

Lists are comma separated expressions enclosed in square brackets.

[ 'Hello', 3.1415, 42 ]
[ 'Hello', 3.1415,
  42	// Works on multiple lines too
]

Maps are comma separated key:value pairs enclosed in curly braces.

{ 'A':65, '@':64, 64:'@' }
{ '@':64, 64:'@',
  'A':65  // Works on multiple lines too
}

Accessing list and map elements is done using an intuitive syntax.

map['A']  // 65
map[64]   // '@'
list[0]   // 'Hello'

Operators

Simple left to right precedence with optional parentheses grouping.
Binary operators: +, +!, |, -, **, *, /, %.
Comparison and logical operators: >, <, <=, >=, ==, !=, ~=, &&, ||
Bitwise operators: &, |, ^, >>>, >>, <<

A = 5 + 3 / 2.0
X = 8 + (F(x + 1) * 3.14) - 12

Function calls

Comma separated list of expressions as function parameters. Functions can return 0 to N values.

F(1,2,'A',b)  // F is the function name
G()           // Parameterless function call

Assignments

Assignments assign values to variables.

A = 12
(x, y) = F(1)   // F MUST return two values
M[0][1] = 3.14  // Assign to list/map element

Macros

Macros are sequences of statements.

M = (a,b,c) -> {  // 3 parameters
  ...
}

Macros can return 0 to N values just like functions, use return as last statement followed by a comma separated list of values to return.

M = (a,b,c) -> {  // 3 parameters
  return F(a,b), G(c)
}

Expected number of return values can be enforced.

M = (a,b,c) -> 2 {  // MUST return 2 values
  return F(a,b), G(c)
}

You can call macros like functions, with a comma separated list of expressions as parameters. Macro name is prefixed with @ like in WarpScript.

@M(1,2,3)
(x,y) = @M(1,2,3)  // Assign return values

When using variables in macros you can either use the name of the variable, e.g. A, the variable will be replaced by its value at the time of the execution, or use the name suffixed with ! to use the value of the variable at the time of the macro definition.

Using FLoWS

FLoWS comes as a WarpScript extension. The extension adds two functions, FLOWS to execute FLoWS code and FLOWS-> to convert FLoWS code to its WarpScript counterpart.

In order to execute FLoWS code from WarpScript simply call FLOWS with a STRING parameter containing the FLoWS code to run. The result of this execution, including the modified or newly set variables, will be available to the enclosing WarpScript code.

The example below demonstrates this:

42 'B' STORE

<'
  // This is FLoWS code using variable B 
  // defined in the WarpScript code above
  A = B + 1
'>
FLOWS

// We use variable A defined in the FLoWS code
$A

All of the WarpLib functions are available to FLoWS, the example below will create a Geo Time Series and add values to it, all using the NEWGTS and ADDVALUE functions.

<'
  GTS = NEWGTS()
  ADDVALUE(GTS, 0, NaN, NaN, NaN, 1)
  ADDVALUE(GTS, 1, 48.0, -4.5, NaN, 2)
'>
FLOWS

$GTS

The FLOWS function executes the FLoWS code in the calling execution environment. This execution is performed directly without an intermediate transpilation to WarpScript. Nevertheless, if you are curious or if you need to in some specific context, you can generate WarpScript code which will perform the exact same steps as your FLoWS code by calling FLOWS->.

The generated WarpScript code contains a lot of additional code to enforce parameters and return values checks. As an example, the latter example will transpile to the following WarpScript version:

<%

  // Assignment #0
  'L1:2-1:15' SECTION
  NULL HIDE '# 0' STORE
    // NEWGTS(...)
    'L1:8-1:15' SECTION
    NULL HIDE '# 1' STORE
      NEWGTS
      1 FLOWS.ASSERTDEPTH
    '# 1' DUP LOAD SHOW FORGET
    1 FLOWS.ASSERTDEPTH
    'GTS' STORE
  '# 0' DUP LOAD SHOW FORGET
  // ADDVALUE(...)
  'L2:2-2:35' SECTION
  NULL HIDE '# 2' STORE
    $GTS
    0
    NaN
    NaN
    NaN
    1
    ADDVALUE
    CLEAR
  '# 2' DUP LOAD SHOW FORGET
  // ADDVALUE(...)
  'L3:2-3:37' SECTION
  NULL HIDE '# 3' STORE
    $GTS
    1
    48.0
    -4.5
    NaN
    2
    ADDVALUE
    CLEAR
  '# 3' DUP LOAD SHOW FORGET

%>

If you run EVAL twice on this code, you will end up with the exact same result as its FLoWS counterpart, except more operations, roughly 80, will be performed.

As a comparison, the WarpScript code that you would write manually to do the same thing as the FLoWS code would execute in 16 operations vs 61 for the FLoWS version and 143 for the WarpScript version resulting from the FLoWS transpilation. The extraneous operations add a little performance overhead. Therefore, the same program will be slightly slower in FLoWS compared with WarpScript, but nothing too dramatic.

Availability of FLoWS

FLoWS is available as a WarpScript extension on WarpFleet. You can deploy it on any Warp 10 instance running version 2.7.1 or above.

The FLoWS extension is distributed under the BSL (Business Source License), making it usable in most contexts and ensuring a license change to Apache 2.0 after a four-year period.

The Warp 10 documentation has been updated. You now have access to FLoWS signatures in every function of the doc. Examples will follow soon.

Conclusion

FLoWS brings enormous value during the first steps with Warp 10. It makes the more than a thousand WarpLib functions available without having to adapt to a syntax new to many.

We expect it will bring more people to Warp 10, some of whom will ultimately switch to WarpScript once they are accustomed to WarpLib.

Feel free to share FLoWS around you.

Read on about FLoWS with two more tutorials:

• FLoWS Basics
• FLoWS vs WarpScript

Join the Warp 10 community on Slack if you have questions or comments.

Mathias Herberts

Co-Founder & Chief Technology Officer