SELECT x WHERE system is not lagging

Thinking in WarpScript Dear Santa

Lagging effect on derivatives

Non real-time systems may lag. This has no effect if your system just stores data: you will end up with missing data points which could be easily handled later.

Now, imagine that your system also computes the longitudinal acceleration of a vehicle. Lower layer software sends data every 100ms to the software. The software computes (wheelspeedN-1 – wheelspeedN)/100ms, applies a low-pass filter, then stores the result. But if there is a lag in the meantime, N-1 will represent data 1s before, instead of 100ms before, and the result will be 10 times too high.

That is what happened with car data I am currently working with. Sometimes, in very rare circumstances, lags happen and the longitudinal acceleration turns wrong:

Outlier spotted

Advice: turn on the “dots” when you work with unknown data. In VSCode WarpScript extension, there is an “always show dots” option.

Showing dots is useful to check data regularity

Around 17:16:12, the 1.5 lag introduces an incorrect value. Moreover, since the software that produced the data applied a low pass filter, neighboring datapoints are also affected.

Ideally, a few datapoints must be discarded before the lag is detected, and around 30 points after the lag (it takes about 20 samples to converge again). How could you do this in WarpScript™?

First, we will detect the lags by computing delta between ticks, then we will discard bad datapoints.

Compute delta between ticks

[ $gLongGTS mapper.tick 0 0 0 ] MAP //put tick in value
[ SWAP 2 2 STRICTMAPPER 1 0 0 ] MAP //delta between ticks

First line: put the tick in the value. mapper.tick output is a GTS with value=tick.

Second line : computes the delta between the last and first values (delta = last – first) of the sliding window. For the very first tick, the sliding window will contain one point, and the result will be zero. The best way to avoid this start condition is to transform the mapper with a STRICTMAPPER that can take a minimum of 2 datapoints and a maximum of 2 datapoints.

My platform is in microsecond. the maximum lag is 1.6s !

Build a validity indicator

Using the previous GTS, we want to obtain a GTS that only keeps the ticks that are valid. Thus, we will be able to use this GTS as a validity indicator.

If a tick has a delta above a certain threshold, we consider it to be invalid along with 10 ticks before, and 30 ticks after (since they have been impacted by the low-pass filter). So, in the following warpscript, we associate the bad delta value to those ticks too, using mapper.max.

[ SWAP mapper.max 30 10 0 ] MAP //maximum delta in a moving window
Apply a max on a moving window. Input GTS is orange and result of MAP is red.

Reverse your brain, be the mapper: if you want to find a bad delta 10 points before, it means your moving window should look 10 points after the current tick. That’s why the “post” parameter of MAP is 10. Same logic with the “pre” parameter.

Now, I just want to keep points less than a threshold of 400ms:

[ SWAP 400 ms 0 0 0 ] MAP //keep delta under 400ms
Red points : Data is OK

Keep valid points

Since Warp 10™ 2.0, you can do arithmetics on GTS directly:

$gtsA $gtsB +

For each tick, if both gtsA and gtsB have a value for this tick, it returns a value (the sum of gtsA and gtsB values).

Read the previous sentence again… again… and try to formulate the contraposition: to remove values from a GTS, you just need to add a GTS with value 0 on each tick you want to keep. Back to the complete example:

[ $gLongGTS  1439572577413977 12061287 TIMECLIP mapper.tick 0 0 0 ] MAP 
[ SWAP 2 2 STRICTMAPPER 1 0 0 ] MAP //delta between ticks
[ SWAP mapper.max 30 10 0 ] MAP //maximum delta in a moving window
[ SWAP 400 ms 0 0 0 ] MAP //keep delta under 400ms
0 GET  0 * //take first gts in the list output, multiply all values by 0
'noLagIndicator' RENAME  'noLagGTS' STORE
Now, add both GTS…

The red curve are the ticks I want to keep, with value 0. I will add it to my original data:

$gLongGTS $noLagGTS +
Blue : data filtered

Here is the full sequence with lagging effect ignored:


SELECT ... WHERE ... efficient equivalent in WarpScript™ is just a GTS sum.

SELECT ... WHERE a AND b AND c efficient equivalent in WarpScript™ is just several GTS sums:

[ $gearGTS $gear mapper.eq 0 0 0 ] MAP 0 GET //keep points with this gear ratio
0 * 'gearSelectorGTS' STORE //multiply value per zero

[ $throttleGTS 90 0 0 0 ] MAP 0 GET //keep points where throttle pedal is pressed
0 * 'throttleSelectorGTS' STORE //multiply value per zero

//keep datapoints for current gear, for throttle over 90%, and no lag :
$rpmGTS $gearSelectorGTS + $throttleSelectorGTS + $noLagGTS + 
'rpm' STORE

This method is the most efficient way I found in WarpScript to remove invalid data. Applied to 2500 cars, the acceleration diagram (max acceleration vs rpm for a given gear, full throttle) is now clean:

The buggy one with rare lags…

The fix took… Well… 7 lines of WarpScript™!

The corrected one