Fetch the first data point in Warp 10

In Warp 10, finding the first tick of a time series can be tricky. This article shows you a simple solution with WarpScript.

When you fetch data, Warp 10 FETCH always starts from the end and walks back in time. Fetching the last data point is easy. Fetching the first data point is less obvious. As you asked us for a solution, here is a way to do it.

Learn more about FETCH function.

FETCH basics

[ $ReadToken "classname" {} NOW -1 ] FETCH will fetch the last point of a stored GTS. To do this, Warp 10 needs to open the right file and decompress the data. Fetching one point or one thousand points does not make a big difference.

The idea to find the first point is to fetch 1000 points one month ago, if there are 1000 points, fetch 1000 points two months ago, or before the first tick of the points, you found… until you do not find 1000 points. Then make a dichotomy until you find strictly less than 1000 points, and return the first one.

Implementation

The macro takes 4 parameters:

The read token
A GTS that will be your selector (possibly empty)
An offset to define the time step you will jump (it depends on your data)
A batch size (1000 is a good one)

If you know your data is around 1 million per month for a few years, a one-month time step and a 10000 data points batch size are reasonable. It really depends on your data.

This macro is useful… So we made it available on WarpFleet resolver. You can use it with @senx/gts/first. You can review the code here.

Going further with attributes

If you never write in the past, and you often need the first data point for your analysis, you can store the first timestamp in a GTS attribute. It will be ready for the next time you need the first point of your GTS!

On a one million point series, this method fetches 4006 data points on the first iteration and the second iteration is immediate.

All these operations could be done on a list of GTS, with LMAP or a FOREACH.

Discover the Lambda functions and use the power of WarpScript in simple JSON API deployments.

Alternatively, you can set an attribute after you push the first point of your GTS (for example, with the meta endpoint). But beware that if you delete the first point or write in the past, the value of this attribute might not reflect what you think it should (unless you update it).

Conclusion

If you know your series is small enough, FETCH all data points and call FIRSTTICK, then store the first tick in the series attributes. If you don't know exactly, use @senx/gts/first macro.