Working with Geo Data in Warp 10

One of the major features of the Warp 10 platform is its support for location data both in its core data model, Geo Time Series, and in the WarpScript data programming language. This one includes many functions for working with geographic information.

This post will walk you through the location data support included in Warp 10. It will explain some advanced features you can use for solving your geographic data problems.

Location encoding

Storing latitude and longitude as floating-point numbers is straightforward. But unfortunately, it does not ease processes such as geofencing or spatial indexing. In order to be efficient for those operations, Warp 10 stores location data in a format called HHCode, for Helical Hyperspatial Code. HHCodes were invented in the 90s by Herman Varma for storing bathymetric information at the Canadian Hydrographic Service. Later they became the basis for Oracle's SDO (Spatial Data Object) cartridge.

HHCodes can encode multiple dimensions, not just a location. At the core of HHCodes is a space-filling curve. A Z-order Curve, or Morton code depending on how you want to call it. The HHCode implementation used in Warp 10 uses a Z-order Curve which starts in the lower-left corner and uses the 16 even levels from 2 to 32.

At level 2, the space is divided into 16 cells. At level 4 into 256 levels (16×16), and at level N into 2^2N. So at finest level 32, space is divided into 2**64 cells, or 18,446,744,073,709,551,616 (roughly 18 billion of billions). We use a hexadecimal naming scheme for cells.

The benefit of this hexadecimal notation is that by using only P hexadecimal digits, you get the cell name at level 2P. You can also convert easily the name to a LONG.

You can experiment to convert latitude and longitude to HHCodes and vice-versa with the WarpScript functions ->HHCODE, ->HHCODELONG and HHCODE->.

Scales

The following table gives approximate scales at the various HHCode levels (resolution). Scale in meters is the width of each cell at the equator. The height of each cell is half that and does not change with latitude.

+---------------------------+-----------+
| Resolution (HHCode level) |   Scale   |
+---------------------------+-----------+
|              2            | 10,000 km |
|              4            |  2,500 km |
|              6            |    625 km |
|              8            |    156 km |
|             10            |     39 km |
|             12            |     10 km |
|             14            |    2.5 km |
|             16            |    600 m  |
|             18            |    150 m  |
|             20            |     40 m  |
|             22            |     10 m  |
|             24            |    2.5 m  |
|             26            |     60 cm |
|             28            |     15 cm |
|             30            |      4 cm |
|             32            |      1 cm |
+---------------------------+-----------+

Working with areas

The next step in working with geo data is to work with zones covering a certain area of the Earth. In WarpScript, areas are called GEOSHAPE. A GEOSHAPE is a collection of HHCode cells and associated resolution. The resolution is stored as a prefix of the cells. So for this reason, GEOSHAPEs can only contain cells down to level 32 in order for the cells to be representable on 64bits.

You can create a GEOSHAPE from a WKT or GeoJSON description. The WarpScript functions GEO.WKT and GEO.JSON are the ones to use.

You can then combine multiple GEOSHAPEs using set algebra with the help of the WarpScript functions GEO.INTERSECTION, GEO.UNION and GEO.DIFFERENCE. This allows you to build complex shapes that you cannot easily express in either WKT or GeoJSON.

Indexing and searching Geo Time Series

You care for the location of your sensors and wish to be able to select them according to their position? So you need to index the associated Geo Time Series according to latitude and longitude.

In Warp 10, the most efficient way of doing this is by adding an attribute to your Geo Time Series with a value set to the HHCode of the location of the sensor. We recommend you use an attribute instead of a label. In this way, you don't have to know the value of the HHCode to push data into the GTS. And you will be able to modify the location should your sensor move at a later time.

When your Geo Time Series have their location stored as an HHCode in an attribute, you can create a GEOSHAPE and use it as a selector for identifying matching Geo Time Series. The GEO.REGEXP WarpScript function will emit a regular expression which will match all HHCodes included in the input GEOSHAPE, you can then use this regular expression as a label selector in a FETCH or FIND call, the matching attribute (or label if you did not follow our recommendation above).

Conclusion

You should now have a better understanding of how you can manage location in Warp 10. And you have learned how you can index and search Geo Time Series. In a future post, we will go beyond indexing the Geo Time Series by covering an advanced technique of spatio-temporal indexing of the content of your GTS.

Mathias Herberts

Co-Founder & Chief Technology Officer