Warp 10 Accelerator – beyond fast!

Warp 10 is the most advanced time series platform, but it also stands as one of the fastest. With release 2.5.0, thanks to a new feature called the Accelerator, Warp 10 is becoming ever faster. Learn how the Warp 10 Accelerator works and what benefits it can bring to your time series data.

What is the accelerator?

The Warp 10 Accelerator is an option of the standalone version of Warp 10 that adds an in-memory cache to a Warp 10 instance. That cache covers a certain period of time and can be used to store data for ultra-fast access.

What is it good for?

The number of use cases that can benefit from the Warp 10 Accelerator is very large. From keeping the most recent data in-memory for monitoring applications to having a fast index when storing geolocations, the benefit of this duality of storage media is opening possibilities that were more difficult to achieve before.

The fine grain control on which parts of this dual media set up are used for update, retrieval and deletion requests is key to solving complex problems.

Enabling the Warp 10 Accelerator

The Warp 10 Accelerator is useable in any standalone instance using disk-based storage. It is simply configured using the following keys:

accelerator = true
accelerator.chunk.length = 3600000000
accelerator.chunk.count = 3

The first line instructs Warp 10 to enable the Accelerator. The last two lines define the amount of time the Accelerator will cover. In the above case, it will manage 3 chunks of 1 hour (3600s expressed in your platform time units, here in microseconds). A timestamp belongs to a chunk with id timestamp / accelerator.chunk.length. The accelerator manages the chunk of the current timestamp and accelerator.chunk.count - 1 other chunks before it. It does not manage any future chunks.

When a chunk is no longer with the accelerator.chunk.count current ones, it is dropped.

Upon startup, the accelerator can be instructed to load data from persistent storage to populate the accelerated (in-memory) time range. This is done using the following configuration keys:

//
// Set to 'true' to preload the accelerator with the persisted data spanning the accelerator time range.
// Preloading can be disabled for setups where the accelerator is used as a temporary side cache only.
//
accelerator.preload = true

//
// Number of threads to use for preloading the Warp 10 Accelerator
//
accelerator.preload.poolsize = 8

//
// Size of GTS batches to process when preloading the Warp 10 Accelerator
//
accelerator.preload.batchsize = 1000

//
// Set to true to take into account the last activity of Geo Time Series when preloading the Warp 10 Accelerator
//
accelerator.preload.activity = false

Note that data beyond the most current chunk will not be loaded later when the chunks they fall in come into the current chunks window.

Using the Warp 10 Accelerator

The use of the accelerator is automatic once it is enabled. Writes for data whose timestamps fall within the accelerated time range will be written both on disk and in memory.

When retrieving data, FETCH operations with a time range included in the accelerated range will be served from memory, others will use the disk store.

If you want more control over the store used by UPDATE, FETCH and DELETE calls (and their counterpart /update, /fetch and /delete endpoints), you can use the new functions ACCEL.CACHE, ACCEL.NOCACHE, ACCEL.PERSIST and ACCEL.NOPERSIST to instruct Warp 10 to use or ignore the in-memory or disk stores. For example, to issue a FETCH which will only access the disk store, use the following:

ACCEL.NOCACHE
ACCEL.PERSIST
[ ..... ] FETCH

When using the /updateendpoint, the header X-Warp10-Accelerator can be set to a combination of nocache and nopersist to disable the in-memory and/or disk based stores.

When using the /fetch or /delete endpoints, the parameters nocache and nopersist can be used to disable the in-memory and/or disk stores.

The ACCEL.REPORT function will return information about the Warp 10 Accelerator, like its configuration and whether it was used for the last FETCH operation.

Takeaways

The Warp 10 Accelerator is a new feature introduced in release 2.5.0. It enables dual storage deployment of standalone instances of Warp 10, mixing in-memory and disk based persistent stores to boost the speed at which recent data can be accessed.

Configuration of the Accelerator is very simple and benefit is transparent, no need to modify your existing WarpScript code.

Please let us know how the Accelerator changed your life!

Be safe, stay at home – (for our readers from the future, half of humanity is currently under lock down due to the COVID-19 outbreak).

Mathias Herberts

Co-Founder & Chief Technology Officer