Introducing the Warp 10 Accelerator for blazing fast data access and improved analytics. Enabling unbelievable time series data applications.
Warp 10 is the most advanced time series platform, but it also stands as one of the fastest. With release 2.5.0, thanks to a new feature called the Accelerator, Warp 10 is becoming ever faster. Learn how the Warp 10 Accelerator works and what benefits it can bring to your time series data.
What is the accelerator?
The Warp 10 Accelerator is an option of the standalone version of Warp 10 that adds an in-memory cache to a Warp 10 instance. That cache covers a certain period of time and can be used to store data for ultra-fast access.
What is it good for?
The number of use cases that can benefit from the Warp 10 Accelerator is very large. From keeping the most recent data in-memory for monitoring applications to having a fast index when storing geolocations, the benefit of this duality of storage media is opening possibilities that were more difficult to achieve before.
The fine grain control on which parts of this dual media set up are used for update, retrieval and deletion requests is key to solving complex problems.
Enabling the Warp 10 Accelerator
The Warp 10 Accelerator is useable in any standalone instance using disk-based storage. It is simply configured using the following keys:
accelerator = true accelerator.chunk.length = 3600000000 accelerator.chunk.count = 3
The first line instructs Warp 10 to enable the Accelerator. The last two lines define the amount of time the Accelerator will cover. In the above case, it will manage 3 chunks of 1 hour (3600s expressed in your platform time units, here in microseconds). A timestamp belongs to a chunk with id
timestamp / accelerator.chunk.length. The accelerator manages the chunk of the current timestamp and
accelerator.chunk.count - 1 other chunks before it. It does not manage any future chunks.
When a chunk is no longer with the
accelerator.chunk.count current ones, it is dropped.
Upon startup, the accelerator can be instructed to load data from persistent storage to populate the accelerated (in-memory) time range. This is done using the following configuration keys:
// // Set to 'true' to preload the accelerator with the persisted data spanning the accelerator time range. // Preloading can be disabled for setups where the accelerator is used as a temporary side cache only. // accelerator.preload = true // // Number of threads to use for preloading the Warp 10 Accelerator // accelerator.preload.poolsize = 8 // // Size of GTS batches to process when preloading the Warp 10 Accelerator // accelerator.preload.batchsize = 1000 // // Set to true to take into account the last activity of Geo Time Series when preloading the Warp 10 Accelerator // accelerator.preload.activity = false
Note that data beyond the most current chunk will not be loaded later when the chunks they fall in come into the current chunks window.
Using the Warp 10 Accelerator
The use of the accelerator is automatic once it is enabled. Writes for data whose timestamps fall within the accelerated time range will be written both on disk and in memory.
When retrieving data,
FETCH operations with a time range included in the accelerated range will be served from memory, others will use the disk store.
If you want more control over the store used by
DELETE calls (and their counterpart
/delete endpoints), you can use the new functions
ACCEL.NOPERSIST to instruct Warp 10 to use or ignore the in-memory or disk stores. For example, to issue a
FETCH which will only access the disk store, use the following:
ACCEL.NOCACHE ACCEL.PERSIST [ ..... ] FETCH
When using the
/updateendpoint, the header
X-Warp10-Accelerator can be set to a combination of
nopersist to disable the in-memory and/or disk based stores.
When using the
/delete endpoints, the parameters
nopersist can be used to disable the in-memory and/or disk stores.
The Warp 10 Accelerator is a new feature introduced in release 2.5.0. It enables dual storage deployment of standalone instances of Warp 10, mixing in-memory and disk based persistent stores to boost the speed at which recent data can be accessed.
Configuration of the Accelerator is very simple and benefit is transparent, no need to modify your existing WarpScript code.
Please let us know how the Accelerator changed your life!
Be safe, stay at home – (for our readers from the future, half of humanity is currently under lock down due to the COVID-19 outbreak).
Warp 10 Raspberry Pi 4 benchmark for industrial IoT
SELECT x WHERE system is not lagging
2 Fast 2 Curious: JMH Benchmarks in WarpScript
Co-Founder & Chief Technology Officer