Are you getting started with the Warp 10 Platform? This blog and the Warp 10 documentation are here to guide you on your journey to become a Warp 10 expert.
Warp 10 is an open source platform designed to collect, store, and analyze sensor / IoT data and any kind of sequence data in a horizontal and industrial perspective.
If you haven’t done so already, we suggest starting with the onboarding tutorials to learn the basics of Warp 10 and WarpScript, then reading the Getting Started page to deploy your own instance.
Where to start?
Here are some resources that will help you to begin with.
Basic concepts
What is a Geo Time Series (GTS)?
A Geo Time Series (GTS) is a time series with a location associated with the value. For example, timestamp / latitude / longitude / elevation / value.
The Warp 10 supported values are:
- boolean (true/false)
- long (ie: 1123456)
- doubles (ie: 15.158)
- strings
- multivariates
Some metadata are associated with a GTS:
- class name (ie:
io.senx.temperature) which represents what is measured - labels: key/value pairs used to specify, for example, the data source (ie:
{serial=123456,cutomerID=456789,unit=kelvin}). Labels are immutable during the lifetime of the series. - attributes are like labels, except they are mutable. (Warp 10 does not keep track of changes)
So it can be represented like that:
When you push data, use the "compact input format":TS/LAT:LON/ELEV NAME{LABELS}{ATTRIBUTES} VALUE
And when you fetch data, the output will be in JSON.
There could be some missing data for latitude, longitude, and elevation, this is supported by the platform (for example, while the GPS is not yet initialized).
Read on
- Master GTS: follow this tutorial to learn how to retrieve, create, modify, store GTS.
- Cyclones on Sandbox: follow this series of tutorials to learn how to upload and analyze real data on the Warp 10 Sandbox
- Discover the list of articles dealing with geolocated data on the blog, with the Geo Time Series tag.
- All about the Geo Time Series Model: understanding the GTS model is the first step in mastering the Warp 10 platform. This model is quite simple, yet powerful, and this post will go through most of its aspects
How to manipulate GTS and data?
The way to manipulate GTS is to use the WarpScript language:
- by posting an HTTP request
- by using WarpStudio
- by using VSCode (recommended configuration here)
WarpScript & FLoWS
WarpScript is a powerful programming language, data-flow oriented. FLoWS is a functional language. It is an alternative to WarpScript for users discovering the Warp 10 Platform.
WarpScript and FLoWS and can interact with the library of more than 1000 functions designed for time series analysis.
- A gentle introduction to WarpScript: Discover the basics of the WarpScript language
- WarpScript 101: about the syntax: WarpScript syntax can look alien at first. We discuss the thoughtful reasons for its design and see how easy it is to get used to it.
- Get started with WarpScript thanks to the Best practices tutorial
- Introducing FLoWS, a functional language for Time Series Analytics: This article will introduce you to the syntax of FLoWS. You will be able to quickly put it to work to solve your Time Series problems.
- Discover the basics concepts of FLoWS and follow this tutorial
- Learn more about WarpScript on the blog with the Thinking in WarpScript category
How to fetch data
- Within a time range: The FETCH function allows you to extract your data from Warp 10 in different manners.
- In a geographical area: Extract data within a geographical area.
- Fetch the first data point: In Warp 10, finding the first tick of a time series can be tricky. This article shows you a simple solution with WarpScript.
- Find and explore your data: Get to know the Warp 10 Directory and discover functions that query it.
Data manipulation
Frameworks and utilities
Discover the essential frameworks: we identified a few frameworks that allow performing all the usual time series processing, and of course, these frameworks are built into WarpScript. Note that we call framework a WarpScript function that has a function among its arguments.
Geographical manipulation
- Working with Geo Data in Warp 10: Discover how location is encoded in Warp 10 and how you can search your Geo Time Series.
- Find data in a specific area: Extract data within a geographical area.
- Spatio-temporal indexation: Learn how to index spatio-temporal data in Warp 10 so you can query your series by geographic area and time range.
- Motion Split: The easiest way to detect motion and to split a Geo Time Series accordingly. Split with distance between two timestamps, detect stopped/moving state, etc.
Usual manipulation functions
- Find which timestamp a list of GTS has in common with COMMONTICKS.
- Normalize a series with values between -1 and 1 or 0 and 1 with ISONORMALIZE and NORMALIZE
- Smoothing transforms: WarpLib has a ton of functionalities. Among them are smoothing functions.
- Forecasting: Present the WarpScript extension for time series forecasting with an example of power consumption data and another one on air passenger data.
- Synchronize timestamps: Learn how to synchronize timestamps of two Geo Time Series and copy the locations from one to the other.
- Aggregate by calendar duration: How to aggregate by calendar duration that can be irregular (like months or years for example).
- Manipulating Multivalue and how to create it. The creation of multivalue GTS from within WarpScript needs a good understanding of how multivalue works in Warp 10.
Pattern and outliers detection
- Detecting a sequence of values: WarpScript is a handy toolkit built for time series manipulation. A common use case could be to detect a sequence of values lasting a given duration.
- Compare data hour to hour, day to day: Comparing every Sunday data in a given timezone is not trivial. With WarpScript language, this is a pretty straightforward time series analytics.
- Compare with the past: We use the auto-correlation function. We see how to implement it in WarpScript, with an exact calculation and then with a fast and accurate approximation.
- Detecting anomalies: WarpScript offers built-in functions to help you detect anomalies.
Advanced usages
- Variables in Macros: Variables and macros are the muscles and bones of your script. Here are concepts and functions to make the best use of variables in macros.
- Speeding up your analytics with SHM: SHM is a Warp 10 extension that allows you to keep data in RAM. It speeds up the data scientist job!
Factorize & share your code
- Author macros: Learn how WarpScript Macros can make your code more concise, robust, and shareable.
- Share macros: How to configure and use WarpFleet resolver to share macros internally or globally with WarpScript developers.
- WarpFleet Resolver: Using the WarpFleet Resolver you can expose your WarpScript macros, making them easily callable from any Warp 10 instance.
- Publish on WarpFleet: This is a very detailed HowTo that covers maven/bintray publication, a necessary step to publish Warp 10 extensions.
Dataviz tricks
- Compute a histogram of values with VALUEHISTOGRAM to feed a pie in your dataviz for example.
- Use LTTB to reduce the number of data points to be displayed
- Use WarpView to build quickly a dashboard.
Basic OPS and settings
- Backup and restore: How you can perform backups and restores of data managed by Warp 10 (with both the standalone and distributed versions).
- S3 archiving & automatic archiving: Learn how to manage the life cycle of your Time Series data by setting up automatic archiving of old data to an external system such as S3.
- Transfer data between 2 Warp 10: How to migrate data from a Warp 10 instance to another in 2 easy steps.
- Demystify LevelDB: LevelDB is very robust and rather simple. Learn how it works and how to read its LOG.
- Adjusting limits: How to configure WarpScript limits and use the full power of Warp 10.
- Warp 10 Accelerator: Introducing the Warp 10 Accelerator for blazing-fast data access and improved analytics. Enabling unbelievable time series data applications.
- All about tokens: Warp 10 uses cryptographic tokens to manage authentication and authorization, learn everything about this advanced mechanism in this comprehensive article.
- Schedule Warp 10 tasks with runners: Schedule Warp 10 tasks with runners, like down-sampling, archiving, deletions, computations, and so on.
Other tools to complete the Warp 10 Platform
- Discover WarpFleet, the Warp 10 artifacts repository, with Introducing WarpFleet, and the WarpFleet tag.
- Edit and execute your code with WarpStudio, a Web IDE dedicated to Warp 10. Start with WarpStudio unleashed.
- Discover how to use the Warp 10 Sandbox, a hosted environment for test-driving Warp 10 without deploying it, with this blog post.
- Explore the collection of charting web components dedicated to Warp 10, with WarpView.
- To create your own dynamic dashboard thanks to Discovery:
- Server monitoring with Warp 10 and Telegraf: Some of our customers are using Warp 10 as an IT monitoring platform. Warp 10 is sized to handle deep history, very large infrastructures at any scale, and obviously, small ones. In this post, you will learn how to build a simple IT monitoring system with Warp 10, Telegraf, and Discovery.
- Covid Tracker built with Warp 10 and Discovery: From datasets to dashboard: the full way to track Covid around the World with Warp 10. Discover the event mechanism of Discovery.
- Santa asset tracking and delivery service: Build a Santa tracker from scratch with Warp 10 and Discovery. Technics for asset tracking with a real-time monitoring dashboard.
- Using Warp 10 as a map tile server for Discovery: Discover how to use Warp 10 as a tiles server and plug it into Discovery.
- Full Discovery documentation is available here
- Interacting with Git repositories from Warp 10: Having the ability to store and read files from Warp 10 is something often requested. This post talks about a WarpScript extension that introduces functions to interact with git repositories.
- Leveraging WarpScript from Pig: The integration of WarpScript with Pig brings the power of advanced time series analytics to big data payloads.
- Leveraging WarpScript from Spark: Learn how you can scale your Time Series analytics using Spark and the Warp 10 Analytics Engine.
- Thrift and ProtoBuf: Learn how to use Thrift and Protocol Buffers directly from within WarpScript and without the need for code generation.
- Kafka Streams: Learn how Kafka Streams topologies can now be defined entirely in WarpScript and benefit from its vast number of functions.
- Kafka Plugin: Learn how you can connect to Kafka and consume messages directly from your Warp 10 instance.
To find more tutorials and documentation about the Warp 10 Platform, please visit warp10.io
| Don't forget to download the WarpScript CheatSheet. |
Need help?
Do you have questions or difficulties? Join the Warp 10 community on Slack. You can also ask your questions on Twitter or Stack Overflow.
SenX can support you in your use of the Warp 10 Platform. Contact the sales team to learn more about our complete offering.
