Captain's Log, Stardate 4353.3

After having been taken inside a gigantic whirlwind caused by an inverted flux, we were lucky enough to rapidly reach Warp 10 and have been under way since then.

Jean-Luc Picard, Captain, USS Enterprise NCC-1701-D

Don't we all love logs?

As a matter of fact, whether we like them or not, logs are produced massively by IT infrastructure and applications.

Historically, application logs were collected as local files on the machines where they were produced. System logs were usually sent using syslog. Then came more advanced systems which could centralize logs. Then even more advanced systems which could index those centralized logs, with open source, commercial and cloud offerings for doing so in a managed way.

Many jumped on the bandwagon, thinking this log indexing trend was a panacea. Recently though, with the emergence of the observability practice, vast amounts of metrics are now being collected. Most of the DevOps look at those metrics first to identify anomalies or events which should be investigated further.

This means that logs are no longer the first source of information being looked at but rather a secondary source of information once the timeline of the issue under investigation has been identified. This also means that searching on logs is not something of utter importance anymore. It is enough to fetch the logs from a short period of time around the issue and look or search in those. This has an immediate impact on the log management infrastructure. Since an indexing layer is no longer needed, the cost of all those indexing servers can therefore be reduced by switching to something lighter.

Grafana announced at the beginning of 2019 the availability of their Loki service. It is a lightweight log collection solution where logs are not subject to fulltext indexing, but simply to indexing of meaningful elements such as the host which produced them or the level at which they were emitted. This led me to suggest to Tom Wilkie, Grafana's VP of product, that Loki was nothing more than a Time Series DataBase (TSDB) for strings, an analogy to which he agreed.

Since Warp 10 is an advanced Time Series DataBase with support for String values, it can very well be used to store and analyze logs. And the way to do it is deceivingly simple.

Leveraging Warp 10 UDP plugin and WarpFleet Resolver for storing GELF log messages

The solution uses two features of Warp 10 available in the Open Source version, the UDP Plugin and the WarpFleet Resolver. We will configure those two components to be able to receive and store log messages in the GELF format into Warp 10.

The UDP Plugin allows your instance of Warp 10 to listen on a port for incoming UDP datagrams and process each of those datagrams using WarpScript code.

The WarpFleet Resolver allows you to use macros hosted on external repositories.

Enabling the UDP plugin

The UDP plugin is enabled by declaring the plugin in your Warp 10 configuration file and setting a directory where listening configurations will be deployed. The following configuration does just that:

warp10.plugin.udp = io.warp10.plugins.udp.UDPWarp10Pluginudp.dir = /path/to/udp.dir

After you restart your Warp 10 instance, you can declare listening configurations by creating WarpScript (.mc2) files in the udp.dir directory. A listening configuration is a WarpScript code snippet which leaves a map on the stack with specific keys.

Using the WarpFleet Resolver

A WarpFleet repository is provided by SenX and configured by default in Warp 10 2.0.3+. You can ensure that the repository is correctly configured by checking that your configuration file contains the following line:

warpfleet.macros.repos = https://warpfleet.senx.io/macros

Putting it all together

After you have enabled both the UDP Plugin and WarpFleet Resolver and restarted your Warp 10 instance, you can define a UDP listener which will process logs in the GELF format. The definition is really simple as it uses a macro exposed by the SenX WarpFleet Repository.

Create a .mc2 file in udp.dir with the following content:

{
  // Listen on all interfaces
  'host' '0.0.0.0'
  
  // Listen on port 12201, the default for GELF
  'port' 12201
    
  // Call the macro every 5s even if no messages arrived
  'timeout' 5000
  
  // Use a single thread for processing incoming logs
  'parallelism' 1
  
  // Call the senx/gelf/udp macro
  'macro' <% @senx/gelf/udp %>
}

You are almost done, if you inspect the senx/gelf/udp macro, you will see that it expects two configuration keys to be defined. One holding the write token to use for storing the log messages. And another one defining a threshold of number of log messages beyond which they should be written to Warp 10 even if some more messages must be processed. You should set those configuration keys in the following manner:

wtoken@senx/gelf/udp = YOUR_WRITE_TOKEN
threshold@senx/gelf/udp = 1000

Once you restart your Warp 10 instance, it will be able to accept GELF formatted log messages on UDP port 12201. Those messages will end up in Warp 10.

Reading the log messages

The macro senx/gelf/udp stores the received log messages as encoded, gzipped strings in Geo Time Series of class logs and with two labels host and level.

After the data has been fetched using the read token associated with the write token used by senx/gelf/udp, the values can be decoded. The WarpScript code below will fetch the log messages values according to the fetch criteria (class, labels, timespan), and will decode them, returning a LIST of MAP elements, each MAP being a GELF log message.

// Fetch the last two hours of log message for host 'myhost' at level '4'
[ 'YOUR_READ_TOKEN' 'logs' { 'host' 'myhost' 'level' '4' } NOW 2 h ] FETCH

// Extract the values into a list
VALUES FLATTEN

// Decode, decompress and parse the JSON values
<%
  DROP // Drop the map inde
  OPB64-> UNGZIP 'UTF-8' BYTES-> JSON->
%> LMAP

Configuring Docker containers to emit logs to Warp 10

As a side note, if you are using Docker containers in your infrastructure, you can very easily instruct them to use GELF over UDP for logging. Simply add the following options to the docker run command line:

--log-driver gelf --log-opt gelf-address=udp://aaa.bbb.ccc.ddd:12201 --log-opt gelf-compression-type=gzip

Where aaa.bbb.ccc.ddd is the IP address bound by your UDP listener.

Captain's Log. Supplemental. It seems our maneuver was a success, the flux has shrunk and now hardly attracts anyone.

Mathias Herberts

Co-Founder & Chief Technology Officer