Captain's Log, Stardate 4353.3

Star Trek Stardate

After having been taken inside a gigantic whirlwind caused by an inverted flux, we were lucky enough to rapidly reach Warp 10 and have been under way since then.

Jean-Luc Picard, Captain, USS Enterprise NCC-1701-D

Don't we all love logs? As a matter of fact, whether we like them or not, logs are produced massively by IT infrastructure and applications.

Historically, application logs were collected as local files on the machines where they were produced, and system logs were usually sent using syslog. Then came more advanced systems which could centralize logs. Then even more advanced systems which could index those centralized logs, with open source, commercial and cloud offerings for doing so in a managed way.

Many jumped on the bandwagon, thinking this log indexing trend was a panacea. Recently though, with the emergence of the observability practice, vast amounts of metrics are now being collected, and most of the devops look at those metrics first to identify anomalies or events which should be investigated further.

This means that logs are no longer the first source of information being looked at but rather a secondary source of information once the timeline of the issue under investigation has been identified. This also means that searching on logs is not something of utter importance anymore. It is enough to fetch the logs from a short period of time around the issue and look or search in those. This has an immediate impact on the log management infrastructure since an indexing layer is no longer needed, the cost of all those indexing servers can therefore be reduced by switching to something lighter.

Grafana announced at the beginning of 2019 the availability of their loki service, a lightweight log collection solution where logs are not subject to fulltext indexing but simply to indexing of meaningful elements such as the host which produced them or the level at which they were emitted. This lead me to suggest to Tom Wilkie, Grafana's VP of product that loki was nothing more than a Time Series DataBase (TSDB) for strings, analogy to which he agreed.

Since Warp 10™ is an advanced Time Series DataBase with support for String values it can very well be used to store and analyze logs. And the way to do it is deceivingly simple.

Leveraging Warp 10™ UDP plugin and WarpFleet™ Resolver for storing GELF log messages

The solution uses two features of Warp 10™ available in the Open Source version, the UDP Plugin and the WarpFleet™ Resolver. We will configure those two components to be able to receive and store log messages in the GELF format into Warp 10™.

WarpFleet

The UDP Plugin allows your instance of Warp 10™ to listen on a port for incoming UDP datagrams and process each of those datagrams using WarpScript™ code.

The WarpFleet™ Resolver allows you to use macros hosted on external repositories.

Enabling the UDP plugin

The UDP plugin is enabled by declaring the plugin in your Warp 10™ configuration file and setting a directory where listening configurations will be deployed. The following configuration does just that:

warp10.plugin.udp = io.warp10.plugins.udp.UDPWarp10Plugin
udp.dir = /path/to/udp.dir

After you restart your Warp 10™ instance, you can declare listening configurations by creating WarpScript™ (.mc2) files in the udp.dir directory. A listening configuration is a WarpScript™ code snippet which leaves a map on the stack with specific keys.

host

A STRING specifying the IP address the UDP Plugin listener should bind to.

port

A LONG defining the port on which the UDP Plugin listener should bind to.

parallelism

A LONG specifying how many threads should be spawned for consuming incoming UDP datagrams. This defaults to 1.

partitioner

A WarpScript™ macro which will take as input the source address (STRING), source port (LONG) and payload (BYTES) of an incoming UDP datagram and will return a LONG which will be used for assigning the processing of the datagram to a thread (by computing the remainder modulo parallelism). Note that if parallelism is 1, partitioner should not be defined.

qsize

The size of the queue used to buffer incoming messages. If the map contains a partitioner key, one such queue will be created per consuming thread, otherwise a single common queue will be used.

maxMessages

Maximum number of messages to consume in a batch, i.e. in a call to macro.

macro

A WarpScript™ macro which will process incoming datagrams. This macro will be fed with a list of lists, each list containing the source IP (STRING), source port (LONG) and payload (BYTES) of an incoming datagram. Note that the macro can be called with a NULL argument when timeout is defined and no messages arrived within the specified delay.

timeout

Maximum time to wait for an incoming datagram, expressed in milliseconds. If no message arrives within this delay, macro will be called with a NULL argument. If timeout is configured at 0, the UDP listener will wait indefinitely for a packet to arrive.

Using the WarpFleet™ Resolver

A WarpFleet™ repository is provided by SenX™ and configured by default in Warp 10™ 2.0.3+. You can ensure that the repository is correctly configured by checking that your configuration file contains the following line:

warpfleet.macros.repos = https://warpfleet.senx.io/macros

Putting it all together

After you have enabled both the UDP Plugin and WarpFleet™ Resolver and restarted your Warp 10™ instance, you can define a UDP listener which will process logs in the GELF format. The definition is really simple as it uses a macro exposed by the SenX WarpFleet™ Repository.

Create a .mc2 file in udp.dir with the following content:

{
  // Listen on all interfaces
  'host' '0.0.0.0'
  // Listen on port 12201, the default for GELF
  'port' 12201
  // Call the macro every 5s even if no messages arrived
  'timeout' 5000
  // Use a single thread for processing incoming logs
  'parallelism' 1
  // Call the senx/gelf/udp macro
  'macro' <% @senx/gelf/udp %> 
}

You are almost done, if you inspect the senx/gelf/udp macro, you will see that it expects two configuration keys to be defined. One holding the write token to use for storing the log messages, and another one defining a threshold of number of log messages beyond which they should be written to Warp 10™ even if some more messages must be processed. Those configuration keys should be set in the following manner:

token@senx/gelf/udp = YOUR_WRITE_TOKEN
threshold@senx/gelf/udp = 1000

Once you restart your Warp 10™ instance, it will be able to accept GELF formatted log messages on UDP port 12201. Those messages will end up in Warp 10™.

Reading the log messages

The macro senx/gelf/udp stores the received log messages as encoded, gzipped strings in Geo Time Series™ of class logs and with two labels host and level.

After the data has been fetched using the read token associated with the write token used by senx/gelf/udp, the values can be decoded. The WarpScript™ code below will fetch the log messages values according to the fetch criteria (class, labels, timespan), and will decode them, returning a LIST of MAP elements, each MAP being a GELF log message.

// Fetch the last two hours of log message for host 'myhost' at level '4'
[ 'YOUR_READ_TOKEN' 'logs' { 'host' 'myhost' 'level' '4' } NOW 2 h ] FETCH

// Extract the values into a list
VALUES FLATTEN

// Decode, decompress and parse the JSON values
<%
  DROP // Drop the map index
  OPB64-> UNGZIP 'UTF-8' BYTES-> JSON->
%> LMAP

Configuring Docker containers to emit logs to Warp 10™

As a side note, if you are using Docker containers in your infrastructure, you can very easily instruct them to use GELF over UDP for logging. Simply add the following options to the docker run command line:

--log-driver gelf --log-opt gelf-address=udp://aaa.bbb.ccc.ddd:12201 --log-opt gelf-compression-type=gzip

Where aaa.bbb.ccc.ddd is the IP address bound by your UDP listener.

Captain's Log. Supplemental. It seems our maneuver was a success, the flux has shrunk and now hardly attracts anyone.

Share