Lennart Koopmann

I am in the last steps of a Graylog2 v0.9.6 beta release these days: There are only a few tickets for the server and web interface left.

I’d like to take some time and give you an overview about what is changing and coming in this next version of Graylog2.

ElasticSearch is the new message storage

MongoDB has been dropped as message storage. It will stay for message counts (see faster graphs), settings and health values but no longer for storing the actual messages. Reason for this are performance problems when storing a lot of messages. To get a good speed it would have to keep all the messages in memory. ElasticSearch offers fast reads and real full text search features. Future versions of Graylog2 will make use of the full text search features of ElasticSearch - In 0.9.6 you will only get a huge performance increase. MongoDB is still great for storing the other stuff, but using it for the log messages seems to have been a mistake.

Faster graphs

In prior Graylog2 releases the graphs (like the analytics graph and the small stream graphs) were generated (and cached) from actual counts against the message collection. This was getting really slow if you had a high message throughput. From 0.9.6 on the server will count and store the message counts for per minute in MongoDB. This is not only much faster and less IO intensive but also more user friendly: Because it is independent from the message storage you can draw graphs over time periods that are no longer in ElasticSearch. You will be able to only keep messages of the last 2 months, but draw graphs over years. The UI for this will also change to allow easy drawing of long term graphs.

The Analytics Shell

Note that this screencast shows an early version of the shell. Count, distinct, and distribution queries for example are now displayed in the shell itself, not below it. You can also use stream names instead of their IDs for stream selectors.

See this blog post for an explanation and the screencast of the new Analytics Shell: http://www.lennartkoopmann.net/post/6801339730/graylog2-analytics-shell

Also check out the wiki page explaining the shell: https://github.com/Graylog2/graylog2-web-interface/wiki/The-Analytics-shell

New stream filter rules

There are some new stream filter rules like filename/line, regex host, full message and an “or higher” option for severities.

Hostgroups are dead

The hostgroups functionality has been removed. Read this explanation blog post for more information.

Bugfixes and improvements

A lot of bugfixes and improvements. There were some bugs in previous versions that could have been avoided. - Sorry about that. To avoid that in the future there will be a beta release and extended testing phase before releases. Expect improvements in the UX like the one for empty streams: When you created a stream it had no rules and was matching all messages coming in. From now on streams with no rules catch no messages. Streams are also disabled until you enable them - For example after fully configuring the stream rules and alarms.

All in all you can expect a double-awesome version 0.9.6 of Graylog2 that focuses on performance for huge amounts of log messages and long term archiving.

There will be a beta version very soon! A preview version with working ElasticSearch integration is already available for download.

Subscribe to this blog, the @graylog2 Twitter stream or the mailing list to stay up to date!