Lennart Koopmann

How to get the Skype icon into the Ubuntu 11.10 Unity panel

hamburban — Thu, 24 Nov 2011 21:56:05 +0100

When you use Skype with Ubutun 11.10 on Unity you will encounter problems when closing it: Usually it just went to the GNOME panel, but now it just disappears. Choosing it from the Unity launcher will complain that there is already an instance of Skype running and you won’t be able to open it again.

The good news: There is a setting that defines which applications are allowed to hook into the panel and Skype is just missing. Run this command to get the current whitelist

lennart ~$ gsettings get com.canonical.Unity.Panel systray-whitelist
['JavaEmbeddedFrame', 'Wine', 'scp-dbus-service', 'Update-notifier']

Now just add Skype to it:

lennart ~$ gsettings set com.canonical.Unity.Panel systray-whitelist "['JavaEmbeddedFrame', 'Wine', 'scp-dbus-service', 'Update-notifier', 'Skype']"

That’s it! Now just log out and in again, start Skype and use it just like you were used to.

Graylog2 now has an internal message queue to better handle load...

hamburban — Mon, 21 Nov 2011 13:26:38 +0100

Graylog2 now has an internal message queue to better handle load spikes and batch-index messages to ElasticSearch.

Video

hamburban — Thu, 10 Nov 2011 10:38:49 +0100

What's coming in Graylog2 v0.9.6

hamburban — Tue, 08 Nov 2011 15:38:13 +0100

I am in the last steps of a Graylog2 v0.9.6 beta release these days: There are only a few tickets for the server and web interface left.

I’d like to take some time and give you an overview about what is changing and coming in this next version of Graylog2.

ElasticSearch is the new message storage

MongoDB has been dropped as message storage. It will stay for message counts (see faster graphs), settings and health values but no longer for storing the actual messages. Reason for this are performance problems when storing a lot of messages. To get a good speed it would have to keep all the messages in memory. ElasticSearch offers fast reads and real full text search features. Future versions of Graylog2 will make use of the full text search features of ElasticSearch - In 0.9.6 you will only get a huge performance increase. MongoDB is still great for storing the other stuff, but using it for the log messages seems to have been a mistake.

Faster graphs

In prior Graylog2 releases the graphs (like the analytics graph and the small stream graphs) were generated (and cached) from actual counts against the message collection. This was getting really slow if you had a high message throughput. From 0.9.6 on the server will count and store the message counts for per minute in MongoDB. This is not only much faster and less IO intensive but also more user friendly: Because it is independent from the message storage you can draw graphs over time periods that are no longer in ElasticSearch. You will be able to only keep messages of the last 2 months, but draw graphs over years. The UI for this will also change to allow easy drawing of long term graphs.

The Analytics Shell

Note that this screencast shows an early version of the shell. Count, distinct, and distribution queries for example are now displayed in the shell itself, not below it. You can also use stream names instead of their IDs for stream selectors.

See this blog post for an explanation and the screencast of the new Analytics Shell: http://www.lennartkoopmann.net/post/6801339730/graylog2-analytics-shell

Also check out the wiki page explaining the shell: https://github.com/Graylog2/graylog2-web-interface/wiki/The-Analytics-shell

New stream filter rules

There are some new stream filter rules like filename/line, regex host, full message and an “or higher” option for severities.

Hostgroups are dead

The hostgroups functionality has been removed. Read this explanation blog post for more information.

Bugfixes and improvements

A lot of bugfixes and improvements. There were some bugs in previous versions that could have been avoided. - Sorry about that. To avoid that in the future there will be a beta release and extended testing phase before releases. Expect improvements in the UX like the one for empty streams: When you created a stream it had no rules and was matching all messages coming in. From now on streams with no rules catch no messages. Streams are also disabled until you enable them - For example after fully configuring the stream rules and alarms.

All in all you can expect a double-awesome version 0.9.6 of Graylog2 that focuses on performance for huge amounts of log messages and long term archiving.

There will be a beta version very soon! A preview version with working ElasticSearch integration is already available for download.

Subscribe to this blog, the @graylog2 Twitter stream or the mailing list to stay up to date!

Winter ride this morning

hamburban — Sat, 05 Nov 2011 19:44:00 +0100

Winter ride this morning

Hostgroups in Graylog2 will die

hamburban — Tue, 18 Oct 2011 16:52:00 +0200

The next version of Graylog2 (0.9.6) will no longer include hostgroups. Let me explain why: You can now build hostgroups using streams since there is a hostname regex matcher rule. The difference is that you are more flexible because of all the other stream filters available. Keeping the hostgroups alive would basically mean having a duplicated feature with less flexibility.

Suggestions or feedback on the mailing list or in the comments of this post.

Me finishing the 102km 2011 Cyclassics

hamburban — Mon, 22 Aug 2011 19:57:55 +0200

Finished in 02:53:35 - Pretty happy with a position in the first third. Really nice race!

Events coming to Graylog2

hamburban — Sun, 31 Jul 2011 22:01:00 +0200

I am currently working on this new Graylog2 feature. (Screenshot shows the very early stage and is pretty ugly) You will be able to define rules on a time axis that trigger an event. You can define actions (like executing a script, sending an email, …) for whenever an event or a number of events are triggered. Also expect some fancy graphs going far into the history which are perfect for

This will require some Java and Rails hacking in the next time and I am not sure yet if this will make it into the 0.9.6 release or not. Let me know about your thoughts, ideas and suggestions!

I am also very interested in what kind of events you would like to set up.

Training

hamburban — Sun, 03 Jul 2011 22:54:00 +0200

What a great tour! Only 50km, but really fast and on slippy roads - Perfect to learn keeping your bike under control! Top Speed: 61,2 km/h - Shhooooossssssh!

Introducing the Graylog2 Analytics Shell

hamburban — Wed, 22 Jun 2011 22:42:00 +0200

I spent my last evenings hacking on a new Graylog2 feature which will be very useful for running powerful analytics on your log messages: The Graylog2 Analytics Shell. The detailled description is in the wiki.

Use this shell to extract information from your log messages. I always like the use case of API monitoring. You could easily find out if that one call is still used by consumers - Or even which consumers are still using it with a simple distinct call:

all.distinct({_oauth_consumer_key}, processed_controller = /^PrivateMessages#.+/)

Watch the shell in action in this short screencast: (Try the full screen mode and HD video)

This is the first working implementation but there is still stuff missing. For example:

Shell history available with up and down arrow keys
Improved design and layout
A history of count and distinct results
Tab completion (ZOMG!!)
Saving and sharing of commands
Short names for streams, to avoid typing the raw stream ID

The shell will be included in the next version: 0.9.6

Post your comments here or discuss in the mailing list.

Coffee time!

hamburban — Mon, 07 Mar 2011 19:03:24 +0100

Coffee time!

mongo_analyzer

hamburban — Wed, 02 Mar 2011 20:06:00 +0100

I wrote a little Sinatra application that can be very helpful at optimizing your MongoDB applications. It was very handy while optimizing the Graylog2 MongoDB parts so I made it public and configurable.

It allows you to enable/disable the MongoDB profiler and see it’s results in an overview page. Just start the profiler, run the MongoDB parts of you application you want to test and reload the mongo_analyzer overview page. It will list all queries that took longer than 100ms when in slow queries mode or all queries that were executed when in all queries mode.

You can also view and manage the indexes of your collections to try out effects on the query execution times.

I consider this software as beta. Please report bugs or feature requests in the GitHub issue tracker.

Download and installation instructions: https://github.com/lennartkoopmann/mongo_analyzer

Graylog2 performance

hamburban — Thu, 24 Feb 2011 18:42:35 +0100

I made a lot of improvements to Graylog2 performance in the last weeks. The server is now noticeably less consuming MongoDB because of host count caching: The message counts of hosts are now updated every 5 seconds instead of with every processed message. (graylog2-server 90d097) This leads to a lot more messages processed per second. I have made tests with 4000 GELF msgs/second that were not stressing my local machine too much.

There are also big speed improvements in the web interface. Using a database with around 17 million log messages (currently 17156568 to be precise) and keeping an eye on the MongoDB slow query log I was able to identify some bottlenecks and programming faults that lead to wasted computing time. The web interface connected to the 17 million messages is now really responsive and fun to use. It is important to optimize your MongoDB and data structure to archive short response times with huge databases. Follow these simple rules:

Avoid filters on the short_message There is no real full text search. The message filters are really expensive and utilizing your MongoDB. Pre-process your messages and set the parts you want to filter on as additional fields. Make sure to add an index on the additional fields you filter on: db.ensureIndex( { _field_name:1 } )
Disable deleting of messages You can disable deleting of messages in your general.yml from version 0.9.5 on. This will no longer add deleted = false to your queries.
Don’t use the blacklists Don’t send in logs you don’t need or block them using the message processing when they are received.
Don’t run the server in debug mode Blowing all the information to STDOUT costs a lot of time. Don’t do that in production.
Identify slow queries Find out what slows down your web interface by activating the MongoDB slow query log and profiling. Your streams or quickfilters might be significantly faster after setting the correct indexes. This is especially true when filtering on additional fields that are perfect for indexes.

Expect Graylog2 v0.9.5 with these performance fixes and some other new features to be released in the next days. Follow @graylog2 on Twitter and subscribe to the Graylog2 blog to stay up to date.

Really nice and easy to use key mapping for jQuery

hamburban — Wed, 23 Feb 2011 19:18:05 +0100

This jQuery plugin to build keyboard navigation is really easy to use and just works great: https://github.com/pixelmatrix/mapkey

AMQP support for Graylog2

hamburban — Sun, 23 Jan 2011 01:11:12 +0100

I built this in the last days: http://blog.graylog2.org/amqp-support-for-graylog2

Graylog2 v0.9.4 released!

hamburban — Sun, 09 Jan 2011 18:45:23 +0100

I released Graylog2 v0.9.4 yesterday. This release brings a lot of new features, bug fixes and speed improvements. Read the full announcement with a lot of screenshots here: http://www.graylog2.org/whatsnew/release/0-9-4

I srsly can't stop laughing: Party Gorilla explanation on #graylog2

hamburban — Sat, 11 Dec 2010 02:30:08 +0100

Hi - question about collection sizes... If the param messages_collection_size is increased after initial startup, and graylog2 server is restarted, will the collection cap be adjusted?
Another question: the tech team is really curious about the gorilla mascot, and it's origin. It looks to me like something from homestar runner, but I haven't watched those cartoons in a long time so I could be way off.
In terms of importance to me, an answer to Q1 is more important than an answer to Q2, but I'll take what I can get :)
doc_at_xen: I can explain the gorilla, but you'd better grab a snickers
Hey! How did you know what my wife calls my ..
oh wait, candy bar. Right, hold one, brb
doc_at_xen: we're watching, always watching
haha
just tell them, he's a motherfucking party gorilla with a motherfucking party hat
http://theoatmeal.com/comics/party_gorilla
that's just about what I said. I warned them that he was there to party and be happy, but that if they fucked up he was going to start killing, so they better keep their shit together.
excellent, thanks for that
I can't wait for graylog2con
gonna be some real live gorillas
If this works for me, I'll wear a gorilla suit to the convention
w t f, no fakes son
you don't want to insult party gorilla man
;)

Songs: Ohia - Farewell Transmission One of the best bands I...

hamburban — Tue, 26 Oct 2010 14:34:00 +0200

Songs: Ohia - Farewell Transmission

One of the best bands I have ever seen, one of the best songs ever written (Album version with better quality: http://www.youtube.com/watch?v=lBLu0lcseEY)

Graylog2 v. 0.9.3 has been released!

hamburban — Sun, 24 Oct 2010 14:15:00 +0200

Yay, celebrate! I just released Graylog2 v. 0.9.3 (graylog2-server and graylog2-web-interface). Here is a list of noteworthy changes:

Host groups

You can now create host groups that show all messages of hosts you added. This can be useful if you want to get an overview of all messages of a cluster or type of hosts.

New messages bar

A line with the number of new log messages since your last reload will appear when new messages arrived. Click on the link included and your web interface will be reloaded.

Additional/User-defined GELF fields

You can now define own GELF fields with own names that will be displayed in the full message view! Read more here: http://www.graylog2.org/about/gelf

MongoDB replica set support

Ever wanted to connect to a MongoDB replica set? You can do this now very easily:

# config/mongodb.yml (graylog2-web-interface)
hostname: [["localhost", 27017], ["localhost", 27018], ["localhost", 27019]]

# /etc/graylog2.conf (graylog2-server)
mongodb_replica_set = localhost:27017,localhost:27018,localhost:27019

Deleting in capped MongoDB collections

The standard Graylog2 MongoDB messages collections are capped. Deleting messages from capped collections did not work - This has been fixed.

Chunked GELF

The most time intensive feature: You can now chunk GELF messages over several UDP datagrams by prepending a byte header if the maximum size of 8192 byte for a single datagram is just not big enough for all your payload. The current versions of my GELF libraries gelf-rb and gelf-php support this and will chunk your messages automatically and transparently. Read more about chunked GELF here: http://www.graylog2.org/about/gelf

Also…

Support for GZIP as addition to ZLIB/Deflate compression of GELF messages in server
Server now using Maven as build tool
JavaDocs for whole public API of server
Several bugfixes and code improvements in the server and the web interface
Moar unit tests
Cleaner structured configuration files in config/*.yml

Thanks to bracki, 5v3n, s0enke, cipherpunk and portertech for contributing some great stuff! I’d also like to thank everybody who gave me feedback and ideas!

Also check out how to easily log all your Rails/Rack exceptions to Graylog2 with a simple Rack middleware gem: http://lennartkoopmann.net/post/1271247142

Follow the blog and @graylog2 on Twitter to stay up to date.

Download

Download the new packages here: http://www.graylog2.org/download

Are you also using Graylog2 in production? Let me know: lennart@socketfeed.com I’d like to include you on graylog2.org

Building a release.

hamburban — Sun, 24 Oct 2010 12:05:48 +0200

Building a release.