Oops Null Pointer

Java programming related

Logging aggregation and filtering for humans and machines

Some thoughts around better approaches logging have been coalescing after reading a few articles. With the filtering power of log aggregation and processing tools in mind like ELK stack, this article challenging the assumption that logs are for only for humans was interesting. I still think human readable logs can be important, so maybe a logging tool can configured to dump both a formatted string and a JSON detailed log into different log targets. Maybe a good log aggregator means there is no need to look elsewhere.

The idea of logs as the central source of data in distributed systems in intriguing. I found Jay Kreps’ article on the subject enlightening.

Finding the right granularity for the format is also an interesting problem. I have been working with a monitoring system that used the Common Base Event format mapped into a database table. The format (more specifically the current implementation) was not ideal for the way we use the system and is a source of performance issues, but is entrenched in the applications of the enterprise. A JSON based version stored in a document database and feed into ELK would be a much better fit.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: