Logging aggregation and filtering for humans and machines
January 29, 2015
Posted by on
Some thoughts around better approaches logging have been coalescing after reading a few articles. With the filtering power of log aggregation and processing tools in mind like ELK stack, this article challenging the assumption that logs are for only for humans was interesting. I still think human readable logs can be important, so maybe a logging tool can configured to dump both a formatted string and a JSON detailed log into different log targets. Maybe a good log aggregator means there is no need to look elsewhere.
The idea of logs as the central source of data in distributed systems in intriguing. I found Jay Kreps’ article on the subject enlightening.
Finding the right granularity for the format is also an interesting problem. I have been working with a monitoring system that used the Common Base Event format mapped into a database table. The format (more specifically the current implementation) was not ideal for the way we use the system and is a source of performance issues, but is entrenched in the applications of the enterprise. A JSON based version stored in a document database and feed into ELK would be a much better fit.