April 8th, 2016
Logging from your application goes roughly like this:
- You have a data which you want to log
- You shove that data into a string which has time, severity and other information.
- Your log gets written to a file
- If you have centralised logging (as you should) your log shipper (like Logstash) reads the file and forwards it to the log server.
- In the log server you parse the log file to extract the separate fields for indexing (for searching purposes)
- Profit! (usually not)
So... we start with data structures at the begin and end up parsing it again at the end back to data structures. Seems kind of silly to create the log file in the middle doesn't it. And don't get me started with syslog protocol... Log files are usually meant for human consumption with grep and other command line tools. They are not best suited for machine processing. There are better formats and protocols for that.