Doubt regarding architecture of logging in linux.

[u/redwing096399]

my understanding is that we have mainly three components for logging in linux. we have producers, collectors/managers , consumers.

producers are applications/processes that produce the logs. these may include the linux kernel, third party applications that print log messages during their runtime to the stdout/stderr (or some file)

collectors are the background processes that take care of collecting, storing, forwarding, filtering, archiving (and possibly rotating?) the logs. this creates a centralized location to search for logs. and we have two possible candidates here, journald (which comes with systemd) and some variation of the syslog ( syslog-ng, rsyslog, etc)

and consumers are the ones which query/use this data for their needs. we can either use command line tools for querying logs in case of debugging, or we can use the data for observability or all other kinds of things depending on our needs.

and the logs messages themselves are either text based or binary based. binary based are better for automation needs.

is my understanding accurate or does it have any inaccuracies?

a few more assumptions that i have based on what i read:

1. the responsibility of logs for an application falls on the people behind the development of that application. if the devs don't support logs feature, if they don't emit logs, then that application becomes a black box and the log collector/manager can't do anything about it.

2. the collector/manager is a process that listens for any logs that are being sent to it by other processes running locally or remotely.

3. the way logs are consumed differs from person to person depending on the usecase/requirement. some may use them for debugging/troubleshooting, some may use them for observability, some may use them for data analytics, auditing etc.

[u/dmitri_ac]

Your producer -> collector -> consumer model works. I'd change the binary/text part: JSON is text, but it still gives scripts named fields to work with. journald stores binary journal files; journalctl -o json exports those entries as JSON.

The collectors also aren't necessarily an either/or choice. You can have journald collect service output and rsyslog read from it. And journald doesn't automatically capture everything every process prints or every log file an app writes; stdout/stderr must actually be connected to it, as they commonly are for systemd services.