Sending events from our servers to a SIEM should be a pretty standard practice nowadays. However, in practice – it seems – still not clear what to send, and how to actually do this properly.The feature itself – to forward events to a remote host – is available in all Linux based systems, and it is the syslog protocol. However, different distributions may use different software to achieve the same goal. But before we would go into technical details, the first task is to decide what to send.
What should we send?
It is really depends on the purpose of remote logging, and the production services provided by the server itself. Let’s see what are the common options:
The Linux Audit system provides a way to track security-relevant information on your system. Based on pre-configured rules, Audit generates log entries to record as much information about the events that are happening on your system as possible. This information is crucial for mission-critical environments to determine the violator of the security policy and the actions they performed.
- Events generated by basic services
These are usually authentication related events generated by SSH, sudo, su, PAM, etc. Technically all the events marked as authpriv syslog facility.
- Kernel messages
Kernel messages are usually about very low level things, and most of them are just information that you normally just don’t care. However you should always monitor the kernel level errors! Means messages with error,crit,alert,emerg syslog severity.
- Production Application messages
Of course, you want to monitor you main production application, right? So if your server acts as a mail/web/database/application server, then all the relevant logs should be sent to a central log collection service.
- Other, misc system messages
There are other services like cron, ntp, dns, dhcp that might be important for you to monitor. Most of them are just simple information, but you should not miss the error related ones.
Surely you can simply send everything, but then you will realise pretty soon, that it may flood your SIEM with millions of ‘not so important’ messages.
How to send events to a remote host?
The short answer is quiet simple: via Syslog
However, there are multiple syslog standards in practice, where the main difference is the header of the messages. Especially the timestamp format used.
I strongly advice to avoid using any location specific time zone in your system events. You should always use GMT/UTC. Any other settings are just wrong, and will cause issues sooner or later.
The old – BSD style – syslog defined by the RFC 3164
Let’s see an example event:
<6>Oct 18 13:14:15 centos-8 kernel: Disabled fast string operations
Where the header parts as follow:
- <syslog facility * 8 + severity>
- date and time
- hostname or IP address
- application (kernel)
As you probably noted, this format having several issues:
- No Year information in the date! Which is just absurd. ‘Saving’ few characters, but leaving the year… What could be go wrong?!
- No time zone information, making the time meaningless without further information.
- On a busy system, hundreds of events are generated in the same second. So this is just not precise enough, making the ordering of the events impossible in practice.
- Hard to parse, impossible to correlate.
The ‘new’ syslog protocol defined by the RFC 5424
The ‘same’ event sent using the new – proper – format:
<6>1 2022-10-18T14:14:33.641241+00:00 centos-8 kernel - - - Disabled fast string operations
Where the header parts as follow:
- <syslog facility * 8 + severity>
- syslog version
- proper date and precise time with time zone!
- hostname or IP address
- application (kernel)
And the most important: this format can be easily parsed, and can be properly correlated to other events – even if those event comes from different system(s)!
How to configure your Linux host?
Now we have the proper syslog standard to follow, we just need to configure the syslog agent. However here comes the diversity, as different distributions using different software with different default settings – that are definitely not tuned for remote event forwarding.
Keep in mind that many modern Linux distribution are tend to use systemd-journald, which is a systemd component responsible for handling the events generated by the system. However it is not using the syslog protocol, but at least it can forward events to any syslog agent locally. Means, if your distribution does not shipped any syslog agent by default, you must choose one from the list below, and have to manually install it.
- sys(k)logd
Old, obsoleted, and can only able send events via UDP protocol. You should use this only as a last resort, if no any modern agent are available.
The community version are installed by default in many distributions, it is a modern, up-to date syslog agent/server with extensive features. Commercial version are available with more enterprise focused features and support.
Installed by default in many distributions (Red Hat and it’s derivatives), it is a modern up-to date syslog agent/server with extensive features.
NXLog offer superior log collection technology that works on all major operating systems, compatible with most SIEM and log analytics products, and can handle data sources that other tools cannot cope with – However this is not part of any distribution, so you need to download and install it manually.
It is also have a commercial, enterprise version with much more features and support.
Once you have installed your preferred syslog agent, you just need to configure it to:
- Use the RFC 5424 compatible message format,
- Filter for your selected events,
- Send those to the remote Syslog Collector and/or SIEM server via TCP
- Use an encrypted and authenticated TLS encapsulation,
- With local disc caching to not loose events in case of a (short) network outage.
See my further articles for specific configuration examples.